Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Operations

Cloud Logging delays on log ingestion affecting 25% of projects.

Incident began at 2020-11-19 08:49 and ended at 2020-11-19 21:16 (all times are US/Pacific).

Date Time Description
Nov 19, 2020 21:16

The issue with Cloud Logging has been resolved for all affected projects as of Thursday, 2020-11-19 21:15 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Nov 19, 2020 19:31

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup is ongoing. Some users may experience slow queries.

We expect a full resolution within 3 hours.

We will provide an update by Thursday, 2020-11-19 22:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 18:26

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup and prevention efforts are ongoing. Some users may experience slow queries.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2020-11-19 19:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 16:38

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup and prevention efforts are ongoing, but the majority of users should see their log data available and queries succeeding.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2020-11-19 18:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 15:14

Description: The backlog has been fully processed, but we are still seeing elevated error rates on queries with new data. Mitigation efforts are still ongoing.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 14:04

Description: Current ETA is ~2 hours to finish processing the backlog.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 13:40

Description: Log ingestion has been resumed, and now the backlog of recent data (about 4.5 hours) is being processed.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 12:21

Description: Mitigation work is still underway by our engineering team. Ingestion is temporarily paused for a couple hours to speedup the recovery efforts.

We will provide more information by Thursday, 2020-11-19 15:00 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.

Nov 19, 2020 11:18

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 12:30 US/Pacific.

Diagnosis: Log writes and reads experiencing latency. 25% of customers may see symptoms.

Workaround: None at this time.

Nov 19, 2020 10:47

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 11:50 US/Pacific.

Diagnosis: Log writes and reads experiencing latency. 25% of customers may see symptoms.

Workaround: None at this time.

Nov 19, 2020 10:18

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 11:45 US/Pacific.

Diagnosis: Log writes and reads in us-central1 experiencing latency. 25% of customers may also see similar symptoms in other locations when using the global logging region.

Workaround: None at this time.