Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Operations

Cloud Logging delays on log ingestion affecting 25% of projects.

Incident began at 2020-11-19 08:49 and ended at 2020-11-19 21:16 (all times are US/Pacific).

Date Time Description
19 Nov 2020 21:16 PST

The issue with Cloud Logging has been resolved for all affected projects as of Thursday, 2020-11-19 21:15 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
19 Nov 2020 19:31 PST

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup is ongoing. Some users may experience slow queries.

We expect a full resolution within 3 hours.

We will provide an update by Thursday, 2020-11-19 22:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 18:26 PST

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup and prevention efforts are ongoing. Some users may experience slow queries.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2020-11-19 19:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 16:38 PST

Description: We believe the issue with Cloud Logging is partially resolved. Additional cleanup and prevention efforts are ongoing, but the majority of users should see their log data available and queries succeeding.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2020-11-19 18:30 US/Pacific with current details.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 15:14 PST

Description: The backlog has been fully processed, but we are still seeing elevated error rates on queries with new data. Mitigation efforts are still ongoing.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 14:04 PST

Description: Current ETA is ~2 hours to finish processing the backlog.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 13:40 PST

Description: Log ingestion has been resumed, and now the backlog of recent data (about 4.5 hours) is being processed.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 16:45 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 12:21 PST

Description: Mitigation work is still underway by our engineering team. Ingestion is temporarily paused for a couple hours to speedup the recovery efforts.

We will provide more information by Thursday, 2020-11-19 15:00 US/Pacific.

Diagnosis: Log writes and reads experiencing latency on logs ingested after 2020-11-19 08:41 US/Pacific. 25% of customers are affected and may see incomplete queries.

Workaround: None at this time.
19 Nov 2020 11:18 PST

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 12:30 US/Pacific.

Diagnosis: Log writes and reads experiencing latency. 25% of customers may see symptoms.

Workaround: None at this time.
19 Nov 2020 10:47 PST

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 11:50 US/Pacific.

Diagnosis: Log writes and reads experiencing latency. 25% of customers may see symptoms.

Workaround: None at this time.
19 Nov 2020 10:18 PST

Description: Mitigation work is currently underway by our engineering team, the current focus is on stabilizing new requests, then proceeding to process backlog.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2020-11-19 11:45 US/Pacific.

Diagnosis: Log writes and reads in us-central1 experiencing latency. 25% of customers may also see similar symptoms in other locations when using the global logging region.

Workaround: None at this time.