Service Health

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 17 April 2022 20:52

Incident End: 18 April 2022 17:33

Duration: 20 hours, 41 minutes

Affected Services and Features:

Google Cloud Logging

Regions/Zones: Global

Description:

Google Cloud Logging experienced elevated error rates and high latency in log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not have been able to see their recently written logs for a duration of 20 hours, 41 minutes.

From preliminary analysis, an unexpected increase in uncached NOT_FOUND errors while validating log bucket configurations caused a large increase in the overall volume of configuration reads. This in turn caused latency of configuration reads and therefore validations to increase, leading to an ingestion backlog and a delay in log ingestion to the storage layer of our query backend.

The issue was resolved at 17:33 US/Pacific once the ingestion backlog cleared.

Customer Impact:

• Customers using global storage buckets or storage buckets in us-central1 to store their logs may not have been able to see their recently written logs for most of the duration of the event.

• Customers may have experienced high latency or failures up to 90% on reads from Google Cloud Logging and other query requests.

The issue on Cloud Logging Ingestion and querying has been resolved for all affected projects as of Monday, 2022-04-18 17:33 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Cloud Logging is experiencing elevated latencies for log querying. Customers using global storage buckets or storage buckets in us-central1 to store their logs may see latencies on queries run against Cloud Logging.

Description: Our engineering team continues to investigate the issue with Log Querying on Cloud Logging. Log ingestion backlogs have cleared but customers may experience issues with querying logs.

We will provide an update by Monday, 2022-04-18 18:00 US/Pacific with current details.

Diagnosis: Customers may experience high latency or failures when listing log entries and for other query requests.

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Our engineering team has determined that further investigation is required to mitigate the issue with log querying.

Log ingestion backlog has cleared but customers may experience issues with querying the logs.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2022-04-18 17:00 US/Pacific.

Diagnosis:

• Customers may experience high latency or failures when listing log entries and for other query requests

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Following the fix roll out, log ingestion backlog has cleared but customers may experience issue with querying the logs.

Mitigation work is currently underway by our engineering team for the issue with log querying.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2022-04-18 16:00 US/Pacific.

Diagnosis:

• Customers may experience high latency or failures when listing log entries and for other query requests

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Following the fix roll out, log ingestion backlog has cleared but customers may experience issue with querying the logs.

Our engineering team is working to fix the log querying issue.

We will provide more information by Monday, 2022-04-18 15:00 US/Pacific.

Diagnosis:

• Customers may experience high latency or failures when listing log entries and for other query requests

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Following the fix roll out, log ingestion backlog has cleared but customers may experience issue with querying the logs.

Our engineering team is working to fix the log querying issue.

We will provide more information by Monday, 2022-04-18 14:00 US/Pacific.

Diagnosis:

• Log ingestion backlog has cleared but customers may experience issue with querying the logs.

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: A fix roll out is completed and customers may still experience delays while the log ingestion backlog is being cleared.

The backlog is expected to clear by Monday, 2022-04-18 15:00 US/Pacific.

We will provide more information by Monday, 2022-04-18 13:00 US/Pacific.

Diagnosis:

• There is a high latency in log ingestion with median latency of around 2 hours and 99% latency of 4.5 hours.

• Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Monday, 2022-04-18 12:00 US/Pacific.

We will provide more information by Monday, 2022-04-18 12:00 US/Pacific.

Diagnosis: * There is a high latency in log ingestion with median latency of around 2 hours and 99% latency of 4.5 hours.

• Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Workaround: None at this time.

Summary: Cloud Logging is experiencing elevated latency for log ingestion. Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Monday, 2022-04-18 12:00 US/Pacific.

We will provide more information by Monday, 2022-04-18 11:00 US/Pacific.

Diagnosis: * There is a high latency in log ingestion in our backends with median latency of around 2 hours and 99% latency of 4.5 hours.

• Customers using global storage buckets or storage buckets in us-central1 to store their logs may not be able to see their recently written logs for a few hours.

Workaround: None at this time.