Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google BigQuery, Operations, Cloud Logging, Eventarc, Google Cloud Pub/Sub, Google App Engine, Google Cloud Storage

Logs delayed in europe-west3

Incident began at 2022-12-14 03:19 and ended at 2022-12-14 06:30 (all times are US/Pacific).

Previously affected location(s)

Frankfurt (europe-west3)

Date Time Description
14 Dec 2022 12:20 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 14 December 2022 03:19

Incident End: 14 December 2022 06:30

Duration: 3 hours, 11 minutes

Affected Services and Features:

Cloud Logging Eventarc BigQuery Google Cloud Storage Google Cloud PubSub

Regions/Zones: europe-west3

Description:

Cloud Logging experienced elevated latency with log ingestion in europe-west3 for a duration of 3 hours, 11 minutes. From preliminary analysis, the root cause of the issue was due to an invalid cross project sink which resulted in a backlog of subscriber messages. The issue was fully mitigated once invalid sink was deleted and the backlog cleared.

Google will be completing a comprehensive incident report in the following days that will provide a full root cause.

Customer Impact:

  • Affected customers may have experienced delayed log entries in Log Explorer or other export destinations for the duration of the incident.
14 Dec 2022 06:31 PST

The issue with Cloud Logging, Eventarc, Google BigQuery has been resolved for all affected users as of Wednesday, 2022-12-14 06:28 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
14 Dec 2022 05:46 PST

Summary: Logs delayed in europe-west3

Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Wednesday, 2022-12-14 08:00 US/Pacific.

We will provide more information by Wednesday, 2022-12-14 08:00 US/Pacific.

Diagnosis: Log entries written in europe-west3 are delayed by 1+ hours. They may not show up in Log Explorer or other export destinations.

Workaround: None at this time.
14 Dec 2022 04:56 PST

Summary: Logs delayed in europe-west3

Description: We are experiencing an issue with Cloud Logging, Eventarc, Google BigQuery beginning at Wednesday, 2022-12-14 03:19 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-14 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Log entries written in europe-west3 are delayed by 1+ hours. They may not show up in Log Explorer or other export destinations.

Workaround: None at this time.
14 Dec 2022 04:17 PST

Summary: Delay in ingesting the logs in europe-west3

Description: We are experiencing an issue with Cloud Logging and Google BigQuery.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-14 05:15 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers would experience a delay in logs visibility.

Workaround: None at this time.