Service Health

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 27 Oct 2021 02:00

Incident End: 30 Oct 2021 12:00

Duration: 82 hours

Affected Services and Features:

Google BigQuery

Regions/Zones: US Multi-region

Description:

Google BigQuery service performance experienced an increase in latency for queries in multiple regions in the US for 82 hours. From preliminary analysis, the root cause of the issue was capacity increase that caused rebalancing of workload and increased the scheduler processing time. This caused delays in query execution.

Customer Impact:

• Queries exhibited higher than normal latency.

• Customers were unable to fully utilize their reserved slots.

The issue with Google BigQuery has been resolved for all affected projects as of Saturday, 2021-10-30 16:30 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: Mitigation work has completed and our engineering team is continuing to monitor to confirm that the issue has been resolved.

We will provide more information by Sunday, 2021-10-31 10:00 US/Pacific.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: Mitigation work is underway and our engineering team continues to monitor progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Saturday, 2021-10-30 16:00 US/Pacific.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: Mitigation work continues to be underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Saturday, 2021-10-30 10:00 US/Pacific.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2021-10-29 18:00 US/Pacific.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: Engineering team continues to investigate the root cause and are currently working on the mitigation steps.

We will provide an update by Friday, 2021-10-29 16:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.

Summary: Multi-region/US: BigQuery queries may show latencies during execution.

Description: We are experiencing an issue with Google BigQuery.

Our engineering team continues to investigate the issue and is working to mitigate the impact for our customers.

We will provide an update by Friday, 2021-10-29 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some queries may may exhibit higher than normal latencies during execution.

Workaround: None at this time.