Service Health

INCIDENT REPORT

Summary:

On 24 May 2022 from 18:04 to 25 May 2022 10:34 Google BigQuery experienced elevated query latencies and job failures in four regions (US multiregion, europe-west1, asia-east1, and asia-east2) for a duration of 16 hours, 30 minutes. A product change roll out was identified as the root cause. The issue was mitigated by rolling back the changes. The rollback was performed in phases to prevent additional impact on the service, which caused some regions to recover earlier than others. Affected customers experienced elevated latencies or failures for QUERY, IMPORT and EXPORT jobs in BigQuery.

Root Cause:

On 11 May 2022, BigQuery began a software rollout to improve audit logging for data that is imported from, queried, or exported to Google Cloud Storage (GCS). This rollout inadvertently contained a memory leak that gradually and incrementally manifested whenever these code paths were executed. This memory leak gradually consumed memory on BigQuery’s compute nodes until they were unable to accept new work to execute jobs, including queries, loads, and exports. The software rollout was deployed to all regions before the issue was detected and rolled back. Given the nature of the root cause, significant user load and time were required for the issue to manifest, meaning that the issue was not observed in internal testing, or in most of BigQuery's production regions.

Remediation and Prevention:

BigQuery will take the following actions to mitigate against such issues happening in the future:

• In the short-term, enable comprehensive memory error detector coverage for the affected code path to detect possible memory leaks ahead of time.
• In the long-term, ensure such analysis is enabled for all BigQuery code paths. Introduce additional internal monitoring to detect memory pressure scenarios.

Detailed Description of Impact

In affected regions, a portion of BigQuery compute nodes were unable to service incoming jobs due to insufficient memory. As a result customer jobs - including queries, loads, and exports - experienced elevated latencies and failures while waiting for nodes with sufficient memory to become available.

Mini Incident-Report:

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 24 May 2022 18:04

Incident End: 25 May 2022 12:19

Duration: 18 hours, 15 minutes

Affected Services and Features:

• Google BigQuery

Regions/Zones: Global

Description:

Google BigQuery experienced elevated query latencies and job failures globally for a duration of 18 hours, 15 minutes. From preliminary analysis, the root cause of the issue is a product change which had been rolled out globally. The issue was mitigated by rolling back the changes. The rollback was performed in a phased manner to prevent additional impact on the service and some regions would have recovered earlier than others.

Customer Impact:

• Affected customers would have experienced elevated query latencies or in some cases query failures.

The issue with Google BigQuery is believed to be affecting a very small number of customers and our Engineering Team continues to work on mitigation across all regions.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We thank you for your patience while we're working on resolving the issue.

The issue with Google BigQuery is believed to be affecting a very small number of customers and our Engineering Team continues to work on mitigation across all regions.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We thank you for your patience while we're working on resolving the issue.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: us multi-region, europe-west1, asia-east1, and asia-east2 regions have recovered and our engineering team continue to monitor the situation for full recovery.

We will provide more information by Wednesday, 2022-05-25 13:00 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: us multi-region, europe-west1, asia-east1, and asia-east2 regions have recovered and our engineering team is currently monitoring the situation for full recovery.

We will provide more information by Wednesday, 2022-05-25 12:00 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: The US multi- region is nearly completely recovered and the europe-west1, asia-east1, and asia-east2 regions have fully recovered. Other regions are currently not affected but precautionary actions are being taking globally by our engineering team.

We will provide more information by Wednesday, 2022-05-25 10:00 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: All Global locations including multi-region locations for BigQuery are impacted.

Europe-west1 and Asia-east2 are currently mitigated

Mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 09:00 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: All Global locations including multi-region locations for BigQuery are impacted.

Europe-west1 is currently mitigated

Mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 08:45 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures.

Description: All Global locations including multi-region locations for BigQuery are impacted.

Europe-west1 is currently mitigated

Mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 07:30 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: Global: BigQuery may experience elevated query latencies or failures

Description: Europe-west1 is currently mitigated

Mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 07:30 US/Pacific.

Diagnosis: Customers will experience Increased Query latency and possible job failures.

Workaround: None at this time.

Summary: BigQuery degraded in us-multiregion, europe-west1, asia-northeast1, asia-southeast1, asia-southeast2, eu-canary, europe-west2, europe-west6, europe-west9, us-central1, us-west4. Customers will experience Increased Query latency and possible job failures.

Description: Europe-west1 is currently mitigated and mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 06:30 US/Pacific.

Diagnosis: Increased Query latency and possible job failures

Workaround: None at this time.

Summary: BigQuery degraded in us-multiregion and europe-west1. Customers will experience Increased Query latency and possible job failures.

Description: Europe-west1 is currently mitigated and mitigation work is currently underway for the remaining impacted regions by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 05:00 US/Pacific.

Diagnosis: Increased Query latency and possible job failures

Workaround: None at this time.

Summary: BigQuery degraded in us-multiregion and europe-west1. Customers will experience Increased Query latency and possible job failures.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 03:36 US/Pacific.

Diagnosis: Increased Query latency and possible job failures

Workaround: None at this time.

Summary: BigQuery degraded in multiple regions. Customers will experience Increased Query latency and possible job failures

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-05-25 01:36 US/Pacific.

Diagnosis: Increased Query latency and possible job failures

Workaround: None at this time.