Service Health
Incident affecting Google BigQuery
Global: BigQuery may experience elevated query latencies or failures.
Incident began at 2022-05-24 18:04 and ended at 2022-05-25 10:34 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Multi-region: euWarsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Multi-region: usIowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
Date | Time | Description | |
---|---|---|---|
| 2 Jun 2022 | 10:20 PDT | INCIDENT REPORT Summary: On 24 May 2022 from 18:04 to 25 May 2022 10:34 Google BigQuery experienced elevated query latencies and job failures in four regions (US multiregion, europe-west1, asia-east1, and asia-east2) for a duration of 16 hours, 30 minutes. A product change roll out was identified as the root cause. The issue was mitigated by rolling back the changes. The rollback was performed in phases to prevent additional impact on the service, which caused some regions to recover earlier than others. Affected customers experienced elevated latencies or failures for QUERY, IMPORT and EXPORT jobs in BigQuery. Root Cause: On 11 May 2022, BigQuery began a software rollout to improve audit logging for data that is imported from, queried, or exported to Google Cloud Storage (GCS). This rollout inadvertently contained a memory leak that gradually and incrementally manifested whenever these code paths were executed. This memory leak gradually consumed memory on BigQuery’s compute nodes until they were unable to accept new work to execute jobs, including queries, loads, and exports. The software rollout was deployed to all regions before the issue was detected and rolled back. Given the nature of the root cause, significant user load and time were required for the issue to manifest, meaning that the issue was not observed in internal testing, or in most of BigQuery's production regions. Remediation and Prevention: BigQuery will take the following actions to mitigate against such issues happening in the future:
Detailed Description of Impact In affected regions, a portion of BigQuery compute nodes were unable to service incoming jobs due to insufficient memory. As a result customer jobs - including queries, loads, and exports - experienced elevated latencies and failures while waiting for nodes with sufficient memory to become available. |
| 25 May 2022 | 14:54 PDT | Mini Incident-Report: We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 24 May 2022 18:04 Incident End: 25 May 2022 12:19 Duration: 18 hours, 15 minutes Affected Services and Features:
Regions/Zones: Global Description: Google BigQuery experienced elevated query latencies and job failures globally for a duration of 18 hours, 15 minutes. From preliminary analysis, the root cause of the issue is a product change which had been rolled out globally. The issue was mitigated by rolling back the changes. The rollback was performed in a phased manner to prevent additional impact on the service and some regions would have recovered earlier than others. Customer Impact:
|
| 25 May 2022 | 12:32 PDT | The issue with Google BigQuery is believed to be affecting a very small number of customers and our Engineering Team continues to work on mitigation across all regions. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We thank you for your patience while we're working on resolving the issue. |
| 25 May 2022 | 12:30 PDT | The issue with Google BigQuery is believed to be affecting a very small number of customers and our Engineering Team continues to work on mitigation across all regions. If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We thank you for your patience while we're working on resolving the issue. |
| 25 May 2022 | 12:00 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: us multi-region, europe-west1, asia-east1, and asia-east2 regions have recovered and our engineering team continue to monitor the situation for full recovery. We will provide more information by Wednesday, 2022-05-25 13:00 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 09:55 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: us multi-region, europe-west1, asia-east1, and asia-east2 regions have recovered and our engineering team is currently monitoring the situation for full recovery. We will provide more information by Wednesday, 2022-05-25 12:00 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 08:51 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: The US multi- region is nearly completely recovered and the europe-west1, asia-east1, and asia-east2 regions have fully recovered. Other regions are currently not affected but precautionary actions are being taking globally by our engineering team. We will provide more information by Wednesday, 2022-05-25 10:00 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 08:26 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: All Global locations including multi-region locations for BigQuery are impacted. Europe-west1 and Asia-east2 are currently mitigated Mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 09:00 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 07:40 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: All Global locations including multi-region locations for BigQuery are impacted. Europe-west1 is currently mitigated Mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 08:45 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 06:44 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures. Description: All Global locations including multi-region locations for BigQuery are impacted. Europe-west1 is currently mitigated Mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 07:30 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 06:25 PDT | Summary: Global: BigQuery may experience elevated query latencies or failures Description: Europe-west1 is currently mitigated Mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 07:30 US/Pacific. Diagnosis: Customers will experience Increased Query latency and possible job failures. Workaround: None at this time. |
| 25 May 2022 | 05:00 PDT | Summary: BigQuery degraded in us-multiregion, europe-west1, asia-northeast1, asia-southeast1, asia-southeast2, eu-canary, europe-west2, europe-west6, europe-west9, us-central1, us-west4. Customers will experience Increased Query latency and possible job failures. Description: Europe-west1 is currently mitigated and mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 06:30 US/Pacific. Diagnosis: Increased Query latency and possible job failures Workaround: None at this time. |
| 25 May 2022 | 03:49 PDT | Summary: BigQuery degraded in us-multiregion and europe-west1. Customers will experience Increased Query latency and possible job failures. Description: Europe-west1 is currently mitigated and mitigation work is currently underway for the remaining impacted regions by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 05:00 US/Pacific. Diagnosis: Increased Query latency and possible job failures Workaround: None at this time. |
| 25 May 2022 | 01:11 PDT | Summary: BigQuery degraded in us-multiregion and europe-west1. Customers will experience Increased Query latency and possible job failures. Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 03:36 US/Pacific. Diagnosis: Increased Query latency and possible job failures Workaround: None at this time. |
| 25 May 2022 | 00:29 PDT | Summary: BigQuery degraded in multiple regions. Customers will experience Increased Query latency and possible job failures Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2022-05-25 01:36 US/Pacific. Diagnosis: Increased Query latency and possible job failures Workaround: None at this time. |
- All times are US/Pacific