Incident Report

Summary

On 13 October 2022, BigQuery Storage WriteAPI observed elevated error rates in the US Multi-Region for a period of 5 hours. To our BigQuery Storage WriteAPI customers who were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. We have conducted an internal investigation and are taking steps to improve our service.

Root Cause

BigQuery Storage Write API is the high-traffic, unified data ingestion API in BigQuery [1].

On 13 October 2022 23:30 US/Pacific, there was an unexpected increase of incoming and logging traffic combined with a bug in Google’s internal streaming RPC library that triggered a deadlock and caused the Write API Streaming frontend to be overloaded.

Google’s internal automation correctly kicked in and attempted to resolve the issue by increasing memory and scaling up more instances. But because of the bug in the Write API, the existing instances remained stuck, and because of the effects of load balancing the elevated error rates continued to affect customers accessing the Write API. The fix was to restart the stuck instances which was completed by our engineers.

Remediation and Prevention

The issue was brought to our attention through internal alert on 13 October 2022 23:47 US/Pacific, and Google engineers immediately started an investigation. Google’s automated systems automatically tried increasing memory on the stuck instances at 13 October 23:40 US/Pacific, and Google engineers manually increased the maximum number of Write API instances from 200 to 1000, however neither of them resolved the issue. Between 14 October 02:30 and 02:50 US/Pacific, Google engineers continued to add resources, until it was discovered that the stuck instances needed to be restarted. This action commenced between 14 October 03:20 US/Pacific and 04:30 US/Pacific. The issue was fully mitigated on 14 October 2022 04:30 US/ Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

Fix the bug in Google’s internal RPC library (COMPLETED) Fix the bug in the Write API which caused the cascading deadlock (COMPLETED) Deploying additional automation in the Write API back end to automatically load balance based on concurrent connections, at the same time providing improved error handling to reduce unavailability errors.

Detailed Description of Impact

On 13 October 2022 23:30 US/Pacific, customers making calls using Write API in the US Multi-Region observed increased levels of connection failures. Customers making calls using InsertAll API in the US also may have observed a slight increase in failures due to the subsequent increase in traffic.

Reference(s): [1] https://cloud.google.com/bigquery/docs/write-api

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. Google will be providing a full Incident Report to include a finalized root cause and appropriate preventative action items to ensure that this type of outage won’t recur.

(All Times US/Pacific)

Incident Start: 13 October 2022 23:30

Incident End: 14 October 2022 04:30

Duration: 5 hours

Affected Services and Features:

BigQuery - Storage Write API

Regions/Zones: Multi-Region (US)

Description:

Customers making calls to the streaming API using BigQuery observed elevated error rates in US regions for a period of 5 hours. The issue was mitigated by appropriately increasing resources and manually restarting old tasks, which were overloaded with pending connections.

From preliminary analysis, an increase in calls to the Write API, a unified data-ingestion API in BigQuery [1], caused frontend resources to be overloaded. A suboptimal distribution of the load resulted in an elevated error rate. Google engineering is continuing to investigate the matter, and we will publish a final root cause analysis on the Google Cloud Service Health Dashboard as part of a formal Incident Report.

Customer Impact:

During this period of time:
Customers making calls using Write API in the US region observed increased levels of connection failures. Customers making calls using InsertAll API in the US region may also have observed a slight increase in failures due to subsequent increase in traffic.

Reference(s): [1] https://cloud.google.com/bigquery/docs/write-api

The issue with Google BigQuery has been resolved for all affected users as of Friday, 2022-10-14 04:50 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: BigQuery is experiencing issues with streaming API in US region

Description: We are experiencing an issue with Google BigQuery.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-10-14 05:10 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers making calls to streaming API in the US region may see raised levels of errors.

Workaround: None.