Service Health

SUMMARY: Between Tuesday 17 February and Wednesday 18 February 2015, load and export jobs to the BigQuery service experienced significant increases in latency for a period of 29 hours. If your application was affected by this increase in latency, we apologize. This is not the standard of service BigQuery strives to uphold and we are taking immediate action to prevent these types of disruptions in the future.

DETAILED DESCRIPTION OF IMPACT: On Tuesday 17 February 2015 at 14:46 PST, all submitted load jobs began to experience elevated latency, ranging from minutes to hours of additional processing latency at the 99th percentile during the incident. The impact varied for each project and was dependent on the size of their respective load jobs. All incoming load jobs were affected until 18 February 2015 at 19:45 PST.

ROOT CAUSE: The BigQuery service received an elevated number of high volume load jobs which overwhelmed the existing ingestion capacity and caused newly created jobs to wait increasingly longer to be scheduled.

REMEDIATION AND PREVENTION: In response to the increased latency, Google engineers provisioned more resources for load jobs at 12:11 PST on 18 February 2015, providing a short term improvement in processing latency. The BigQuery engineering team then isolated the subset of jobs which were overwhelming system capacity. Google Support engineers were able to confirm with the job owners that the jobs were unnecessary and safe to terminate, which the BigQuery engineering team proceeded to do, allowing processing latency to return to normal.

Google engineers are working on the resource allocation system for large scale import jobs to limit the possibility of a concentrated number of intensive load jobs overwhelming system resources. They will also be optimizing monitoring and alerting for increased latency and active jobs waiting to be executed. Finally, they will be exploring new ways of exposing job management to customers.

The problem with Google BigQuery service should be resolved as of 19:38 US/Pacific time. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We are seeing that the backlog of BigQuery jobs are now being processed, and you should be seeing improvements. We are continuing to monitor the situation, and will provide an update by 2015-02-18 20:00 US/Pacific.

We are still investigating the issue with Google BigQuery service. Some load jobs are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide another status update by 2015-02-18 19:00 US/Pacific

We are still investigating the issue with Google BigQuery service. Some load jobs are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide another status update by 2015-02-18 18:00 US/Pacific

We are still investigating the issue with Google BigQuery service where some load jobs are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide another status update by 2015-02-18 17:00 US/Pacific

We are still investigating the issue with Google BigQuery service. Some load jobs are experiencing elevated latency. We will provide another status update by 2015-02-18 16:00 US/Pacific

We are still investigating issue with Google BigQuery where some load jobs are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide another status update by 2015-02-18 15:00 US/Pacific

We are still investigating the issue with Google BigQuery service where some load jobs are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide another status update by 2015-02-18 14:00 US/Pacific

We're investigating an issue with Google BigQuery service beginning at 2015-02-17 14:50 US/Pacific. Execution of load jobs are taking longer than usual. We will provide more information by 2015-02-18 13:00 US/Pacific