Service Health

SUMMARY:

On Monday March 9 2015, for a duration of 96 minutes, users of the Google Cloud Storage JSON API experienced elevated error rates. During the impacted period, 5% of all JSON API requests were erroneously denied with rate limiting and authorization errors. This affected all users of the JSON API, including App Engine applications, but did not affect users of the XML API.

DETAILED DESCRIPTION OF IMPACT:

On March 9 2015 from 17:49 PDT to 19:25 Google Cloud Storage JSON API clients experienced a period of elevated error rates. During this period, 5% of requests to the Google Cloud Storage JSON API experienced Rate Limit Exceeded errors. The error rate peaked at 18% at 19:22.

ROOT CAUSE:

A single large project generated a spike in requests to the Cloud Storage JSON API. This triggered a monitoring system to alert Google engineers who, following standard procedures, severely limited the project’s API access rate to prevent impact on other projects.

The project’s clients responded to the resulting Rate Limit Exceeded errors with retry behavior causing an even larger traffic spike that, at its peak, was an order of magnitude greater than normal traffic rates. Because the service was not provisioned for this spike in traffic, it responded to this flood of requests by denying many of them with Rate Limit Exceeded errors, including a portion of traffic from other projects.

REMEDIATION AND PREVENTION:

Shortly after identifying the source of the traffic, Google engineers contacted the owners of the project and worked with them to analyze and mitigate the traffic surge. The limitations placed on the project protected the majority of traffic. Once Google engineers confirmed there was no more risk from the project, they removed the limit, and the remaining impact was resolved.

To prevent this class of issues in the future, Google engineers will implement a traffic control mechanism that is more robust in the face of client retry amplification. Additionally, Google engineers are creating more detailed monitoring systems that provide clearer, quicker insight into the source of traffic spikes of this nature. Finally, Google engineers are updating and refining standard procedures to rapidly and effectively respond to extreme situations of this nature.

The issue with the Cloud Storage JSON API was resolved as of 19:35 on Monday 2015-03-09. We are sorry for any issues this may have caused to you or your users and thank you for your patience and continued support. System reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We will provide a detailed analysis of this incident once we have completed our internal investigation.

We believe the issue is resolved as of 19:35. Our engineers are continuing to monitor the service closely until we are certain we have ascertained root cause. We will provide another status update by Tuesday 2015-03-10 12:00 (midday) Pacific time.

We believe the issue is resolved as of 19:35. Our engineers are continuing to monitor the service closely until we are certain we have ascertained root cause. We will provide another status update by 22:30 Pacific time.

We are making progress resolving the issue. Since 19:45 the Google Cloud Storage JSON API quota throttling issues are largely resolved. We are still investigating and monitoring the issue. We will provide another status update by 21:30 Pacific time.

We are still investigating the issue with Cloud Storage JSON API. We will provide another status update by 20:40 Pacific time.

We are still investigating the issue with Cloud Storage JSON API. We will provide another status update by 20:00 Pacific time.

We are currently experiencing an issue with Google Cloud Platform JSON API and some applications are experiencing an increase in quota throttling. For everyone who is affected, we apologize for any inconvenience you may be experiencing.

We will provide an update by 19:30 Pacific time with current details.

Correction, issue started at 2015-03-09 17:55 Pacific Time.

We're investigating an issue with Google Cloud Storage JSON API quota throttling beginning at 2015-03-09 16:55 Pacific Time. We will provide more information shortly.