ISSUE SUMMARY

On Tuesday 4 September 2018, Google Cloud Storage experienced 1.1% error rates and increased 99th percentile latency for US multiregion buckets for a duration of 5 hours 38 minutes. After that time some customers experienced 0.1% error rates which returned to normal progressively over the subsequent 4 hours. To our Google Cloud Storage customers whose businesses were impacted during this incident, we sincerely apologize; this is not the level of tail-end latency and reliability we strive to offer you. We are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT

On Tuesday 4 September 2018 from 02:55 to 08:33 PDT, customers with buckets located in the US multiregion experienced a 1.066% error rate and 4.9x increased 99th percentile latency, with the peak effect occurring between 05:00 PDT and 07:30 PDT for write-heavy workloads. At 08:33 PDT 99th percentile latency decreased to 1.4x normal levels and error rates decreased, initially to 0.146% and then subsequently to nominal levels by 12:50 PDT.

ROOT CAUSE

At the beginning of August, Google Cloud Storage deployed a new feature which among other things prefetched and cached the location of some internal metadata. On Monday 3rd September 18:45 PDT, a change in the underlying metadata storage system resulted in increased load to that subsystem, which eventually invalidated some cached metadata for US multiregion buckets. This meant that requests for that metadata experienced increased latency, or returned an error if the backend RPC timed out. This additional load on metadata lookups led to elevated error rates and latency as described above.

REMEDIATION AND PREVENTION

Google Cloud Storage SREs were alerted automatically once error rates had risen materially above nominal levels. Additional SRE teams were involved as soon as the metadata storage system was identified as a likely root cause of the incident. In order to mitigate the incident, the keyspace that was suffering degraded performance needed to be identified and isolated so that it could be given additional resources. This work completed by the 4th September 08:33 PDT. In parallel, Google Cloud Storage SREs pursued the source of additional load on the metadata storage system and traced it to cache invalidations.

In order to prevent this type of incident from occurring again in the future, we will expand our load testing to ensure that performance degradations are detected prior to reaching production. We will improve our monitoring diagnostics to ensure that we more rapidly pinpoint the source of performance degradation. The metadata prefetching algorithm will be changed to introduce randomness and further reduce the chance of creating excessive load on the underlying storage system. Finally, we plan to enhance the storage system to reduce the time needed to identify, isolate, and mitigate load concentration such as that resulting from cache invalidations.

The issue with Google Cloud Storage errors on requests to US multiregional buckets has been resolved for all affected users as of Tuesday, 2018-09-04 12:52 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to help prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

The mitigation efforts have further decreased the error rates to less than 1% of requests. We are still seeing intermittent spikes but these are less frequent. We expect a full resolution in the near future. We will provide another status update by Tuesday, 2018-09-04 16:15 US/Pacific with current details.

We are rolling out a potential fix to mitigate this issue. Impact is intermittent but limited to US Multiregional Cloud Storage Buckets buckets. We will provide another status update by Tuesday, 2018-09-04 15:30 US/Pacific with current details.

Mitigation is still ongoing but the error rates are decreasing. Latency in the 90th percentile have returned to normal levels but for the 99th percentile <1% of requests are still seeing increased latency. We will provide another status update by Tuesday, 2018-09-04 14:00 US/Pacific with current details.

We are still seeing intermittent errors and latency on some requests. Our Engineering Team is investigating the root cause and pursuing additional mitigation. We will provide another status update by Tuesday, 2018-09-04 12:45 US/Pacific with current details.

Temporary mitigation efforts have significantly reduced the error rate but we are still seeing intermittent errors or latency on requests. Full resolution efforts are still ongoing. We will provide another status update by Tuesday, 2018-09-04 11:15 US/Pacific with current details.

Mitigation efforts are starting to become effective and the rate of errors is decreasing, we are continuing to monitor and apply mitigation where necessary. Current data indicates that a small percentage of requests in the US region only are affected. Further updates will be provided by Tuesday, 2018-09-04 10:00 US/Pacific.

Mitigation work is currently underway by our Engineering Team. Current data indicates that a small percentage of requests in the US region only are affected. Further updates will be provided by Tuesday, 2018-09-04 08:45 US/Pacific.

We are still seeing intermittent errors for requests to Google Cloud Storage in the US region. Our Engineering Team is continuing mitigation work. We will provide another status update by Tuesday, 2018-09-04 08:00 US/Pacific with current details.

We are seeing intermittent errors for requests to Google Cloud Storage in the US region. Our Engineering Team is continuing mitigation work. We will provide another status update by Tuesday, 2018-09-04 07:00 US/Pacific with current details.