Service Health
Incident affecting Google Cloud Storage
We are currently experiencing an issue with Cloud Storage and some users are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing.
Incident began at 2014-07-07 13:05 and ended at 2014-07-07 20:42 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 18 Jul 2014 | 13:32 PDT | SUMMARY: On Monday 7 July 2014, some Google Cloud Storage clients experienced elevated latency and error rates for uploads of large objects for a duration of 7 hours and 37 minutes. We apologize if you were affected by this incident. We are taking immediate steps to improve the service’s performance and availability. DETAILED DESCRIPTION OF IMPACT: On Monday 7 July 2014 from 13:05 to 20:42 US/Pacific, Cloud Storage clients that connected to Google’s production network via our Iowa datacenter experienced elevated latency and error rates when uploading large objects. The issue affected Google Compute Engine instances in US zones and some App Engine applications hosted in the US. Clients affected by this issue had an average error rate of 55% on PUT or POST requests that were larger than 5 MB. The error rate for requests less than 5 MB during the incident was 1.4%. ROOT CAUSE: The root cause of the incident was an unplugged link in a network circuit. This resulted in a reduction of network capacity between datacenters used by Cloud Storage. REMEDIATION AND PREVENTION: The incident was resolved by finding and resolving the networking issue. To prevent recurrence, we will raise the priority of Cloud Storage’s network traffic so that a similar loss of network capacity will not affect performance or availability. |
| 18 Jul 2014 | 13:31 PDT | The problem with Cloud Storage was resolved as of 2014-07-07 20:42 US/Pacific time. We apologize for any issues this may have caused you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. |
| 18 Jul 2014 | 13:30 PDT | We are still investigating the issue with Cloud Storage. Some users continue to experience elevated latency. We will provide another status update by Monday, 2014-07-07 20:40 (all times are in US/Pacific). |
| 18 Jul 2014 | 13:29 PDT | We are still investigating the issue with Cloud Storage. We will provide another status update by Monday, 2014-07-07 19:40 (all times are in US/Pacific). |
| 18 Jul 2014 | 13:28 PDT | We are currently experiencing an issue with Cloud Storage and some users are experiencing elevated latency. For everyone who is affected, we apologize for any inconvenience you may be experiencing. |
- All times are US/Pacific