Service Health
Incident affecting Google App Engine
We're investigating an issue with Google App Engine Cloud Endpoints beginning at approximately Thursday, 2014-05-22 17:00 (all times are in US/Pacific)]. We will provide more information shortly.
Incident began at 2014-05-22 15:50 and ended at 2014-05-23 02:35 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 10 Jul 2014 | 03:14 PDT | SUMMARY: From 15:50 US/Pacific on the 22nd May, to 02:35 on the 23rd May, applications using the Google API (GAPI) JavaScript client library experienced intermittent errors when attempting to access Google Cloud Endpoints. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability. DETAILED DESCRIPTION OF IMPACT: Starting at 15:50 US/Pacific on 22nd May, clients which loaded a new version of the GAPI library began to receive 404 “Not Found” responses from the Google API servers. The issue reached a plateau at 21:20. Resolution began at 01:30 on 23rd May. From that time, errors decreased steadily until 02:35, at which point the source of the issue was fully resolved. Clients using a cached copy of the faulty library version may have continued to experience errors after 02:35. Over the duration of the incident, 3.9% of requests received a 404 response; these requests were widely distributed, such that approximately 57% of Cloud Endpoints referrers made at least one request with a corrupt path. ROOT CAUSE: The root cause of the incident was an error in the GAPI JavaScript library. If a ‘root’ URL parameter was specified in the gapi.client.load() function, the path segment of the URL was duplicated in the generated endpoint URL, causing a 404 error in the endpoint RPC call. Due to another error in the automated release and deployment testing mechanisms, this issue was not detected until it was fully deployed on Google’s infrastructure. REMEDIATION AND PREVENTION: To remedy the issue, as soon as the code error was identified, Google engineers began a rollback to the previous version of the software package that includes the GAPI library. This process took approximately 1 hours 5 minutes to complete across the entire Google estate. To prevent recurrences, Google engineers are engaged in an ongoing review of the GAPI JavaScript library testing and monitoring to identify and improve weaknesses in the integration testing. The review also includes the integration with Google’s deployment automation to ensure the automatic reversion of problematic software rollouts. We are also upgrading GAPI alerting to automatically notify us quickly, should such a problem recur in the future. |
| 10 Jul 2014 | 03:14 PDT | The problem with Google App Engine Cloud Endpoints in JavaScript clients was resolved as of 02:45 AM Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. |
| 10 Jul 2014 | 03:13 PDT | We have fixed the issue affecting Google App Engine Cloud Endpoints in JavaScript clients and are gradually rolling-out a fixed version. We estimate full resolution of the issue by 04:30 AM Pacific. We will provide an update by 03:00 AM Pacific. |
| 10 Jul 2014 | 03:13 PDT | We are currently experiencing an issue with Google App Engine Cloud Endpoints in JavaScript clients. For everyone who is affected, we apologize - we know you count on Google to work for you and we're working hard to restore normal operation. We will provide an update by 01:45 AM Pacific with current details, and if available an estimated time for resolution. |
| 10 Jul 2014 | 03:12 PDT | We are currently experiencing an issue with Google App Engine Cloud Endpoints in JavaScript clients. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 01:15 AM Pacific with current details. |
| 10 Jul 2014 | 03:12 PDT | We're investigating an issue with Google App Engine Cloud Endpoints beginning at approximately Thursday, 2014-05-22 17:00 (all times are in US/Pacific)]. We will provide more information shortly. |
- All times are US/Pacific