Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google App Engine

Google App Engine issue with Cloud Endpoints service beginning at 21:00 US/Pacific on September 22, 2014

Incident began at 2014-09-22 21:00 and ended at 2014-09-23 08:00 (all times are US/Pacific).

Date Time Description
26 Sep 2014 12:11 PDT

SUMMARY: Between Monday 22 September 2013 and Tuesday 23 September 2014, some requests to Google Cloud Endpoints from clients using the Google APIs Client Library for Javascript failed for a duration of 12 hours and 33 minutes. We apologize for any impact this incident may have had on your Cloud Endpoints applications. We hold ourselves to a high standard, and we failed to meet that standard. We are taking immediate action to ensure incidents like this do not happen in the future.

DETAILED DESCRIPTION OF IMPACT: From Monday 22 September 2014 19:27 PDT until Tuesday 23 September 2014 08:00 PDT, 3% of requests to Google Cloud Endpoints made using the Google APIs Client Library for JavaScript failed with an HTTP 400 or HTTP 500. API calls that passed data as part of the request body were affected by this issue.

ROOT CAUSE: Google engineers published a new configuration for the Google APIs Client Library for Javascript. This configuration enabled an experimental feature for communicating with Cloud Endpoints, and was intended only for a subset of clients but erroneously was applied to all clients. The experimental feature contained a bug that caused API call requests structured a specific way to fail under certain conditions.

REMEDIATION AND PREVENTION: Google engineers were alerted to the issue via an external report at 01:05 PDT. The engineers immediately began investigating and identified the configuration change as the cause of the issue at 03:50 PDT. To resolve this issue, the engineers began a rollback of the configuration change at 04:00 PDT which completed at 08:00 PDT.

To prevent this issue from occurring in the future we will improve testing of configuration changes. We will also improve internal documentation to make identifying and rectifying incorrect configuration pushes easier.
23 Sep 2014 08:10 PDT

The problem with Google Cloud Endpoints JavaScript client was completely resolved as of Tuesday, 2014-09-23 08:00 (all times are in US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.
23 Sep 2014 06:23 PDT

Fix for the issue affecting Google Cloud Endpoints JavaScript client has been rolled-out to majority of users. We will provide another update by 08:00 US/Pacific.
23 Sep 2014 05:13 PDT

We have fixed the issue affecting Google Cloud Endpoints JavaScript client and are gradually rolling-out a fixed version. We estimate full resolution of the issue by 06:30 US/Pacific Pacific. We will provide an update by 06:00 AM.
23 Sep 2014 03:28 PDT

We are currently experiencing an issue with Google Cloud Endpoints where the GAPI Javascript client is unable to pass request parameters. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by Tuesday, 2014-09-23 05:00 (all times are in US/Pacific) with current details, and if available an estimated time for resolution.