Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google App Engine

We experienced an issue with Google App Engine app deployments starting at 2014-06-03 08:40 AM US/Pacfic. Some users received timeouts when attempting to deploy their applications. The problem was resolved as of 2014-06-03 10:54 AM US/Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

Incident began at 2014-06-03 08:40 and ended at 2014-06-03 14:20 (all times are US/Pacific).

Date Time Description
9 Jul 2014 10:01 PDT

SUMMARY: On Tuesday 3 June 2014, some administrators of Google App Engine applications were unable to deploy new versions of their applications for a duration of 204 minutes. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT: On Tuesday 3 June 2014 from 08:40 AM to 10:54 AM and from 13:10 PM to 14:20 PM US/Pacific, some administrators of App Engine applications were unable to deploy new versions. In addition, some administrators experienced elevated latency for successful deployments. The overall rate of deployment failures was 27% during the incident.

ROOT CAUSE: The incident was caused by the update of App Engine’s traffic routers to version 1.9.6 which caused an increase in load on the service that tracks deployments, leading to a higher error rate.

REMEDIATION AND PREVENTION: Our engineers halted the software update at 10:34 AM. The error rate and latency returned to normal levels by 10:54 AM, but began to rise at 13:10 PM. We began a rollback of the partial update at 13:25 PM which completed at 13:57 PM. Error rates and latency returned to normal at 14:20 PM. Our engineering team is currently working on fixing the underlying issue before we resume the update to version 1.9.6. We have identified failures in monitoring and testing which we will address after the issue is resolved.
9 Jul 2014 09:59 PDT

The problem with Google App Engine deployment was resolved as of 2014-06-03 14:20 PM US/Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.
9 Jul 2014 09:59 PDT

We are currently re-experiencing the issue with Google App Engine deployment we've resolved at 2014-06-03 10:54 AM US/Pacific. It started again at 2014-06-03 13:10 PM US/Pacific and some users are receiving timeouts when attempting to deploy their applications. For everyone who is affected, we apologize - we know you count on Google to work for you and we're working hard to restore normal operation. We will provide an update by 2014-06-03 15:00 PM US/Pacific with current details, and if available an estimated time for resolution.
9 Jul 2014 09:58 PDT

We experienced an issue with Google App Engine app deployments starting at 2014-06-03 08:40 AM US/Pacfic. Some users received timeouts when attempting to deploy their applications. The problem was resolved as of 2014-06-03 10:54 AM US/Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.