Google Cloud Status

This page provides status information on the services that are part of the Google Cloud Platform. Check back here to view the current status of the services listed below. For additional information on these services, please visit cloud.google.com.

Google App Engine Incident #15013

AppEngine deployments failing for some customers

Incident began at 2015-04-28 12:05 and ended at 2015-04-28 17:00 (all times are US/Pacific).

Date Time Description
Apr 30, 2015 04:24

SUMMARY:

For 4 hours and 37 minutes on Tuesday 28 April 2015, 45% of Google App Engine deployments failed. If you were trying to deploy new versions of your app during this time, we apologize for the extended outage and the delay on public announcement.

DETAILED DESCRIPTION OF IMPACT:

On Tuesday 28 April 2015 from 12:05 to 16:42 PDT, 45% of App Engine deployments experienced delays that resulted in the operation timing out. A small number of deployments were exceptionally slow but succeeded. There was no impact to already-deployed applications, nor to applications after a successful deployment.

Affected deployments showed the messages, "Checking if deployment succeeded" and "Will check again in 60 seconds" for long periods. Deployments that timed out showed, "Version still not ready to serve, aborting" and "An unexpected error occurred. Aborting".

ROOT CAUSE:

Google engineers ran an internal software upgrade that has a side-effect of slowing some deployments for a period of up to one hour. The upgrade failed in its first run and had to be restarted, extending the duration of its impact. Most affected deployments were delayed beyond their timeout period, causing the deployment to be aborted.

REMEDIATION AND PREVENTION:

Google engineers are improving the deployment infrastructure to minimize the impact of routine maintenance. Google will also make app deployment speed and reliability more transparent to customers.

In case of future deployment delays, Google is updating procedures to ensure that public notifications are issued in a timely fashion.

SUMMARY:

For 4 hours and 37 minutes on Tuesday 28 April 2015, 45% of Google App Engine deployments failed. If you were trying to deploy new versions of your app during this time, we apologize for the extended outage and the delay on public announcement.

DETAILED DESCRIPTION OF IMPACT:

On Tuesday 28 April 2015 from 12:05 to 16:42 PDT, 45% of App Engine deployments experienced delays that resulted in the operation timing out. A small number of deployments were exceptionally slow but succeeded. There was no impact to already-deployed applications, nor to applications after a successful deployment.

Affected deployments showed the messages, "Checking if deployment succeeded" and "Will check again in 60 seconds" for long periods. Deployments that timed out showed, "Version still not ready to serve, aborting" and "An unexpected error occurred. Aborting".

ROOT CAUSE:

Google engineers ran an internal software upgrade that has a side-effect of slowing some deployments for a period of up to one hour. The upgrade failed in its first run and had to be restarted, extending the duration of its impact. Most affected deployments were delayed beyond their timeout period, causing the deployment to be aborted.

REMEDIATION AND PREVENTION:

Google engineers are improving the deployment infrastructure to minimize the impact of routine maintenance. Google will also make app deployment speed and reliability more transparent to customers.

In case of future deployment delays, Google is updating procedures to ensure that public notifications are issued in a timely fashion.
Apr 28, 2015 18:57

We have experienced a problem with Google AppEngine from 2015-04-28 12:05 US/Pacific to 2015-04-28 17:00 US/Pacific, which manifested itself as follows: unexpected errors or high latency deploying AppEngine applications. Other AppEngine components were unaffected.

The issue was fully resolved as of Tuesday, 2015-04-28 17:00 US/Pacific. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We have experienced a problem with Google AppEngine from 2015-04-28 12:05 US/Pacific to 2015-04-28 17:00 US/Pacific, which manifested itself as follows: unexpected errors or high latency deploying AppEngine applications. Other AppEngine components were unaffected.

The issue was fully resolved as of Tuesday, 2015-04-28 17:00 US/Pacific. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. We will provide a more detailed analysis of this incident once we have completed our internal investigation.