Service Health

SUMMARY: On Tuesday 20 May 2014, some Google App Engine (GAE) applications incorrectly served 503 errors. If your service or application was affected, we apologize - this is not the level of reliability and performance we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT: On Tuesday 20 March 2014, 0.83 % of applications with billing enabled served 503 errors from 06:45 AM to 06:48 AM US/Pacific. The error rate aggregated across affected applications during this period was 26 %. 0.12 % of applications were affected for longer than 3 minutes. No application was affected for longer than 2 hours.

ROOT CAUSE: The incident occurred during a configuration change to App Engine’s quota checking system. This caused a type of error during quota checks by some applications that was incorrectly interpreted by App Engine as a quota denial.

REMEDIATION AND PREVENTION: The issue was resolved when the configuration change completed. Our engineers have determined that this type of configuration change to the quota checking system will not recur. We are improving our monitoring to provide better data on the root cause of quota checking issues.

The problem with Google App Engine has been resolved by now. We apologize for the inconvenience and thank you for your patience and continued support. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

Normal serving state has already been restored for most applications, and we expect a resolution for all apps in the near future. Please note that this time frame is an estimate and may change. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide the next update by Tuesday, 2014-05-20 10:00 (Pacific Time) with further details.

Normal serving state has already been restored for some applications, and we expect a resolution for all apps in the near future. Please note that this time frame is an estimate and may change. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide the next update by Tuesday, 2014-05-20 09:00 (Pacific Time) with further details.

We are currently experiencing an issue with Google App Engine and some applications are not being served with a 503 "Over Quota" response. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide the next update by Tuesday, 2014-05-20 08:45 (Pacific Time) with further details.

We're investigating an issue with Google App Engine serving beginning at Tuesday, 2014-05-20 06:45 (all times are in US/Pacific). We will provide more information shortly.