Service Health
Incident affecting Google App Engine
We are currently experiencing an issue with Google App Engine Shared Memcache and some applications are receiving Out of Quota errors. For everyone who is affected, we apologize - we know you count on Google to work for you and we're working hard to restore normal operation. We will provide an update by Wednesday, 2014-08-13 14:00 (all times are in US/Pacific) with current details, and if available an estimated time for resolution.
Incident began at 2014-08-13 10:06 and ended at 2014-08-13 12:45 (all times are US/Pacific).
Date | Time | Description | |
---|---|---|---|
| 11 Sep 2014 | 11:08 PDT | SUMMARY: On Wednesday, 13 August 2014, some Google App Engine applications using Shared Memcache received “Quota Exceeded” errors for a duration of 2 hours and 39 minutes. If your service or application was affected, we apologize. We will implement preventative measures to avoid these types of incidents in the future. DETAILED DESCRIPTION OF IMPACT: On Wednesday, 13 August 2014 from 10:06 to 12:45 US/Pacific, 6.1% of applications using Shared Memcache received Quota Exceeded Errors. In aggregate, 2.1% of Memcache API calls from affected applications failed. The issue only affected applications with heavy use of Shared Memcache. Applications using Dedicated Memcache were not affected. Tokens received from the App Identity API are stored in Memcache. Some applications increased the number of tokens requested from the App Identity API because the tokens were no longer cached. As a result, 1% of applications using the App Identity API exceeded the quota for token requests. ROOT CAUSE: The incident was caused by a change to Shared Memcache to ensure that each application gets fair usage. The change caused some applications that use a high number of Memcache Compute Units to exceed quotas. REMEDIATION AND PREVENTION: The issue was resolved by rolling back the Memcache quota change. In addition, we reset the quotas for the App Identity API for all applications affected so that these API calls would resume working. To prevent future issues similar to this, we will increase the quota for token requests for all applications and improve the caching of the tokens. We will also improve our monitoring to detect quota denials for token requests and we will create a test to validate that changes do not impact Memcache quota. |
| 14 Aug 2014 | 13:43 PDT | The problem with Google App Engine Shared Memcache was resolved as of Wednesday, 2014-08-13 12:45 (all times are in US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. |
| 14 Aug 2014 | 13:42 PDT | We are currently experiencing an issue with Google App Engine Shared Memcache and some applications are receiving Out of Quota errors. For everyone who is affected, we apologize - we know you count on Google to work for you and we're working hard to restore normal operation. We will provide an update by Wednesday, 2014-08-13 14:00 (all times are in US/Pacific) with current details, and if available an estimated time for resolution. |
- All times are US/Pacific