Service Health

SUMMARY:

On Tuesday 20 January 2015, for a duration of 5 hours 2 minutes, Google App Engine applications using the Channel API experienced a 4.6% error rate in creating new channels. We take this incident very seriously and apologize for any impact it had on your service or application. We have made immediate changes to prevent this issue from recurring.

DETAILED DESCRIPTION OF IMPACT:

On Tuesday 20 January 2015 from 10:03 to 15:05 PST, some clients were unable to establish new Channel API sessions. Clients affected by this issue were unable to load the Channel API Javascript library at /_ah/channel/jsapi, and received HTTP 404 “Not Found” responses due to an incorrect redirect. Overall, 4.6% of requests to load the relevant client library were affected.

ROOT CAUSE:

Google engineers began to roll out a new configuration for the service that handles client connections for the Channel API. This new configuration contained an error that caused requests for the Google App Engine Channel API JavaScript client library to redirect to an erroneous location, resulting in an HTTP 404 “Not Found” response.

REMEDIATION AND PREVENTION:

Google engineers were alerted to the issue via an external report at 11:49 PST. The engineers identified the configuration changes as the cause of the issue at 13:05. To resolve this issue, the engineers began a rollback of the configuration change at 13:07 which completed at 15:05.

To prevent this type of incident from recurring, Google engineers are adding more thorough monitoring and testing to the configuration deployment process to detect errors of this nature. Google engineers are also tuning the existing monitoring and alerting system to facilitate quicker identification and resolution of elevated error rates.

The problem with Google App Engine Channel API was resolved as of Tuesday, 2015-01-20 04:10 PM US/Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

We are currently experiencing an issue with Google App Engine Channel API and some clients are getting 404 errors while fetching /_ah/channel/jsapi. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by Tuesday, 2014-01-20 04:00 PM US/Pacific with current details.

We're investigating an issue with Google App Engine Channel API beginning at Tuesday, 2014-01-20 12:10 US/Pacific. We will provide more information shortly.