Google Cloud Service Health

Google Cloud Service Health
Incidents
We are currently experiencing an issue with blobstore file uploads either failing or returning non-HTTP 200 return codes. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by Wednesday, 2014-10-22 14:20 PDT.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google App Engine

We are currently experiencing an issue with blobstore file uploads either failing or returning non-HTTP 200 return codes. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by Wednesday, 2014-10-22 14:20 PDT.

Incident began at 2014-10-22 10:45 and ended at 2014-10-22 14:22 (all times are US/Pacific).

	Date	Time	Description
	28 Oct 2014	15:12 PDT	SUMMARY: On Wednesday 22 October 2014 for a duration of 3 hours 48 minutes, 45.2% of uploads to Google App Engine received incorrect response codes. Impacted customers can remove orphaned uploads from the Google Developers Console. If uploads to your App Engine application were impacted by this issue, we apologize. This is not the level of reliability we strive to offer and we are taking immediate steps to improve. DETAILED DESCRIPTION OF IMPACT: On Wednesday 22 October 2014 from 10:35 to 14:23 PDT, App Engine Blobstore API uploads that had authentication-related fields in their HTTP request headers incorrectly returned HTTP 303, 401 or 403 response codes. The App Engine upload infrastructure successfully processed the upload requests, but notifications to customer applications did not contain the fields from the original request. ROOT CAUSE: Google Engineers deployed a configuration change on Wednesday 22 October 2014 at 10:35. This change disabled the forwarding of some authentication-related header fields in the HTTP requests for uploads handled by the Blobstore API. Requests to POST handlers that required one of these fields failed with HTTP 303, 401 or 403 response codes. REMEDIATION AND PREVENTION: At 12:39 Google Cloud Support received multiple customer tickets and informed Google Engineers of an issue with uploads to App Engine applications using the Blobstore API. We identified the root cause of the issue at 13:29 and began rolling back the change at 13:50. The rollback was completed at 14:23. Google engineers are improving App Engine’s release test suite to prevent this issue from recurring.
	22 Oct 2014	14:47 PDT	The problem with Google App Engine blobstore upload was resolved as of October 2014-10-22 14:22 (US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

Date

Time

Description

28 Oct 2014

15:12 PDT

SUMMARY: On Wednesday 22 October 2014 for a duration of 3 hours 48 minutes, 45.2% of uploads to Google App Engine received incorrect response codes. Impacted customers can remove orphaned uploads from the Google Developers Console. If uploads to your App Engine application were impacted by this issue, we apologize. This is not the level of reliability we strive to offer and we are taking immediate steps to improve.

DETAILED DESCRIPTION OF IMPACT: On Wednesday 22 October 2014 from 10:35 to 14:23 PDT, App Engine Blobstore API uploads that had authentication-related fields in their HTTP request headers incorrectly returned HTTP 303, 401 or 403 response codes. The App Engine upload infrastructure successfully processed the upload requests, but notifications to customer applications did not contain the fields from the original request.

ROOT CAUSE: Google Engineers deployed a configuration change on Wednesday 22 October 2014 at 10:35. This change disabled the forwarding of some authentication-related header fields in the HTTP requests for uploads handled by the Blobstore API. Requests to POST handlers that required one of these fields failed with HTTP 303, 401 or 403 response codes.

REMEDIATION AND PREVENTION: At 12:39 Google Cloud Support received multiple customer tickets and informed Google Engineers of an issue with uploads to App Engine applications using the Blobstore API. We identified the root cause of the issue at 13:29 and began rolling back the change at 13:50. The rollback was completed at 14:23.

Google engineers are improving App Engine’s release test suite to prevent this issue from recurring.

22 Oct 2014

14:47 PDT

The problem with Google App Engine blobstore upload was resolved as of October 2014-10-22 14:22 (US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

All times are US/Pacific