Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google App Engine

We're investigating an issue with Google App Engine UrlFetch service beginning at approximately Wednesday, 2014-06-11 18:04 (US/Pacific Time). We will provide more information shortly.

Incident began at 2014-06-11 18:03 and ended at 2014-06-11 20:41 (all times are US/Pacific).

Date Time Description
9 Jul 2014 09:49 PDT

SUMMARY: On Wednesday 11 June 2014, Google App Engine applications experienced elevated errors from the URL Fetch API for a duration of 158 minutes. If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT: On Wednesday 11 June 2014 from 18:03 to 20:41 US/Pacific, App Engine applications had a 36% error rate from the URL Fetch API due to this incident. In addition, the calls with errors often failed after hitting the call deadline and hence had higher latency than successful calls. Applications in all regions were affected. Calls to Google APIs (except Cloud Storage) and appspot.com URLs were not affected.

ROOT CAUSE: The incident occurred after a configuration change which caused a portion of outbound IP addresses to become unusable. Any requests routed through those IP addresses were impacted by the issue.

REMEDIATION AND PREVENTION: Google engineers rolled back the configuration change to resolve the issue. Our monitoring detected the problem but did not trigger an alert for our engineering team. We fixed our alerting to immediately notify on high error rates from URL Fetch API calls. We are fixing the configuration change system so that IP addresses cannot be removed unless they are unused by clients.
9 Jul 2014 09:48 PDT

The problem with Google App Engine UrlFetch service was resolved as of 21:56 Pacific. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.
9 Jul 2014 09:47 PDT

We have identified the issue affecting Google App Engine UrlFetch service and latency is returning to normal. We will provide another update by 22:00 Pacific.
9 Jul 2014 09:47 PDT

We are currently experiencing a network issue with Google App Engine UrlFetch API. For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 21:00 Pacific Time with current details.
9 Jul 2014 09:41 PDT

We're investigating an issue with Google App Engine UrlFetch service beginning at approximately Wednesday, 2014-06-11 18:04 (US/Pacific Time). We will provide more information shortly.