SUMMARY:

On Monday, 27 July 2015, the Google Developers Console was unavailable to all users for a duration of 41 minutes. We apologize for the inconvenience and any impact on your operations that this may have caused. We are urgently working to implement preventative measures to ensure similar incidents do not occur in the future.

DETAILED DESCRIPTION OF IMPACT:

On Monday, 27th July 2015 from 13:21 to 14:02 PDT, the Google Developers Console was unavailable to users. All requests tohttps://console.developers.google.comreturned a 404 "Not Found" response.

Existing Cloud Platform resources such as Compute Engine instances or App Engine applications were not affected. All Google Cloud Platform APIs remained fully functional, allowing most Cloud Platform resources to be managed through the Google Cloud SDK and other API-based tools until Console access was restored.

ROOT CAUSE:

At 13:21, while reviewing the production status of the Developer Console, a Google engineer inadvertently disabled the production instance of the console. The engineer immediately recognised the error and began remediating the problem, but the configuration change had also engaged a security mechanism which restricted the application to the Google corporate network. This mechanism was identified and disengaged at 14:01, which restored public access to the Console.

REMEDIATION AND PREVENTION:

To prevent similar incidents, Google Engineers are currently adding safeguards to make it harder to change application settings by mistake, implementing external monitoring to detect errors outside of the Google network, and creating alerts based on serving errors from the Developers Console.

The issue with Developers Console should be resolved for all affected users as of 14:02 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are experiencing an issue with the Developers Console beginning at Monday, 2015-07-27 13:21 US/Pacific.

Current data indicates that all users are affected by this issue.

For everyone who is affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 16:15 US/Pacific with current details.

We are currently investigating an issue where some users are getting a 404 error when accessing the Developers Console. We will update the status of this issue within 15 minutes