SUMMARY:

On Friday 8 May 2015, users were unable to reliably create projects in the Google Cloud Developers Console for 6 hours 58 minutes in aggregate. We treat the reliability and availability of the Developers Console with the highest priority and we apologize to any users who were unable to create new projects during that time.

DETAILED DESCRIPTION OF IMPACT:

On Friday 8 May 2015, from 04:53 to 07:25 and from 09:35 to 12:31 PDT, users were unable to reliably create new projects using the Developers Console. Specifically, up to 90% of project creation requests failed and returned a "Backend Error" message to the user. Existing projects were not affected by this incident.

ROOT CAUSE AND REMEDIATION:

A configuration change in the access management system of Developers Console triggered an existing software bug in a previously unused code path. As the change was rolled out, the access management system could no longer correctly verify user’s permissions, resulting in an error response.

Google engineers were automatically alerted to the issue at 05:04. At 05:22 a preliminary source of the issue was identified. At 09:32 the root cause was confirmed and a solution was developed. An expedited roll-out of the fix commenced at 11:41 and completed at 12:31.

The Developer Console issue that prevented users from creating projects was resolved as of 2015-05-08 13:15 US/Pacific. We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

We are continuing to work toward a resolution of the Developers Console issue that is preventing users from creating projects. We will provide another status update no later than 13:25 pacific time.

We are currently experiencing an issue with the Developers Console where users are unable to create new projects. We are currently working toward a resolution. For everyone who is affected, we apologize for any inconvenience you may be experiencing.