SUMMARY:

On Saturday 2 May 2015, the Logs Viewer in the Google Cloud Developers Console was unavailable for a duration of 7 hours 10 minutes. We apologize to users who were affected by this issue — this is not the level of quality and reliability we strive to offer you, and we are working to improve the availability of the affected components.

DETAILED DESCRIPTION OF IMPACT:

On Saturday 2 May 2015 from 11:10 to 18:20 PDT, the Logs Viewer in the Google Cloud Developers Console was unavailable. Users that tried to access the Logs Viewer saw a loading animation which continued indefinitely.

Log recording was not affected and messages logged during the incident were available when the Logs Viewer service was restored. Additionally, App Engine logs were still viewable in the App Engine console under appengine.google.com for the duration of the incident.

ROOT CAUSE:

At 11:10 Google engineers deployed a networking configuration change which prevented the Logs Viewer from contacting a backend component of the Cloud logging subsystem. This change affected requests that were issued from outside the Google network.

The Cloud Developers Console has continuous monitoring to detect elevated error rates, but the monitoring submits requests from within the Google network and therefore reported that the Logs Viewer was working correctly, with the result that Google engineers did not become aware of the issue until it was reported by Cloud Platform customers.

REMEDIATION AND PREVENTION:

Google engineers identified the problematic change at 17:52 and immediately started a rollback to resolve the issue. This was complete at 18:20.

To ensure prompt detection of any similar issues in future, Google engineers are extending the monitoring and testing of the Cloud Developer Console to include probes from outside Google's network.

The problem with the Developers Console should be resolved as of 18:22 PST. We apologize for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We are still investigating the issue with the Developers Console where the Logs page is failing to load. Impacted users may still view logs in the Admin Console (https://appengine.google.com). We will provide another status update by 19:00 PDT.

We are currently experiencing an issue with the Developers Console and some users are unable to view the Logs page. Impacted users may still view logs in the Admin Console (https://appengine.google.com). For everyone who is affected, we apologize for any inconvenience you may be experiencing.