Service Health

SUMMARY:

On Friday, June 26 2015 13:04 PDT, connections to Cloud SQL instances in one datacenter in the United States failed for a duration of 19 minutes. We apologise if you were affected by this issue. Our engineers have completed a postmortem analysis of this incident and are performing remediation tasks to prevent similar issues in the future.

DETAILED DESCRIPTION OF IMPACT:

On Friday, June 26 2015 from 13:04 to 13:23 PDT, traffic to affected Cloud SQL instances in United States was impacted. Existing connections were dropped and new connections could not be established. Approximately 4% of Cloud SQL instances in the United States also were also rebooted.

ROOT CAUSE:

A manual procedure to reprogram the power supply of a router as part of a maintenance routine didn’t have the desired effect and caused the router to restart in an undesired manner. The team performing the routine detected the issue immediately and was able to restore the system to a stable state within 20 minutes.

REMEDIATION AND PREVENTION:

We have identified areas of improvement needed to prevent similar issues in the future. We have taken measures to prevent issues with the same root cause to reappear in the future. We are also adding additional monitoring metrics to alert for failures at this level earlier and improve cross-team communication for planned maintenance events.

The problem with Google Cloud SQL is resolved as of shortly after 13:28 US/Pacific. We are sorry for any issues this may have caused to you or your users and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are constantly working to improve the reliability of our systems.

We're investigating an issue with connectivity to Google Cloud SQL beginning at 13:25 US/Pacific. We'll provide an new update in the next 30 min.