Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud SQL

The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered

Incident began at 2021-01-29 09:39 and ended at 2021-01-29 19:45 (all times are US/Pacific).

Date Time Description
29 Jan 2021 19:45 PST

The issue with Cloud SQL API instance mutations and SQL Proxy operations has been resolved for all affected users as of Friday, 2021-01-29 18:45 US/Pacific.

All operations including instance restart and delete are healthy.

We thank you for your patience while we worked on resolving the issue.

29 Jan 2021 18:59 PST

Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific

Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. The product team has identified the root cause of the issue and has begun implementing a fix.

The product team has started rolling out the potential fix. We are monitoring the effects of this rollout to determine if the fix has solved the issue.

We will provide an update by Friday, 2021-01-29 20:00 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.

Workaround: None at this time. We believe other regions are not impacted at this time.

29 Jan 2021 16:28 PST

Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific

Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. The product team has identified the root cause of the issue and has begun implementing a fix. We expect that the Cloud SQL API will remain in the current partially resolved state for several hours until the fix is fully verified and rolled out.

We will provide an update by Friday, 2021-01-29 19:00 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.

Workaround: None at this time. We believe other regions are not impacted at this time.

29 Jan 2021 15:31 PST

Description: The issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1 has partially recovered. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific. Partial recovery occurred at 14:54 US/Pacific

Our engineering team has rolled out a mitigation and partially resolved the issue. We believe most of the service has recovered. At this time we believe only instances.delete and instances.restart continue to be impacted. Some operations such as instances.patch which require an instance.restart may also be impacted. We are working on further mitigation to bring the service back to a nominal operating state.

We will provide an update by Friday, 2021-01-29 16:30 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.

Workaround: None at this time. We believe other regions are not impacted at this time.

29 Jan 2021 14:38 PST

Description: We are experiencing an issue with Cloud SQL API instance mutations and SQL Proxy operations in us-central1. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific.

Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected. New connections without Cloud SQL Proxy are also unaffected.

The product team has identified a potential mitigation and is working to implement it. We believe the change will be finished rolling out by 15:00 US/Pacific, we will be continuously monitoring the service to evaluate the effectiveness of the mitigation.

We will provide an update by Friday, 2021-01-29 15:30 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.

Workaround: None at this time. We believe other regions are not impacted at this time.

29 Jan 2021 13:59 PST

Description: We are experiencing an issue with Cloud SQL API instance mutations in us-central1. We believe impact began at 08:50 US/Pacific with the error rate increasing starting at 09:25 US/Pacific.

Our engineering team continues to investigate the issue. Instance mutation operations (such as instances.create or instances.delete) as well as requests for certificates (sslCerts.createEphemeral) when using Cloud SQL Proxy are impacted. Existing connections to Cloud SQL instances should be unaffected.

The product team has identified a potential mitigation and is working to implement it. As part of the mitigation, all requests to instances delete and instances restart will receive an error.

We will provide an update by Friday, 2021-01-29 15:00 US/Pacific with current details.

Diagnosis: Customers impacted by this issue may see increased latency or errors when performing Instance mutation requests, such as create, delete and update. Connection to Cloud SQL instances may be disrupted if a new ephemeral cert is required by the Cloud SQL Proxy. Errors returned may be 502 or 503.

Workaround: None at this time. We believe other regions are not impacted at this time.

29 Jan 2021 13:41 PST

Description: We are experiencing an issue with Cloud SQL.

Our engineering team continues to investigate the issue. New instance creation, other instance operations, and SQL queries to already existing instances may be failing. We are troubleshooting the error at this time.

We will provide an update by Friday, 2021-01-29 16:00 US/Pacific with current details.

Diagnosis: Instance creation, operations, and queries in us-central1 region may result in 502 (backend timeout) or 503 (service unavailable) errors.

Customers may also experience Cloud SQL Proxy connectivity related issues, due in part to this incident.

Workaround: None at this time.

29 Jan 2021 13:01 PST

Description: We are experiencing an issue with Cloud SQL.

Our engineering team continues to investigate the issue. New instance creation, other instance operations, and SQL queries to already existing instances may be failing.

We will provide an update by Friday, 2021-01-29 13:00 US/Pacific with current details.

Diagnosis: Instance creation, operations, and queries in us-central1 region may result in 502 (backend timeout) or 503 (service unavailable) errors.

Workaround: None at this time.

29 Jan 2021 12:52 PST

Description: We are experiencing an issue with Cloud SQL.

Our engineering team continues to investigate the issue. New instance creation, other instance operations, and SQL queries to already existing instances may be failing. We are troubleshooting the error at this time.

We will provide an update by Friday, 2021-01-29 16:00 US/Pacific with current details.

Diagnosis: Instance creation, operations, and queries in us-central1 region may result in 502 (backend timeout) or 503 (service unavailable) errors.

Workaround: None at this time.