Google Cloud Service Health

Google Cloud Service Health
Incidents
Issue with Google Cloud SQL

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Cloud SQL, Eventarc, Google Cloud Networking, Operations, Cloud Developer Tools, Cloud Endpoints, Google App Engine, Google Cloud Bigtable, Google Cloud Console, Google Cloud Infrastructure Components, Container Registry, Cloud Load Balancing

Issue with Google Cloud SQL

Incident began at 2021-11-12 00:30 and ended at 2021-11-12 02:14 (all times are US/Pacific).

Date	Time	Description
12 Nov 2021	16:32 PST	We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 12 November 2021 00:30 Incident End: 12 November 2021 02:14 Duration: 1 hour 44 minutes Affected Services and Features: Cloud Developer Tools Cloud Endpoints Cloud Eventarc Google App Engine Google Cloud Bigtable Google Cloud Console Google Cloud Infrastructure Components Google Cloud Networking Google Cloud SQL Operations Regions/Zones: Europe Description: Google’s Front End load balancing service experienced failures resulting in impact to several downstream Google Cloud services in Europe. From preliminary analysis, the root cause of the issue was caused by a new infrastructure feature triggering a latent issue within internal network load balancer code. Customer Impact: Google Cloud Console - Affected customers in Europe were unable to load the console, or experienced timeouts. Container Registry - Affected customers were unable to connect to the service using the API in europe-west1. Cloud Bigtable - Affected customers in Europe experienced unavailable errors and latency for both data and admin operations. Cloud Spanner - Affected customers in Europe experienced unavailable errors and latency for both data and admin operations. Cloud Endpoints - Service was unavailable in europe-west1 and europe-west4. Firebase Messaging - Affected customers experienced issues receiving notifications in asia-southeast1. Cloud SQL - Affected customers experienced probe creation request failures in europe-west1, europe-west4, and europe-west5. Cloud Eventarc - Affected customers were unable to create triggers in europe-west1. App Engine - Affected customers experienced elevated errors in europe-west1. Additional details: The error was caught within 4 minutes by automated safety systems, and further spread was slowed at this point. The issue was fully mitigated approximately 1hr 44m later, when our engineering team completed a rollout to disable the vulnerable code path. The issue will be fully prevented going forward via a root cause fix, which will complete rollout by 12 November 2021 21:00 US/Pacific.
12 Nov 2021	03:46 PST	The issue with Google Cloud infrastructure components has been resolved for all affected products as of Friday, 2021-11-12 03:38 US/Pacific. If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved. We thank you for your patience while we worked on resolving the issue.
12 Nov 2021	02:19 PST	Summary: Issue with Google Cloud SQL Description: We are experiencing an issue with multiple Google Cloud components beginning on Friday, 2021-11-12 00:30 US/Pacific US/Pacific. Our engineering team continues to investigate the issue. For regular status updates, please follow: https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZEZ7, no further updates will be provided here. We apologize to all who are affected by the disruption. Diagnosis: None Workaround: None

Date

Time

Description

12 Nov 2021

16:32 PST

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 12 November 2021 00:30

Incident End: 12 November 2021 02:14

Duration: 1 hour 44 minutes

Affected Services and Features:

Cloud Developer Tools
Cloud Endpoints
Cloud Eventarc
Google App Engine
Google Cloud Bigtable
Google Cloud Console
Google Cloud Infrastructure Components
Google Cloud Networking
Google Cloud SQL
Operations

Regions/Zones: Europe

Description:

Google’s Front End load balancing service experienced failures resulting in impact to several downstream Google Cloud services in Europe. From preliminary analysis, the root cause of the issue was caused by a new infrastructure feature triggering a latent issue within internal network load balancer code.

Customer Impact:

Google Cloud Console - Affected customers in Europe were unable to load the console, or experienced timeouts.
Container Registry - Affected customers were unable to connect to the service using the API in europe-west1.
Cloud Bigtable - Affected customers in Europe experienced unavailable errors and latency for both data and admin operations.
Cloud Spanner - Affected customers in Europe experienced unavailable errors and latency for both data and admin operations.
Cloud Endpoints - Service was unavailable in europe-west1 and europe-west4.
Firebase Messaging - Affected customers experienced issues receiving notifications in asia-southeast1.
Cloud SQL - Affected customers experienced probe creation request failures in europe-west1, europe-west4, and europe-west5.
Cloud Eventarc - Affected customers were unable to create triggers in europe-west1. App Engine - Affected customers experienced elevated errors in europe-west1.

Additional details:

The error was caught within 4 minutes by automated safety systems, and further spread was slowed at this point. The issue was fully mitigated approximately 1hr 44m later, when our engineering team completed a rollout to disable the vulnerable code path. The issue will be fully prevented going forward via a root cause fix, which will complete rollout by 12 November 2021 21:00 US/Pacific.

12 Nov 2021

03:46 PST

The issue with Google Cloud infrastructure components has been resolved for all affected products as of Friday, 2021-11-12 03:38 US/Pacific.

If you have questions or are still impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We thank you for your patience while we worked on resolving the issue.

12 Nov 2021

02:19 PST

Summary: Issue with Google Cloud SQL

Description: We are experiencing an issue with multiple Google Cloud components beginning on Friday, 2021-11-12 00:30 US/Pacific US/Pacific.

Our engineering team continues to investigate the issue. For regular status updates, please follow: https://status.cloud.google.com/incidents/1xkAB1KmLrh5g3v9ZEZ7, no further updates will be provided here.

We apologize to all who are affected by the disruption.

Diagnosis: None

Workaround: None

All times are US/Pacific