Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Memorystore, Google Cloud Networking, AlloyDB for PostgreSQL, Google Cloud SQL, Cloud Data Fusion, Google Cloud Composer, Backup and DR, Memorystore for Memcached, Memorystore for Redis

We are experiencing an issue with Cloud Memorystore, AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Google Cloud Composer, Google Cloud SQL. All products are mitigated.

Incident began at 2023-11-20 22:29 and ended at 2023-11-21 03:07 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Berlin (europe-west10)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Dammam (me-central2)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
22 Nov 2023 09:59 PST

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 20 November 2023 22:29

Incident End: 21 November 2023 03:07

Duration: 4 hours, 38 minutes

Affected Services and Features:

  • Cloud SQL
  • AlloyDB
  • Cloud Data Fusion
  • Cloud Composer
  • Cloud Backup and DR
  • Cloud Memorystore

Regions/Zones: Global

Description:

Multiple Google Cloud products experienced elevated latency and error rates for a duration of 4 hours, 38 minutes in the launch of new producer service instances (like Cloud SQL, etc.) This did not impact existing instances.

From preliminary analysis, the root cause of the issue was an unexpected load that caused a surge in API requests to Service Networking, which led to increased latency for these requests. Requests to create Service Networking from consumers to create connections, or to producer services to launch service instances, which in turn relies on Service Networking, slowed down.

The service networking consumer-side API experienced up to 10 times the normal levels of create connection requests, and up to 3.5 times the normal levels of add subnetwork requests. While our engineering team was working on troubleshooting the issue, the surge started decreasing at around 01:25 US/Pacific and once the requests were back to normal, the service recovered at 03:07 US/Pacific

Customer Impact:

Cloud SQL:

  • CREATE operations that require a subnet creation in the tenant project were slow, and took up to one hour to complete.

AlloyDB:

  • Instance creations failed with timeouts due to service networking creation taking more than 10 minutes.

Cloud Data Fusion:

  • Private Data Fusion instance creation failed or timed out due to Service Networking and Cloud SQL creation.

Cloud Composer:

  • Environment creations fail in multiple regions due to failed Cloud SQL creations.

Cloud Backup and DR:

  • ManagementServer creations failed globally for the duration of the outage. .

Cloud Memorystore:

  • ServiceNetwork instance creation experienced elevated latency and failures.
21 Nov 2023 03:12 PST

Google Cloud SQL

  • Impact: CREATE operations that require a subnet creation in the tenant project are slow, and can take up to one hour to complete.
  • Mitigated as of 2023-11-21 02:12:09 US/Pacific

AlloyDB

  • Impact: Instance creations fail with timeouts due to service networking creation taking more than 10 minutes.
  • Mitigated as of 2023-11-21 02:30:31 US/Pacific

Cloud Data Fusion

  • Impact: Private Data Fusion instance creation failing/timing out due to Service Networking and Cloud SQL creation.
  • Mitigated as of 2023-11-21 02:28:10 US/Pacific

Cloud Composer

  • Impact: Environment creations fail in multiple regions due to failed Cloud SQL creations.
  • Mitigated as of 2023-11-21 02:28:23 US/Pacific

Cloud Backup and DR

  • Impact: All ManagementServer creations are failing globally.
  • Mitigated as of 2023-11-21 02:37:04 US/Pacific

MemoryStore

  • Impact: ServiceNetwork operations may be slow.
  • Mitigated as of 2023-11-21 02:38:00 US/Pacific

We thank you for your patience while we worked on resolving the issue.
21 Nov 2023 02:47 PST

Summary: We are experiencing an issue with Cloud Memorystore. AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Google Cloud Composer, Google Cloud SQL are mitigated

Description:

Mitigated:

Google Cloud SQL Impact: CREATE operations that require a subnet creation in the tenant project are slow, and can take up to one hour to complete.

Mitigated as of 2023-11-21 02:12:09 US/Pacific

AlloyDB

Impact: Instance creations fail with timeouts due to service networking creation taking more than 10 minutes.

Mitigated as of 2023-11-21 02:30:31 US/Pacific

Cloud Data Fusion

Impact: Private Data Fusion instance creation failing/timing out due to Service Networking and Cloud SQL creation.

Mitigated as of 2023-11-21 02:28:10 US/Pacific

Cloud Composer

Impact: Environment creations fail in multiple regions due to failed Cloud SQL creations.

Mitigated as of 2023-11-21 02:28:23 US/Pacific

Cloud Backup and DR

Impact: All ManagementServer creations are failing globally.

Mitigated as of 2023-11-21 02:37:04 US/Pacific

Ongoing impact:

MemoryStore

Impact: ServiceNetwork operations may be slow.

We will provide an update by Tuesday, 2023-11-21 03:15 US/Pacific with current details.

Diagnosis: Creation of resources related to Cloud Service Networking may experience failures or delays.

Workaround: Attempt retry or await for the operation to complete.
21 Nov 2023 02:13 PST

Summary: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL

Description: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-11-21 04:27 US/Pacific with current details.

Diagnosis: Creation of resources related to Cloud Service Networking may experience failures or delays.

Workaround: Attempt retry or await for the operation to complete.
21 Nov 2023 02:10 PST

Summary: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL

Description: We are experiencing an issue with AlloyDB for PostgreSQL, Backup and DR, Cloud Data Fusion, Cloud Memorystore, Google Cloud Composer, Google Cloud SQL beginning at Monday, 2023-11-20 22:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-11-21 02:40 US/Pacific with current details.

Diagnosis: None at this time.

Workaround: None at this time.