Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking

High control-plane latency in southamerica-east1

Incident began at 2023-02-02 06:20 and ended at 2023-02-02 09:42 (all times are US/Pacific).

Previously affected location(s)

São Paulo (southamerica-east1)

Date Time Description
2 Feb 2023 13:43 PST

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 02 February 2023 06:20

Incident End: 02 February 2023 09:42

Duration: 3 hours, 22 minutes

Affected Services and Features:

Google Cloud Networking - Load Balancing Cloud SQL MemoryStore for Redis

Regions/Zones: southamerica-east1

Description:

Google Cloud Networking experienced elevated control plane latency and timeouts of new operations for customers using Cloud Networking in southamerica-east1 for a duration of 3 hours, 22 minutes. From preliminary analysis, the root cause was identified due to a control plane issue which had an impact on access for downstream services. The issue was resolved after restarting a backend component.

Engineering team is also reviewing the control plane jobs and rolling back any recent changes that could contribute to higher latencies in instance creation/update operations.

Customer Impact:

Google Cloud Networking

Customers experienced elevated latency and ~50 of API requests for Cloud Networking resources timed out.

Cloud SQL

Affected customers were unable to create CloudSQL instances. Affected customers experienced elevated CREATE|FAILOVER operation latencies.

MemoryStore for Redis

Some customers experienced CreateInstance timeouts.
2 Feb 2023 09:43 PST

The issue with Google Cloud Networking has been resolved for all affected projects as of Thursday, 2023-02-02 09:32 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
2 Feb 2023 09:20 PST

Summary: High control-plane latency in southamerica-east1

Description: We are experiencing an issue with Google Cloud Networking beginning on Thursday, 2023-02-02 06:20 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2023-02-02 09:50 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Users may experience high latency and potential timeouts of new operations.

Workaround: None at this time
2 Feb 2023 08:23 PST

Summary: High control-plane latency in southamerica-east1

Description: We are experiencing an issue with Google Cloud Networking beginning on Thursday, 2023-02-02 06:20 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2023-02-02 09:20 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Users may experience high latency and potential timeouts of new operations.

Workaround: None at this time