Google Cloud Service Health

Google Cloud Service Health
Incidents
us-central1-a,c: Cloud spanner jobs reporting high read latencies. Incident mitigated.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Cloud Spanner

us-central1-a,c: Cloud spanner jobs reporting high read latencies. Incident mitigated.

Incident began at 2021-10-06 13:50 and ended at 2021-10-06 17:42 (all times are US/Pacific).

Date	Time	Description
7 Oct 2021	10:33 PDT	We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 06 October 2021 13:50 Incident End: 06 October 2021 17:42 Duration: 3 hours, 52 minutes Affected Services and Features: Cloud Spanner: Latency Regions/Zones: us-central1 Description: Cloud Spanner experienced elevated p99.9 read latencies in us-central1 in some limited cases for a duration of 3 hours, 52 minutes. From preliminary analysis, the root cause of the issue was a resource contention issue on a small portion of the Spanner frontend servers in us-central1 that affected p99.9 latency for some customers. Customer Impact: A small portion of the regional Cloud Spanner instances in us-central1 experienced increased p99.9 latency for read requests. Multi-regional Cloud Spanner instances (nam6, nam7, nam9, nam10, nam11, nam12, and nam-eur-asia1) where the requests were routed to the impacted frontend servers in us-central1 were affected similarly. Additional details: The issue has been fully mitigated by isolating the impacted frontend servers, and we do not believe there is a risk of recurrence at this time.
6 Oct 2021	18:28 PDT	The issue with Cloud Spanner has been resolved for all affected projects as of Wednesday, 2021-10-06 17:42 US/Pacific. We thank you for your patience while we worked on resolving the issue.
6 Oct 2021	17:51 PDT	Summary: us-central1-a,c: Cloud spanner jobs reporting high read latencies. Investigations ongoing. Description: We are experiencing an issue with high read latencies on Cloud Spanner jobs at us-central1-a,c beginning at Wednesday, 2021-10-06 14:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2021-10-06 18:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: None at this time. Workaround: None at this time.

Date

Time

Description

7 Oct 2021

10:33 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 06 October 2021 13:50

Incident End: 06 October 2021 17:42

Duration: 3 hours, 52 minutes

Affected Services and Features:

Cloud Spanner: Latency

Regions/Zones: us-central1

Description:

Cloud Spanner experienced elevated p99.9 read latencies in us-central1 in some limited cases for a duration of 3 hours, 52 minutes. From preliminary analysis, the root cause of the issue was a resource contention issue on a small portion of the Spanner frontend servers in us-central1 that affected p99.9 latency for some customers.

Customer Impact:

A small portion of the regional Cloud Spanner instances in us-central1 experienced increased p99.9 latency for read requests. Multi-regional Cloud Spanner instances (nam6, nam7, nam9, nam10, nam11, nam12, and nam-eur-asia1) where the requests were routed to the impacted frontend servers in us-central1 were affected similarly.

Additional details:

The issue has been fully mitigated by isolating the impacted frontend servers, and we do not believe there is a risk of recurrence at this time.

6 Oct 2021

18:28 PDT

The issue with Cloud Spanner has been resolved for all affected projects as of Wednesday, 2021-10-06 17:42 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

6 Oct 2021

17:51 PDT

Summary: us-central1-a,c: Cloud spanner jobs reporting high read latencies. Investigations ongoing.

Description: We are experiencing an issue with high read latencies on Cloud Spanner jobs at us-central1-a,c beginning at Wednesday, 2021-10-06 14:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2021-10-06 18:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: None at this time.

Workaround: None at this time.

All times are US/Pacific