Incident affecting Cloud Spanner
us-central1-a,c: Cloud spanner jobs reporting high read latencies. Incident mitigated.
Incident began at 2021-10-06 13:50 and ended at 2021-10-06 17:42 (all times are US/Pacific).
| ||7 Oct 2021||10:33 PDT|| |
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support
(All Times US/Pacific)
Incident Start: 06 October 2021 13:50
Incident End: 06 October 2021 17:42
Duration: 3 hours, 52 minutes
Affected Services and Features:
Cloud Spanner experienced elevated p99.9 read latencies in us-central1 in some limited cases for a duration of 3 hours, 52 minutes. From preliminary analysis, the root cause of the issue was a resource contention issue on a small portion of the Spanner frontend servers in us-central1 that affected p99.9 latency for some customers.
A small portion of the regional Cloud Spanner instances in us-central1 experienced increased p99.9 latency for read requests. Multi-regional Cloud Spanner instances (nam6, nam7, nam9, nam10, nam11, nam12, and nam-eur-asia1) where the requests were routed to the impacted frontend servers in us-central1 were affected similarly.
The issue has been fully mitigated by isolating the impacted frontend servers, and we do not believe there is a risk of recurrence at this time.
| ||6 Oct 2021||18:28 PDT|| |
The issue with Cloud Spanner has been resolved for all affected projects as of Wednesday, 2021-10-06 17:42 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||6 Oct 2021||17:51 PDT|| |
Summary: us-central1-a,c: Cloud spanner jobs reporting high read latencies. Investigations ongoing.
Description: We are experiencing an issue with high read latencies on Cloud Spanner jobs at us-central1-a,c beginning at Wednesday, 2021-10-06 14:00 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Wednesday, 2021-10-06 18:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: None at this time.
Workaround: None at this time.