Service Health
-
-
-
-
Incident affecting Cloud Spanner
Cloud Spanner instance creation failure
Incident began at 2022-03-28 15:30 and ended at 2022-03-29 02:58 (all times are US/Pacific).
Previously affected location(s)
Warsaw (europe-central2)South Carolina (us-east1)
| Date | Time | Description | |
|---|---|---|---|
| | 29 Mar 2022 | 16:36 PDT | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 28 March 2022 15:30 Incident End: 29 March 2022 2:58 Duration: 11 hours, 28 minutes Affected Services and Features: Cloud Spanner - Creation of new instance , Scaling existing instances. Regions/Zones: us-east1, europe-central2, multi-region location nam6 Description: Cloud Spanner experienced delays in creation and scaling of instances in us-east1, europe-central2, and multi-region location nam6 [1] for a duration of 11 hours, 28 minutes. The cause of this was a rollout of a new feature that affected the ability to create or modify instance compute capacity (via instance scaling or new instance creation). This issue was discovered internally and mitigation was performed via updating the configuration back to the previous settings. [1] https://cloud.google.com/spanner/docs/instance-configurations#available-configurations-multi-region Customer Impact: Affected customers would have experienced the following operations being stuck or failing with a timeout:
|
| | 29 Mar 2022 | 03:46 PDT | The issue with Cloud Spanner has been resolved for all affected users as of Tuesday, 2022-03-29 03:13 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 29 Mar 2022 | 02:56 PDT | Summary: Cloud Spanner instance creation failure Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Tuesday, 2022-03-29 04:30 US/Pacific. We will provide more information by Tuesday, 2022-03-29 04:30 US/Pacific. Diagnosis: The impacted users might experience timeout failures while:
Workaround: As a workaround, users may try to create new cloud spanner instance in a different region |
| | 29 Mar 2022 | 02:06 PDT | Summary: Cloud Spanner instance creation failure Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Tuesday, 2022-03-29 03:30 US/Pacific. We will provide more information by Tuesday, 2022-03-29 03:30 US/Pacific. Diagnosis: The impacted users might experience timeout failures while:
Workaround: As a workaround, users may try to create new cloud spanner instance in a different region |
| | 29 Mar 2022 | 00:15 PDT | Summary: Cloud Spanner instance creation failure Description: We are experiencing an issue with Cloud Spanner beginning at Monday, 2022-03-28 15:30 US/Pacific. Updating and creating a new Cloud Spanner instance in us-east1 and europe-central2 might fail with the timeout errors. Our engineering team continues to investigate the issue and working towards the mitigation. We will provide an update by Tuesday, 2022-03-29 05:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: The impacted users might experience timeout failures while:
Workaround: As a workaround, users may try to create new cloud spanner instance in a different region |
| | 28 Mar 2022 | 22:48 PDT | Summary: Cloud Spanner instance creation failure Description: We are experiencing an issue with Cloud Spanner beginning at Monday, 2022-03-28 15:30 US/Pacific. Updating and creating a new Cloud Spanner instance in us-east1 might be failing with timeout errors. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-03-29 01:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: The impacted users might experience timeout failures while:
Workaround: As a workaround, users may try to create new cloud spanner instance in a different region |
| | 28 Mar 2022 | 22:29 PDT | Summary: Cloud Spanner instance creation failure Description: We are experiencing an issue with Cloud Spanner beginning at Monday, 2022-03-28 15:30 US/Pacific. A new Cloud Spanner instance creation in us-east1 might be failing with timeout errors. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-03-29 01:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: The impacted users might experience timeout failures while creating a new Cloud Spanner instance Workaround: As a workaround, users may try to create new cloud spanner instance in a different region |
| | 28 Mar 2022 | 21:52 PDT | Summary: Cloud Spanner instance creation failure Description: We are experiencing an issue with Cloud Spanner beginning at Monday, 2022-03-28 19:00 US/Pacific. A new Cloud Spanner instance creation in us-east1 might be failing with timeout errors. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2022-03-28 23:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: The impacted users might experience timeout failures while creating a new Cloud Spanner instance Workaround: As a workaround, users may try to create new instance in a different region |
- All times are US/Pacific