Service Health
-
-
-
-
Incident affecting Cloud Filestore
Cloud Filestore ListInstances API failed with error code 429 globally
Incident began at 2022-09-13 07:38 and ended at 2022-09-13 10:42 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
| Date | Time | Description | |
|---|---|---|---|
| | 26 Sep 2022 | 15:10 PDT | INCIDENT REPORTSummary: On Tuesday, 13 September 2022 07:30 US/Pacific, an issue with Google Cloud Filestore resulted in all read-only API requests failing for a period of 3 hours, 12 minutes. The issue was fully mitigated on Tuesday, 13 September 2022 at 10:42 US/Pacific. We would like to apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability of the service in the future. Root Cause: Filestore enforces a global limit on API requests to limit impact in overload scenarios. The outage was triggered when an internal Google service managing a large number of GCP projects malfunctioned and overloaded the Filestore API with requests, causing global throttling of the Filestore API. This continued until the internal service was manually paused. As a result of this throttling, read-only API access was unavailable for all customers. This affected customers in all locations, due to a global quota that applies to Filestore. Console, gcloud and API access (List, GetOperation, etc.) calls all failed for a duration of 3 hours, 12 minutes. Mutate operations (CreateInstance, UpdateInstance, CreateBackup, etc.) still succeeded, but customers were unable to check on operation progress. Remediation and Prevention Our engineers were alerted of the issue through automated alerts on Tuesday, 13 September 2022 07:47 US/Pacific. Once the nature and scope of the issue became clear, Google engineers started shutting down the internal service overloading our APIs at 10:33 and the issue was completely resolved at 10:42. Google is committed to preventing recurrence by implementing the following action items:
We appreciate your patience and apologize again for the impact to your organization. We thank you for your business. Detailed Description of Impact: On Tuesday, 13 September 2022 from 07:30 to 10:42 US/Pacific, Google Cloud Filestore experienced an issue where ListInstances API requests failed with error code 429. This impacted read-only API access for customers globally. Pantheon, gcloud and API access (List, GetOperation, etc.) calls failed during this time. Mutate operations (CreateInstance, UpdateInstance, CreateBackup, etc.) still succeeded, however clients were not able to check on operation progress. |
| | 13 Sep 2022 | 11:46 PDT | MINI INCIDENT REPORTWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 13 September 2022 07:38 Incident End: 13 September 2022 10:42 Duration: 3 hours, 04 minutes Affected Services and Features: Google Cloud Filestore - API requests Regions/Zones: Global Description: Google Cloud Filestore experienced an issue where ListInstances API requests failed with error code 429 for a duration of 3 hours, 04 minutes. From preliminary analysis, the root cause of the issue is an unexpected spike in traffic from internal backend services. Customer Impact: Customers encountered error 429 when listing Cloud Filestore instances using API, gcloud CLI, and Google Cloud Console UI. |
| | 13 Sep 2022 | 10:46 PDT | The issue with Cloud Filestore has been resolved for all affected users as of Tuesday, 2022-09-13 10:42 US/Pacific. We will publish an analysis of this incident once we have completed our internal investigation. We thank you for your patience while we worked on resolving the issue. |
| | 13 Sep 2022 | 10:33 PDT | Summary: Cloud Filestore ListInstances API failed with error code 429 globally Description: We are experiencing an issue with Cloud Filestore beginning at Tuesday, 2022-09-13 07:38 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-09-13 11:32 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may see the error code 429 globally in Pantheon when they are viewing the list of instances in the Filestore UI. Workaround: None at this time. |
| | 13 Sep 2022 | 09:46 PDT | Summary: Cloud Filestore ListInstances API failed with error code 429 globally Description: We are experiencing an issue with Cloud Filestore. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2022-09-13 10:45 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may see the error code 429 globally in Pantheon when they are viewing the list of instances in the Filestore UI. Workaround: None at this time. |
- All times are US/Pacific