Service Health
-
-
-
-
Incident affecting Google Cloud Networking
Google engineer are currently investigating a issue with the cloud networking product
Incident began at 2022-06-19 12:01 and ended at 2022-06-19 14:27 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
| Date | Time | Description | |
|---|---|---|---|
| | 20 Jun 2022 | 04:52 PDT | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213. (All Times US/Pacific) Incident Start: 19 June 2022 10:14 Incident End: 19 June 2022 13:39 Duration: 3 hours, 25 minutes Affected Services and Features: Google Cloud Networking. Regions/Zones: Global Description: Google Cloud Networking experienced an issue with the Cloud Armour config changes that got stalled for a duration of 3 hours, 25 minutes. From preliminary analysis, the root cause of the issue is Google Cloud Armor Configuration Server experienced errors in writing configs to the config storage system. Customer Impact: Customer configs propagation depending on the dos_cloud configs got delayed. Some impacted customers would not be able to make Cloud Armor config changes. |
| | 19 Jun 2022 | 14:27 PDT | The issue with Google Cloud Networking has been resolved for all affected projects as of Sunday, 2022-06-19 14:27 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 19 Jun 2022 | 13:31 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 14:30 US/Pacific. Diagnosis: Customer configs propagation depending on the dos_cloud configs is delayed. Customer will be able to make cloud armour config changes but expect higher latency for backlog to clear Workaround: n/a |
| | 19 Jun 2022 | 13:30 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 14:30 US/Pacific. Diagnosis: Customer configs propagation depending on the dos_cloud configs is delayed. Customer will be able to make cloud armour config changes but expect higher latency for backlog to clear Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a |
| | 19 Jun 2022 | 13:00 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Sunday, 2022-06-19 13:40 US/Pacific. Diagnosis: Customer are unable to make cloud armour config changes globally. Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a |
| | 19 Jun 2022 | 12:47 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud armour config changes globally. Customer configs propagation depending on the dos_cloud configs is delayed. Workaround: n/a |
| | 19 Jun 2022 | 12:10 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud armour config changes globally. Workaround: n/a |
| | 19 Jun 2022 | 12:03 PDT | Summary: Google engineer are currently investigating a issue with the cloud networking product Description: We are experiencing an issue with Google Cloud Networking beginning at Sunday, 2022-06-19 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Sunday, 2022-06-19 13:35 US/Pacific with current details. Diagnosis: Customer are unable to make cloud amour config changes globally. Workaround: n/a |
- All times are US/Pacific