Service Health
-
-
-
-
Incident affecting Cloud Armor, Google Cloud Networking, Google App Engine, Cloud CDN
Global: Delays creating or modifying load balancer configurations
Incident began at 2022-03-31 11:07 and ended at 2022-03-31 15:13 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)GlobalMontréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
| Date | Time | Description | |
|---|---|---|---|
| | 1 Apr 2022 | 12:26 PDT | We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support. (All Times US/Pacific) Incident Start: 31 March 2022 11:07 Incident End: 31 March 2022 15:13 Duration: 4 hours, 6 minutes, Affected Services and Features:
Regions/Zones: Global Locale Description: Cloud Load Balancers experienced delays creating or modifying configurations for 4 hours 6 minutes. From the preliminary analysis, the root cause of the issue is a rollout on the configuration management system that manages and pushes configuration for multiple Google infrastructure systems. The roll out caused an increase in the task counts which overloaded the configuration management system as all tasks pulled the configuration at the same time. Customer Impact:
|
| | 31 Mar 2022 | 14:56 PDT | The issue with Cloud Armor, Google App Engine, Google Cloud Networking has been resolved for all affected users as of Thursday, 2022-03-31 14:55 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 31 Mar 2022 | 13:17 PDT | Summary: Global: Delays creating or modifying load balancer configurations Description: Mitigation efforts are reducing the backlog of delayed configuration changes, which are slowly being processed. We will provide more information by Thursday, 2022-03-31 15:15 US/Pacific. Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted. Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried. |
| | 31 Mar 2022 | 12:43 PDT | Summary: Global: Delays creating or modifying load balancer configurations Description: Our engineering team is exploring multiple mitigations options currently. We will provide more information by Thursday, 2022-03-31 15:00 US/Pacific. Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted. Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried. |
| | 31 Mar 2022 | 12:11 PDT | Summary: Global: Unable to create or modify some Cloud Networking configurations Description: Mitigation work is still underway by our engineering team. Current data indicates that approximately 15% of projects globally are affected by this issue. We will provide more information by Thursday, 2022-03-31 14:00 US/Pacific. Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted. Workaround: None at this time. |
| | 31 Mar 2022 | 11:51 PDT | Summary: Global: Unable to create or modify some Cloud Networking configurations Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2022-03-31 13:00 US/Pacific. Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted. Workaround: None at this time. |
| | 31 Mar 2022 | 11:41 PDT | Summary: Global: Unable to create or modify regional external HTTP(S) load balancers Description: We are experiencing an issue with Google Cloud Networking beginning at Thursday, 2022-03-31 11:07 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2022-03-31 12:45 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Affected customers are not able to create or change Regional external HTTP(S) load balancer balancers. Existing balancers are not impacted. Workaround: None at this time. |
- All times are US/Pacific