Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Armor, Google Cloud Networking, Google App Engine

Global: Delays creating or modifying load balancer configurations

Incident began at 2022-03-31 11:07 and ended at 2022-03-31 15:13 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)GlobalMontréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
1 Apr 2022 12:26 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 31 March 2022 11:07

Incident End: 31 March 2022 15:13

Duration: 4 hours, 6 minutes,

Affected Services and Features:

  • Global External HTTP/S load balancing
  • Cloud CDN
  • Cloud Armor
  • Appengine flex

Regions/Zones: Global Locale

Description:

Cloud Load Balancers experienced delays creating or modifying configurations for 4 hours 6 minutes. From the preliminary analysis, the root cause of the issue is a rollout on the configuration management system that manages and pushes configuration for multiple Google infrastructure systems. The roll out caused an increase in the task counts which overloaded the configuration management system as all tasks pulled the configuration at the same time.

Customer Impact:

  • Affected customers would have experienced delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN.
  • App Engine Flexible apps might have failed to deploy.
31 Mar 2022 14:56 PDT

The issue with Cloud Armor, Google App Engine, Google Cloud Networking has been resolved for all affected users as of Thursday, 2022-03-31 14:55 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

31 Mar 2022 13:17 PDT

Summary: Global: Delays creating or modifying load balancer configurations

Description: Mitigation efforts are reducing the backlog of delayed configuration changes, which are slowly being processed.

We will provide more information by Thursday, 2022-03-31 15:15 US/Pacific.

Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted.

Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried.

31 Mar 2022 12:43 PDT

Summary: Global: Delays creating or modifying load balancer configurations

Description: Our engineering team is exploring multiple mitigations options currently.

We will provide more information by Thursday, 2022-03-31 15:00 US/Pacific.

Diagnosis: Affected customers will see delays in creating or modifying the configuration of Global External HTTP(S) load balancers, Cloud Armor and Cloud CDN. App Engine Flexible apps may fail to deploy. Configuration changes will be accepted, but not take effect until the issue is resolved. Existing configurations and deployments are not impacted.

Workaround: Configuration changes will eventually take effect. Failed App Engine Flexible deployments can be retried.

31 Mar 2022 12:11 PDT

Summary: Global: Unable to create or modify some Cloud Networking configurations

Description: Mitigation work is still underway by our engineering team.

Current data indicates that approximately 15% of projects globally are affected by this issue.

We will provide more information by Thursday, 2022-03-31 14:00 US/Pacific.

Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted.

Workaround: None at this time.

31 Mar 2022 11:51 PDT

Summary: Global: Unable to create or modify some Cloud Networking configurations

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2022-03-31 13:00 US/Pacific.

Diagnosis: Affected customers are not able to create or change External HTTP(S) load balancers. Cloud Armor and Cloud CDN have similar symptoms. App Engine Flexible apps may fail to deploy. In general, config change will be accepted, but will not take effect. Existing configurations are not impacted.

Workaround: None at this time.

31 Mar 2022 11:41 PDT

Summary: Global: Unable to create or modify regional external HTTP(S) load balancers

Description: We are experiencing an issue with Google Cloud Networking beginning at Thursday, 2022-03-31 11:07 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2022-03-31 12:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Affected customers are not able to create or change Regional external HTTP(S) load balancer balancers. Existing balancers are not impacted.

Workaround: None at this time.