Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud DNS, Google Cloud Networking

Regional L7XLB, Regional L7ILB, L4 Load Balancers, HCaaS, Cloud DNS configuration changes in asia-south1 are failing

Incident began at 2023-01-30 00:10 and ended at 2023-01-30 02:45 (all times are US/Pacific).

Previously affected location(s)

Mumbai (asia-south1)

Date Time Description
30 Jan 2023 12:17 PST

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 30 January 2023 00:10 PST

Incident End: 30 January 2023 02:45 PST

Duration: 2 hours, 35 minutes

Affected Services and Features:

  • Google Cloud DNS
  • Google Cloud Networking - Regional Internal HTTP(S) Load Balancers, Regional External HTTP(S) Load Balancers, L4 Load Balancers, Health Check as a Service (HCaaS)

Regions/Zones: asia-south1

Description:

Regional Internal HTTP(S) Load Balancers, Regional External HTTP(S) Load Balancers, L4 Load Balancers, Health Check as a Service (HCaaS), and Cloud DNS experienced high latency and failures for configuration changes in asia-south1 for 2 hours 35 minutes. From preliminary analysis, the root cause of the issue was an unresponsive backend service responsible for the distribution of configuration changes.

Customer Impact:

Customers may have experienced errors while creating or changing configuration of Regional Internal HTTP(S) Load Balancers, Regional External HTTP(S) Load Balancers, L4 Load Balancers, HCaaS, and Cloud DNS in the affected region. Existing configurations should have continued to work as expected.
30 Jan 2023 02:53 PST

The issue with Google Cloud DNS, Google Cloud Networking has been resolved for all affected users as of Monday, 2023-01-30 02:20 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
30 Jan 2023 02:30 PST

Summary: Regional L7XLB, Regional L7ILB, L4 Load Balancers, HCaaS, Cloud DNS configuration changes in asia-south1 are failing

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Monday, 2023-01-30 03:48 US/Pacific.

Diagnosis: Customers may experiencing errors while creating and changing configuration of load balancers. Existing LBs should continue to work as expected.

Workaround: None at this time.
30 Jan 2023 02:06 PST

Summary: L7XLB and L7ILB configuration changes for loadbalancers in asia-south1 are delayed

Description: We are experiencing an issue with Google Cloud Networking, Cloud DNS beginning at Monday, 2023-01-30 01:07 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2023-01-30 03:15 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may experiencing errors while creating load balancers. Existing LBs should continue to work as expected.

Workaround: None at this time.