Incident affecting Google Cloud DNS, Google Cloud Networking
Regional L7XLB, Regional L7ILB, L4 Load Balancers, HCaaS, Cloud DNS configuration changes in asia-south1 are failing
Incident began at 2023-01-30 00:10 and ended at 2023-01-30 02:45 (all times are US/Pacific).
Previously affected location(s)
| ||30 Jan 2023||12:17 PST|| |
Mini Incident Report
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
Incident Start: 30 January 2023 00:10 PST
Incident End: 30 January 2023 02:45 PST
Duration: 2 hours, 35 minutes
Affected Services and Features:
Regional Internal HTTP(S) Load Balancers, Regional External HTTP(S) Load Balancers, L4 Load Balancers, Health Check as a Service (HCaaS), and Cloud DNS experienced high latency and failures for configuration changes in asia-south1 for 2 hours 35 minutes. From preliminary analysis, the root cause of the issue was an unresponsive backend service responsible for the distribution of configuration changes.
Customers may have experienced errors while creating or changing configuration of Regional Internal HTTP(S) Load Balancers, Regional External HTTP(S) Load Balancers, L4 Load Balancers, HCaaS, and Cloud DNS in the affected region. Existing configurations should have continued to work as expected.
| ||30 Jan 2023||02:53 PST|| |
The issue with Google Cloud DNS, Google Cloud Networking has been resolved for all affected users as of Monday, 2023-01-30 02:20 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||30 Jan 2023||02:30 PST|| |
Summary: Regional L7XLB, Regional L7ILB, L4 Load Balancers, HCaaS, Cloud DNS configuration changes in asia-south1 are failing
Description: Mitigation work is currently underway by our engineering team.
We do not have an ETA for mitigation at this point.
We will provide more information by Monday, 2023-01-30 03:48 US/Pacific.
Diagnosis: Customers may experiencing errors while creating and changing configuration of load balancers. Existing LBs should continue to work as expected.
Workaround: None at this time.
| ||30 Jan 2023||02:06 PST|| |
Summary: L7XLB and L7ILB configuration changes for loadbalancers in asia-south1 are delayed
Description: We are experiencing an issue with Google Cloud Networking, Cloud DNS beginning at Monday, 2023-01-30 01:07 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2023-01-30 03:15 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Customers may experiencing errors while creating load balancers. Existing LBs should continue to work as expected.
Workaround: None at this time.
- All times are US/Pacific