Incident affecting Hybrid Connectivity, Virtual Private Cloud (VPC), Google Cloud Networking, Cloud NAT
Google Cloud Networking packet loss issue
Incident began at 2022-06-16 01:11 and ended at 2022-06-16 02:44 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Jakarta (asia-southeast2)Finland (europe-north1)
| ||16 Jun 2022||17:10 PDT|| |
MINI INCIDENT REPORT
We apologize for the inconvenience this service disruption/outage caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
Incident Start: 16 June 2022 01:11
Incident End: 16 June 2022 02:44
Duration: 1 hour, 33 minutes
Affected Services and Features:
Regions/Zones: Europe and Asia
Google Cloud Networking experienced elevated packet loss and latency with Google Cloud Services across Europe and Asia for a duration of 1 hour, 36 minutes. Any service that uses Cloud Networking may have observed impact. We have included available details of service specific impact below; however, this may not be a comprehensive accounting of all downstream networking impact.
From preliminary analysis, the root cause of the issue was due to a large reconvergence event on our user facing backbone, which was triggered by a typical fiber cut in North America. The failure resulted in packet loss due to either some MPLS tunnels going down, or due to congestion on remaining tunnels that quickly grew to accommodate demand on next-best paths. Tunnels going down is not expected behavior of MPLS control plane in response to best-path capacity going down when adjacent capacity remains available and is being investigated with highest priority with the hardware vendor. Mitigation of the packet loss required Google engineers to manually intervene on the network control plane. The underlying Cloud Networking impact ended at 02:58, however, some Cloud Services may have taken longer to recover.
Due to backbone control plane behavior during this event, network forwarding may have been suboptimal for some destinations resulting in higher network latency.
Fiber cuts on the Google network are quite common with no observable malfunction to the MPLS control plane. This incident surfaces a novel MPLS control plane behavior. We are committed to fully identifying the root cause and addressing it to prevent it with our partners.
| ||16 Jun 2022||03:03 PDT|| |
The issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) has been resolved for all affected projects as of Thursday, 2022-06-16 03:02 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||16 Jun 2022||02:46 PDT|| |
Summary: Google Cloud Networking packet loss issue
Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Thursday, 2022-06-16 01:11 US/Pacific.
Our engineering team continues to investigate the issue.
We will provide an update by Thursday, 2022-06-16 03:15 US/Pacific with current details.
Diagnosis: The customers may experience persistent packet loss for some connection flows
Workaround: None at this time.