Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Hybrid Connectivity, Virtual Private Cloud (VPC), Google Cloud Networking, Cloud NAT

Google Cloud Networking packet loss issue

Incident began at 2022-06-16 01:11 and ended at 2022-06-16 02:44 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Jakarta (asia-southeast2)Finland (europe-north1)

Date Time Description
16 Jun 2022 17:10 PDT

MINI INCIDENT REPORT

We apologize for the inconvenience this service disruption/outage caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 16 June 2022 01:11

Incident End: 16 June 2022 02:44

Duration: 1 hour, 33 minutes

Affected Services and Features:

  • Google Cloud Networking
  • Cloud Monitoring
  • Cloud NAT
  • Hybrid Connectivity
  • Cloud Run
  • Cloud Interconnect
  • Virtual Private Cloud (VPC)
  • Gmail
  • Cloud Bigtable

Regions/Zones: Europe and Asia

Description:

Google Cloud Networking experienced elevated packet loss and latency with Google Cloud Services across Europe and Asia for a duration of 1 hour, 36 minutes. Any service that uses Cloud Networking may have observed impact. We have included available details of service specific impact below; however, this may not be a comprehensive accounting of all downstream networking impact.

From preliminary analysis, the root cause of the issue was due to a large reconvergence event on our user facing backbone, which was triggered by a typical fiber cut in North America. The failure resulted in packet loss due to either some MPLS tunnels going down, or due to congestion on remaining tunnels that quickly grew to accommodate demand on next-best paths. Tunnels going down is not expected behavior of MPLS control plane in response to best-path capacity going down when adjacent capacity remains available and is being investigated with highest priority with the hardware vendor. Mitigation of the packet loss required Google engineers to manually intervene on the network control plane. The underlying Cloud Networking impact ended at 02:58, however, some Cloud Services may have taken longer to recover.

Due to backbone control plane behavior during this event, network forwarding may have been suboptimal for some destinations resulting in higher network latency.

Customer Impact:

  • Google Cloud Networking - Public IP traffic connectivity failed from 01:22 to 02:58 US/Pacific.
  • Cloud Monitoring - Affected customers observed metric data failing to publish in europe-north1.
  • Cloud NAT - Affected customers experienced packet loss on some flows between regions in Europe, South America, and North America.
  • Hybrid Connectivity - Affected customers experienced packet loss on some flows between regions in Europe, South America, and North America.
  • Cloud Run - Affected customers experienced packet loss in europe-north1.
  • Cloud Interconnect - Affected customers experienced packet loss on some flows between regions in Europe, South America, and North America.
  • Virtual Private Cloud (VPC) - Affected customers experienced up to 100% packet loss with traffic between Europe and Asia over private and public IP.
  • Cloud Bigtable - Affected customers experienced elevated latency with read and write operations in europe-north1.
  • Gmail - Affected web client users experienced transient service unavailability.

Additional details:

Fiber cuts on the Google network are quite common with no observable malfunction to the MPLS control plane. This incident surfaces a novel MPLS control plane behavior. We are committed to fully identifying the root cause and addressing it to prevent it with our partners.
16 Jun 2022 03:03 PDT

The issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) has been resolved for all affected projects as of Thursday, 2022-06-16 03:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
16 Jun 2022 02:46 PDT

Summary: Google Cloud Networking packet loss issue

Description: We are experiencing an issue with Cloud NAT, Google Cloud Networking, Hybrid Connectivity, Virtual Private Cloud (VPC) beginning at Thursday, 2022-06-16 01:11 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2022-06-16 03:15 US/Pacific with current details.

Diagnosis: The customers may experience persistent packet loss for some connection flows

Workaround: None at this time.