Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking, Virtual Private Cloud (VPC), Hybrid Connectivity

Regions: (Asia, Western US) <-> (Eastern US, Europe). Packet loss on inter region traffic.

Incident began at 2021-09-07 15:04 and ended at 2021-09-07 15:45 (all times are US/Pacific).

Date Time Description
8 Sep 2021 10:25 PDT

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 07 September 2021 15:04

Incident End: 07 September 2021 15:45

Duration: 41 minutes

Affected Services and Features:

  • Google Cloud Networking - Cloud VPC, Cloud Interconnect.
  • Ingress and Egress traffic of any Google Cloud services to and from the affected regions.

Regions/Zones:

The traffic between multiple regions (inter-region) was affected, while the traffic within each region (intra-region) remained unaffected. The regions asia-east, asia-northeast, asia-southeast, australia-southeast, us-west, and us-central had trouble connecting to the regions of us-east, northamerica-northeast, europe-central, europe-north, and europe-west, and vice-versa.

Description:

Google Cloud Networking experienced elevated packet loss for traffic between multiple regions for a duration of 41 minutes. Preliminary analysis indicates that during routine and typically non-disruptive router maintenance, a software issue occurred which unintentionally impacted packet forwarding.

Customer Impact:

  • Google Cloud Networking - elevated packet loss
  • Other services using Cloud Networking - elevated latencies or other networking related delays.
7 Sep 2021 16:16 PDT

The issue with Cloud Networking has been resolved for all affected projects as of Tuesday, 2021-09-07 15:45 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

7 Sep 2021 16:07 PDT

Summary: Regions: (Asia, Western US) <-> (Eastern US, Europe). Packet loss on inter region traffic. Mitigation completed.

Description: We believe the issue with Cloud Networking is mitigated starting 15:45 US/Pacific . Our engineering team continues to work towards full resolution of the issue.

We will provide an update by Tuesday, 2021-09-07 16:37 US/Pacific with current details.

Diagnosis: Customer will see packet loss on traffic between regions in US and Asia

Workaround: None at this time.

7 Sep 2021 16:05 PDT

Summary: Regions: (Asia, Western US) <-> (Eastern US, Europe). Packet loss on inter region traffic. Mitigation completed.

Description: We believe the issue with Cloud Networking is mitigated starting 15:45. Our engineering team continues to work towards full resolution of the issue.

We will provide an update by Tuesday, 2021-09-07 16:37 US/Pacific with current details.

Diagnosis: Customer will see packet loss on traffic between regions in US and Asia

Workaround: None at this time.

7 Sep 2021 15:57 PDT

Summary: Regions: US, Asia: Packet loss reported on GCE traffic between US and Asia

Description: We are experiencing an issue with Cloud Networking beginning at Tuesday, 2021-09-07 15:04 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2021-09-07 16:28 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customer will see packet loss on traffic between regions in US and Asia

Workaround: None at this time.