Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking

Google Cloud Networking customers may experience packet loss between some regions in Asia

Incident began at 2023-05-03 08:47 and ended at 2023-05-04 03:10 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Singapore (asia-southeast1)

Date Time Description
5 May 2023 08:59 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 3 May 2023 08:47

Incident End: 4 May 2023 03:10

Duration: 18 hours, 23 minutes

Affected Services and Features:

Google Cloud Networking

Regions/Zones: asia-southeast1, asia-east1

Description:

Google Cloud Networking experienced intermittent periods of packet loss in asia-southeast1 and asia-east1 for 18 hours, 23 minutes. From preliminary analysis, the root cause of the issue was a single router that was experiencing undetected and intermittent control plane protocol pathing issues related to Multiprotocol Label Switching RSVP-TE and IS-IS. The incident was triggered by a minor loss of network capacity between the affected regions, despite there being more than enough capacity and paths available. .

Customer Impact:

Customers may have experienced periods of up to 6% packet loss, each lasting 1-2 minutes, primarily for cross-region traffic to or from the asia-southeast1and asia-east1 regions.
4 May 2023 04:18 PDT

The issue with Google Cloud Networking has been resolved for all affected users as of Thursday, 2023-05-04 04:11 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
4 May 2023 00:34 PDT

Summary: Google Cloud Networking customers may experience packet loss between some regions in Asia

Description: Mitigation work is currently underway by our engineering team.

The mitigation is taking longer than expected.

We will provide more information by Thursday, 2023-05-04 05:00 US/Pacific.

Diagnosis: Customers may see spikes of intermittent latency and/or mild to moderate packet loss

Workaround: None at this time.
3 May 2023 22:07 PDT

Summary: Google Cloud Networking customers may experience packet loss between some regions in Asia

Description: Mitigation work is currently underway by our engineering team.

The mitigation is taking longer than expected.

We will provide more information by Thursday, 2023-05-04 01:00 US/Pacific.

Diagnosis: Customers may see spikes of intermittent latency and/or mild to moderate packet loss

Workaround: None at this time.