Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking, Google Compute Engine, Google Kubernetes Engine

Global: Cloud Networking connectivity issues for customers using packet mirroring.

Incident began at 2021-09-13 10:20 and ended at 2021-09-18 01:27 (all times are US/Pacific).

Date Time Description
20 Sep 2021 12:25 PDT

We apologize for the inconvenience this service outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 13 September 2021 10:20

Incident End: 18 September 2021 01:27

Duration: 4 days, 15 hours, 7 minutes

Affected Services and Features:

  • Google Compute Engine – Access to Google Cloud services such as Google Cloud Storage, Cloud Bigtable, Cloud Spanner, and Google Container Registry were affected for Virtual Machine (VMs) with packet mirroring enabled.
  • Google Kubernetes Engine – Clusters with mirroring enabled had issues accessing Google endpoints including Google Container Registry and Google Cloud storage buckets.

Regions/Zones: Global

Description:

Google Compute Engine (GCE) VM instances and Google Kubernetes Engine (GKE) instances were unable to access Google Cloud services if the packet mirroring feature was enabled.

From preliminary analysis, the root cause was identified as an issue in the packet mirroring functionality that resulted in packet drops for VM to Google Cloud Services endpoint connections.

Customer Impact:

  • GCE VMs with packet mirroring enabled were unable to access Google services.
    • Affected VMs were unable to retrieve Google Container Registry (GCR) images.
    • Affected VMs experienced timeouts when accessing Google Services.
  • GKE clusters with packet mirroring enabled were unable to access Google services.

Additional details:

  • As a workaround, customers were informed to disable packet mirroring (if enabled) in order to reduce the impact on affected projects.
18 Sep 2021 02:47 PDT

The issue with Cloud Networking has been mitigated for all affected projects and there is no customer impact as of Saturday, 2021-09-18 01:29 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
18 Sep 2021 00:07 PDT

Summary: Global: Cloud Networking connectivity issues for customers using packet mirroring.

Description: Engineering team is continuing to monitor the mitigation steps currently being rolled out and expect it to complete by Saturday, 2021-09-18 03:00 US/Pacific.

Customers can disable packet mirroring as a workaround.

We will provide more information by Saturday, 2021-09-18 03:00 US/Pacific.

Diagnosis: Customers might experience connection failures or delays for connections from GCE VMs to Google endpoints including gcr.io and Google Cloud Storage buckets.

Workaround: Disable packet mirroring.
17 Sep 2021 22:43 PDT

Summary: Global: Cloud Networking connectivity issues for customers using packet mirroring.

Description: Engineering team is continuing to monitor the mitigation steps currently being rolled out and expect it to complete by Saturday, 2021-09-18 01:00 US/Pacific.

Customers can disable packet mirroring as a workaround.

We will provide more information by Saturday, 2021-09-18 01:00 US/Pacific.

Diagnosis: Customers might experience connection failures or delays for connections from GCE VMs to Google endpoints including gcr.io and Google Cloud Storage buckets.

Workaround: Disable packet mirroring.
17 Sep 2021 19:22 PDT

Summary: Global: Cloud Networking connectivity issues for customers using packet mirroring.

Description: Mitigation work is underway by our engineering team. The mitigation is expected to complete by Friday, 2021-09-17 23:00 US/Pacific.

Customers can disable packet mirroring as a workaround.

We will provide more information by Friday, 2021-09-17 23:00 US/Pacific.

Diagnosis: Customers might experience connection failures or delays for connections from GCE VMs to Google endpoints including gcr.io and Google Cloud Storage buckets.

Workaround: Disable packet mirroring.
17 Sep 2021 13:10 PDT

Summary: Global: Cloud Networking connectivity issues for customers using packet mirroring.

Description: Mitigation work is underway by our engineering team. The mitigation is expected to complete by Friday, 2021-09-17 10:00 US/Pacific.

Customers can disable packet mirroring as a workaround.

We will provide more information by Friday, 2021-09-17 23:00 US/Pacific.

Diagnosis: Customers might experience connection failures or delays for connections from GCE VMs to Google endpoints including gcr.io and Google Cloud Storage buckets.

Workaround: Disable packet mirroring.