Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Networking, Google Compute Engine, Persistent Disk, Google Kubernetes Engine, Google App Engine, Cloud Run, Google Cloud Bigtable, Google Cloud Storage, Cloud Filestore, Cloud Memorystore, Cloud Monitoring, Hybrid Connectivity, Memorystore for Memcached, Memorystore for Redis

europe-west2-c: Cloud networking is currently experiencing issues.

Incident began at 2021-09-03 05:40 and ended at 2021-09-03 06:20 (all times are US/Pacific).

Date Time Description
3 Sep 2021 12:27 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 03 September 2021 05:40

Incident End: 03 September 2021 06:20

Duration: 40 minutes

Affected Services and Features:

  • Cloud Networking
  • Compute Engine
  • Persistent Disk
  • Kubernetes Engine
  • App Engine
  • Cloud Run
  • Cloud Bigtable
  • Cloud Storage
  • Cloud Filestore
  • Cloud MemoryStore
  • Cloud Monitoring
  • Cloud VPN

Regions/Zones: europe-west2-c

Description:

Google Cloud Networking experienced elevated packet loss and latency in europe-west2-c for a duration of 40 minutes. Additional Cloud services that use Cloud Networking experienced similar symptoms. From preliminary analysis, the root cause of the issue was a sudden redirection of traffic from a datacenter that was experiencing an issue.

Customer Impact:

  • Affected customers may have experienced elevated packet loss and latency as their traffic was temporarily redirected.
  • Google Compute Engine instances experienced failed connectivity and packet loss on ingress, egress, and inter-zonal traffic. Autoscaling was also impacted and may have failed to make decisions.
  • Persistent Disk experienced elevated I/O latency on regional and zonal disks
  • Google Kubernetes Engine clusters experienced 30% of zonal and 5% of regional clusters reporting as unhealthy.
  • Google App Engine and Cloud Run served elevated 500 errors within europe-west2.
  • Cloud Bigtable experienced an elevated error rate and latency on operations within europe-west2.
  • Cloud SQL experienced issues connecting to instances in europe-west2. Affected instances may have lost telemetry data, and HA instances failed over if the primary was in the affected zone.
  • Google Cloud Storage experienced elevated read errors of up to 2.3% in europe-west2.
  • Cloud Filestore instances within europe-west2-c may have been unreachable.
  • Cloud MemoryStore instances within europe-west2 may have been unreachable.
  • Cloud Monitoring experienced delayed or missing metrics from resources in the zone.
  • Cloud VPN experienced a short duration where tunnels went down for a short time before being reestablished.
3 Sep 2021 07:17 PDT

The issue with Cloud Networking has been resolved for all affected users as of Friday, 2021-09-03 06:20 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
3 Sep 2021 07:06 PDT

Summary: europe-west2-c: Cloud networking is currently experiencing issues.

Description: On further investigation we have confirmed the issue is affecting a single zone europe-west2-c.

Our engineering team is currently investigating the issue and confirming the current impact to our users.

We will provide an update by Friday, 2021-09-03 07:30 US/Pacific with current details.

Diagnosis: Customer may high latency and packet loss.

Workaround: Other Zone and region can be utilized for workaround.
3 Sep 2021 07:02 PDT

Summary: europe-west2: Cloud networking is currently experiencing issues.

Description: We are experiencing an issue with Cloud Networking , beginning at Friday, 2021-09-03 0535 US/Pacific

Our engineering team is currently investigating the issue and confirming the current impact to our users.

We will provide an update by Friday, 2021-09-03 07:30 US/Pacific with current details.

Diagnosis: Customer may high latency and packet loss.

Workaround: Other Zone and region can be utilized for workaround.
3 Sep 2021 06:39 PDT

Summary: Cloud networking issues being investigated impacting some user in eu-west2c

Description: We are experiencing an issue with Cloud Networking , beginning at Friday, 2021-09-03 0535 US/Pacific Our engineering team continues to investigate the issue. We will provide an update by Friday, 2021-09-03 06:59 US/Pacific with current details.

Diagnosis: Customer may high latency and packet loss

Workaround: Other Zone and region are unaffected