Google Cloud Service Health

Google Cloud Service Health
Incidents
We are experiencing an issue with Google Kubernetes Engine

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Kubernetes Engine

We are experiencing an issue with Google Kubernetes Engine

Incident began at 2023-09-13 16:05 and ended at 2023-09-28 12:52 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Berlin (europe-west10)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Dammam (me-central2)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date	Time	Description
28 Sep 2023	12:52 PDT	The issue with Google Kubernetes Engine has been resolved for all affected projects as of Thursday, 2023-09-28 12:52 US/Pacific. We thank you for your patience while we worked on resolving the issue.
25 Sep 2023	13:51 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: Google Kubernetes Engine clusters on version 1.25 in all locations may experience stuck Persistent Volumes The fix is gradually rolling out to all affected clusters. Newly created clusters will no longer be impacted. Existing clusters on version 1.25 may show stuck volumes, and our engineers are actively monitoring and mitigating We will provide more information by Thursday, 2023-09-28 13:30 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status for few minutes because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck for few minutes in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
22 Sep 2023	12:32 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: Google Kubernetes Engine clusters on version 1.25 in all locations may experience stuck Persistent Volumes The engineering team is actively fixing all affected clusters: Newly created clusters will no longer be impacted. Existing clusters on version 1.25 may show stuck volumes, but we are actively monitoring and mitigating We will provide more information by Monday, 2023-09-25 13:30 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status for few minutes because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck for few minutes in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
15 Sep 2023	09:52 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: Google Kubernetes Engine clusters on version 1.25 in all locations may experience stuck Persistent Volumes The engineering team is actively fixing all affected clusters: Newly created clusters will no longer be impacted. Existing clusters on version 1.25 may show stuck volumes, but we are actively monitoring and mitigating We will provide more information by Friday, 2023-09-22 12:30 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status for few minutes because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck for few minutes in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
14 Sep 2023	23:31 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Our engineering team continues the mitigation efforts. We do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Friday, 2023-09-15 12:30 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
14 Sep 2023	10:41 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Mitigation work remians underway by our engineering team, we do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Thursday, 2023-09-14 23:30 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	21:37 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Mitigation work is currently underway by our engineering team, we do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Thursday, 2023-09-14 11:00 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	19:45 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Mitigation work is currently underway by our engineering team, we do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Thursday, 2023-09-14 01:00 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	18:21 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Mitigation work is currently underway by our engineering team, we do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Wednesday, 2023-09-13 20:00 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	17:11 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Mitigation work is currently underway by our engineering team, we do not have a confirmed ETA for completion and will continue to provide regular updates. We will provide more information by Wednesday, 2023-09-13 18:40 US/Pacific. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	16:20 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2023-09-13 17:30 US/Pacific with current details. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	16:15 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine impacting customers using Kubernetes version 1.25. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2023-09-13 17:00 US/Pacific with current details. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PersistentVolumeClaim (PVC) may be stuck in pending status because the PersistentVolume (PV) and underlaying disks cannot be created. The PersistentVolume (PV) deletion may be stuck in deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.
13 Sep 2023	16:05 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2023-09-13 16:41 US/Pacific with current details. Diagnosis: GKE customers using Kubernetes version 1.25 may experience issues with Persistent Disk creation or deletion failures. The PVC my be stuck in Pending status because the PV and the underlaying disks cannot be created. The PV deletion may be stuck in Deleting status because the underlaying disk cannot be deleted. Workaround: None at this time.

All times are US/Pacific