Google Cloud Service Health

Google Cloud Service Health
Incidents
Google Kubernetes Engine Nodepool Upgrade Failures

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Kubernetes Engine

Google Kubernetes Engine Nodepool Upgrade Failures

Incident began at 2023-10-02 11:29 and ended at 2023-10-12 12:28 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Berlin (europe-west10)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Dammam (me-central2)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date	Time	Description
12 Oct 2023	12:28 PDT	A very small number of customers were unable to manually initiate a Nodepool upgrade. All other Nodepool operations have been running properly, and there was no impact to workloads or the ability to use the Kubernetes API. The failure rate is now limited to < 0.1% of Nodepool upgrades requests, and only occurring in us-central-1, us-west-1, us-east-1, europe-west1, europe-west4, and asia-northeast-1. This is down from a maximum failure rate of 0.4% Nodepool upgrades across GKE. Our engineering team is continuing to resolve the remaining (<0.1%) failure rate. Full resolution is expected to complete by Friday, 2023-10-13 17:00 US/Pacific. We thank you for your patience while we worked on resolving the issue.
12 Oct 2023	11:58 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: We believe the issue with Google Kubernetes Engine is partially resolved. We do not have an ETA for full resolution at this point. Our engineers are actively working on this, and we will provide an update by Thursday, 2023-10-12 13:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
11 Oct 2023	11:10 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: We believe the issue with Google Kubernetes Engine is partially resolved. We do not have an ETA for full resolution at this point. Our engineers are actively working on this, and we will provide an update by Thursday, 2023-10-12 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
10 Oct 2023	13:27 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: We believe the issue with Google Kubernetes Engine is partially resolved. We do not have an ETA for full resolution at this point. We will provide an update by Wednesday, 2023-10-11 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
10 Oct 2023	11:51 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Tuesday, 2023-10-10 14:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
9 Oct 2023	12:02 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Tuesday, 2023-10-10 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
9 Oct 2023	10:57 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Monday, 2023-10-09 16:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
6 Oct 2023	10:28 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: A mitigation has been rolling out and we are assessing its effectiveness. We will provide an update by Monday, 2023-10-09 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
3 Oct 2023	11:07 PDT	Summary: Global: Google Kubernetes Engine Nodepool Upgrade Failures Description: Our engineering team is working on a mitigation which will rollout over the next few days. We will provide an update by Friday, 2023-10-06 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested but may not always work. Workaround: Customers can re-create nodepools at the new version instead of upgrading in-place.
2 Oct 2023	14:57 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: Our engineering team continues to investigate mitigation pathways. We will provide an update by Tuesday, 2023-10-03 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this, may see "Internal error" in Google Cloud Console. Retrying is suggested however may not work. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.
2 Oct 2023	14:40 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: Our engineering team continues to investigate mitigation pathways. We will provide an update by Tuesday, 2023-10-03 12:00 US/Pacific with current details. Diagnosis: A small number of customers are experiencing failed nodepool upgrades. Customers experiencing this will see "Internal error" in Pantheon. In this scenario, retrying may not work. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.
2 Oct 2023	14:30 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-10-02 14:45 US/Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.
2 Oct 2023	12:14 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-10-02 14:30 US/Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.
2 Oct 2023	12:11 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-10-02 12:47 US/Pacific with current details. Diagnosis: Customers with nodepools at version 1.24 cannot upgrade them to the next minor version, 1.25. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.
2 Oct 2023	11:47 PDT	Summary: We are experiencing an issue with Google Kubernetes Engine Description: We are experiencing an issue with Google Kubernetes Engine. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-10-02 12:47 US/Pacific with current details. Diagnosis: Customers with nodepools at version 1.24.14-gke.2700 cannot upgrade them to the next minor version, 1.25.12-gke.500. Workaround: Customers can re-create nodepool at the new version instead of upgrading in-place.

All times are US/Pacific