Google Cloud Service Health

Google Cloud Service Health
Incidents
Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Google Kubernetes Engine

Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters

Incident began at 2023-01-12 04:08 and ended at 2023-01-16 01:48 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)

Date	Time	Description
16 Jan 2023	01:48 PST	The issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters has been resolved for all affected users as of Monday, 2023-01-16 01:00 US/Pacific. We thank you for your patience while we worked on resolving the issue.
13 Jan 2023	10:18 PST	Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. The rollback is being applied to impacted clusters or is waiting for the user's time specified maintenance window. We will provide an update by Monday, 2023-01-16 09:00 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: Users with time specified maintenance windows can change the window to have the fix applied as soon as possible.
13 Jan 2023	08:26 PST	Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. The rollback is being applied to impacted clusters or is waiting for the user's time specified maintenance window. We will provide an update by Friday, 2023-01-13 10:30 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: Users with time specified maintenance windows can change the window to have the fix applied as soon as possible.
13 Jan 2023	03:49 PST	Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. We will provide an update by Friday, 2023-01-13 09:30 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: None at this time.
12 Jan 2023	06:56 PST	Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: Horizontal pod autoscaler (HPA) does not work in newly created GKE minor version 1.24+ clusters. A rollback of the affected feature is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2023-01-13 05:00 US/Pacific. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: None at this time.

All times are US/Pacific