Service Health
Incident affecting Google Kubernetes Engine
Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters
Incident began at 2023-01-12 04:08 and ended at 2023-01-16 01:48 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)
Date | Time | Description | |
---|---|---|---|
| 16 Jan 2023 | 01:48 PST | The issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters has been resolved for all affected users as of Monday, 2023-01-16 01:00 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| 13 Jan 2023 | 10:18 PST | Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. The rollback is being applied to impacted clusters or is waiting for the user's time specified maintenance window. We will provide an update by Monday, 2023-01-16 09:00 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: Users with time specified maintenance windows can change the window to have the fix applied as soon as possible. |
| 13 Jan 2023 | 08:26 PST | Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. The rollback is being applied to impacted clusters or is waiting for the user's time specified maintenance window. We will provide an update by Friday, 2023-01-13 10:30 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: Users with time specified maintenance windows can change the window to have the fix applied as soon as possible. |
| 13 Jan 2023 | 03:49 PST | Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: We believe the issue with Horizontal Pod Autoscaler (HPA) not working in newly created GKE minor version 1.24+ clusters is partially resolved. A rollout of a fix disabling the affected version for new clusters was completed on January 12, 20:14 PST and this fix makes HPA work in clusters created after the completion of the rollout. Our engineering team continues to work on a rollback of the affected feature for clusters created before the completion of the fix. We will provide an update by Friday, 2023-01-13 09:30 US/Pacific with current details. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: None at this time. |
| 12 Jan 2023 | 06:56 PST | Summary: Horizontal Pod Autoscaler (HPA) doesn't work in newly created GKE Minor version 1.24+ clusters Description: Horizontal pod autoscaler (HPA) does not work in newly created GKE minor version 1.24+ clusters. A rollback of the affected feature is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Friday, 2023-01-13 05:00 US/Pacific. Diagnosis: Horizontal pod autoscaler might not scale workloads within newly created clusters. Customers might observe "No recommendation " in the HPA object status. Workaround: None at this time. |
- All times are US/Pacific