Service Health
-
-
-
-
Incident affecting Google Kubernetes Engine
We are investigating an issue with Google Kubernetes Engine where some nodes in recently upgraded clusters (see affected versions) may be experiencing elevated numbers of kernel panics
Incident began at 2019-11-04 11:46 and ended at 2019-11-13 15:38 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 13 Nov 2019 | 15:38 PST | The issue with Google Kubernetes Engine clusters with node pools experiencing an elevated number of kernel panics has been resolved in a new release of GKE available as of Wednesday, 2019-11-11 16:00 US/Pacific. The fix is contained in the following versions of GKE which is currently rolling out to node pools with auto upgrade enabled [1]. This should complete by Friday, 2019-11-15. Any customer on manual updates will need to manually upgrade their nodes to the following versions.: 1.13.11-gke.14 1.13.12-gke.8 1.14.7-gke.23 1.14.8-gke.12 Please note that this fix has downgraded the version of CoS to cos-73-11647-293-0 [2] as a temporary mitigation, we expect the next release of GKE to have an upgraded kernel and fix for the panics seen in the below releases.. Affected versions were: 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4, and 1.14.8-gke.7 We thank you for your patience while we've worked on resolving the issue. [1] - https://cloud.google.com/kubernetes-engine/versioning-and-upgrades#rollout_schedule [2] - https://cloud.google.com/container-optimized-os/docs/release-notes#cos-73-11647-293-0 |
| | 11 Nov 2019 | 18:53 PST | Description: The following fixed versions are now available and should fix the kernel panic issue: 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12. Mitigation work is currently underway by our engineering team to roll out the fixed versions to clusters configured with node auto-update, and is expected to be complete by Wednesday, 2019-11-13. Clusters not configured with node auto-update can be manually upgraded. At this time the following versions are still affected: 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4, and 1.14.8-gke.7. We will provide more information as it becomes available or by Wednesday, 2019-11-13 17:00 US/Pacific at the latest. Diagnosis: Affected users may notice elevated levels of kernel panics on nodes running one of the affected versions listed above. Workaround: Users seeing this issue can upgrade to a fixed release. |
| | 8 Nov 2019 | 17:10 PST | Description: This issue was downgraded to an orange category Service Disruption as the number of projects actually affected is very low. Mitigation work is currently underway by our engineering team and is expected to be complete by Wednesday, 2019-11-13. At this time the following versions are still affected: 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4, and 1.14.8-gke.7. We will provide more information as it becomes available or by Wednesday, 2019-11-13 17:00 US/Pacific at the latest. Diagnosis: Affected users may notice elevated levels of kernel panics on nodes upgraded to one of the affected versions listed above. Workaround: Users seeing this issue can downgrade to a previous release (not listed in the affected versions above). Users on a Release Channel affected by this issue should reach out to support for assistance with downgrading their nodes. |
| | 7 Nov 2019 | 14:49 PST | Description: This issue was downgraded to an orange category Service Disruption as the number of projects actually affected is very low. Mitigation work is currently underway by our engineering team and is expected to completed by early next week. At this time the following versions are still affected: 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4, and 1.14.8-gke.7. We will provide more information as it becomes available or by Friday, 2019-11-08 17:00 US/Pacific at the latest. Diagnosis: Affected users may notice elevated levels of kernel panics on nodes upgraded to one of the affected versions listed above. Workaround: Users seeing this issue can downgrade to a previous release (not listed in the affected versions above). Users on a Release Channel affected by this issue should reach out to support for assistance with downgrading their nodes. |
| | 7 Nov 2019 | 08:49 PST | Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by early next week. At this time the following versions are still affected: 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4, and 1.14.8-gke.7. We will provide more information by Monday, 2019-11-11 15:00 US/Pacific. Diagnosis: Kernel panics following an upgrade in GKE version. Workaround: Users seeing this issue may go to a previous patch release (not listed in the affected versions above). |
- All times are US/Pacific