Service Health
-
-
-
-
Incident affecting Google Kubernetes Engine
Issues with GKE 1.20 (lower than 1.20.9-gke.2100) node pools using Docker as runtime.
Incident began at 2021-07-27 00:00 and ended at 2021-09-23 18:53 (all times are US/Pacific).
| Date | Time | Description | |
|---|---|---|---|
| | 27 Sep 2021 | 10:29 PDT | We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support (All Times US/Pacific) Incident Start: 27 July 2021, GKE clusters began being upgraded from 1.19 to 1.20 in the REGULAR release channel. Incident End: 23 September 2021, incident is mitigated by pausing automatic upgrades. Duration: 59 days Affected Services and Features: Google Kubernetes Engine (GKE) - Pods on nodes with affected versions will restart when docker restarts. Clusters on the REGULAR release channel were automatically upgraded into versions affected by this issue. Regions/Zones: Global Description: Containers within GKE cluster node pools using docker are getting restarted in the event of docker restarts. This issue affects the following node versions:
The engineering team has halted the rollout from 1.19 to 1.20 in release channels to prevent any new impact to our customers. Customer Impact: GKE cluster pods restart when docker restarts. This issue affects the following node versions:
Additional details: Customer action required: To fix this issue, either use containerd or upgrade nodes to version:
The recommended action for clusters on release channels are:
Our engineering team is currently releasing a fixed version for 1.20 for the STABLE release channel. This release is currently scheduled to come out by 8 October, 2021. |
| | 24 Sep 2021 | 12:53 PDT | GKE clusters running node pools that use docker may experience containers restarting every time docker restarts. Affected node versions:
To fix this issue, either use containerd or upgrade nodes to version:
Recommended fix per release channel: STATIC - Upgrade to 1.20.10-gke.301 or higher RAPID - N/A - All available versions have the fix REGULAR - Upgrade to 1.20.10-gke.301 STABLE - Downgrade affected nodepools to a 1.19 version Our engineering team is continuing to work on a release with the fix to the issue in a 1.20 version of the stable channel, which is scheduled to be released before the first week of October. If you have questions, please open a case with the Support Team and we will work with you until this issue is resolved. No further updates will be provided here. We thank you for your patience while we're working on resolving the issue. |
| | 23 Sep 2021 | 23:48 PDT | Summary: Issues with GKE node pools upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) using Docker as runtime. Description: We are experiencing an issue with GKE node pools using Docker as a runtime upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100). Customers may see containers restarting when docker daemon is restarted. Problematic versions:
Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue. Our engineering team continues to work on the fix. We will provide an update by Friday, 2021-09-24 15:00 US/Pacific. We apologize to all who are affected by the disruption. Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts. Workaround: Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue. |
| | 23 Sep 2021 | 21:59 PDT | Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts. Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted. Problematic versions:
Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue. Our engineering team continues to investigate the issue at the back end. We will provide an update by Friday, 2021-09-24 00:00 US/Pacific. We apologize to all who are affected by the disruption. Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts. Workaround: Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue. |
| | 23 Sep 2021 | 20:28 PDT | Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts. Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted. Problematic versions:
Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Available versions with fix:
Our engineering team continues to investigate the issue at the back end. We will provide an update by Thursday, 2021-09-23 22:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts. Workaround: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded, to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Versions above 1.20.9-gke.2100 is not impacted. |
| | 23 Sep 2021 | 20:26 PDT | Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts. Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted. Problematic versions:
Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Available versions with fix:
Our engineering team continues to investigate the issue at the back end. We will provide an update by Thursday, 2021-09-23 20:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts. Workaround: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded, to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Versions above 1.20.9-gke.2100 is not impacted. |
- All times are US/Pacific