Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Kubernetes Engine

Issues with GKE 1.20 (lower than 1.20.9-gke.2100) node pools using Docker as runtime.

Incident began at 2021-07-27 00:00 and ended at 2021-09-23 18:53 (all times are US/Pacific).

Date Time Description
27 Sep 2021 10:29 PDT

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 27 July 2021, GKE clusters began being upgraded from 1.19 to 1.20 in the REGULAR release channel. Incident End: 23 September 2021, incident is mitigated by pausing automatic upgrades.

Duration: 59 days

Affected Services and Features:

Google Kubernetes Engine (GKE) - Pods on nodes with affected versions will restart when docker restarts. Clusters on the REGULAR release channel were automatically upgraded into versions affected by this issue.

Regions/Zones: Global

Description:

Containers within GKE cluster node pools using docker are getting restarted in the event of docker restarts. This issue affects the following node versions:

  • All 1.20 versions below 1.20.9-gke.2100
  • All 1.21 versions below 1.21.3-gke.1600

The engineering team has halted the rollout from 1.19 to 1.20 in release channels to prevent any new impact to our customers.

Customer Impact:

GKE cluster pods restart when docker restarts. This issue affects the following node versions:

  • All 1.20 versions below 1.20.9-gke.2100
  • All 1.21 versions below 1.21.3-gke.1600

Additional details:

Customer action required: To fix this issue, either use containerd or upgrade nodes to version:

  • 1.20: 1.20.9-gke.2100 or higher
  • 1.21: 1.21.3-gke.1600 or higher

The recommended action for clusters on release channels are:

  • STATIC - Upgrade to 1.20.10-gke.301 or higher
  • RAPID - N/A - All available versions have the fix
  • REGULAR - Upgrade to 1.20.10-gke.301
  • STABLE - Downgrade affected nodepools to a 1.19 version

Our engineering team is currently releasing a fixed version for 1.20 for the STABLE release channel. This release is currently scheduled to come out by 8 October, 2021.

24 Sep 2021 12:53 PDT

GKE clusters running node pools that use docker may experience containers restarting every time docker restarts. Affected node versions:

  • All 1.20 versions below 1.20.9-gke.2100
  • All 1.21 versions below 1.21.3-gke.1600

To fix this issue, either use containerd or upgrade nodes to version:

  • 1.20: 1.20.9-gke.2100 or higher
  • 1.21: 1.21.3-gke.1600 or higher

Recommended fix per release channel: STATIC - Upgrade to 1.20.10-gke.301 or higher RAPID - N/A - All available versions have the fix REGULAR - Upgrade to 1.20.10-gke.301 STABLE - Downgrade affected nodepools to a 1.19 version

Our engineering team is continuing to work on a release with the fix to the issue in a 1.20 version of the stable channel, which is scheduled to be released before the first week of October. If you have questions, please open a case with the Support Team and we will work with you until this issue is resolved.

No further updates will be provided here.

We thank you for your patience while we're working on resolving the issue.

23 Sep 2021 23:48 PDT

Summary: Issues with GKE node pools upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) using Docker as runtime.

Description: We are experiencing an issue with GKE node pools using Docker as a runtime upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100). Customers may see containers restarting when docker daemon is restarted.

Problematic versions:

  • All 1.20.0.* - 1.20.8.*
  • 1.20.9.* below 1.20.9-gke.1600

Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue.

Our engineering team continues to work on the fix.

We will provide an update by Friday, 2021-09-24 15:00 US/Pacific.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts.

Workaround: Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue.

23 Sep 2021 21:59 PDT

Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts.

Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted.

Problematic versions:

  • All 1.20.0.* - 1.20.8.*
  • 1.20.9.* below 1.20.9-gke.1600

Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue.

Our engineering team continues to investigate the issue at the back end.

We will provide an update by Friday, 2021-09-24 00:00 US/Pacific.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts.

Workaround: Customers can upgrade to 1.20.9-gke.2100+ or 1.20.10-gke.301 to mitigate the issue.

23 Sep 2021 20:28 PDT

Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts.

Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted.

Problematic versions:

  • All 1.20.0.* - 1.20.8.*
  • 1.20.9.* below 1.20.9-gke.1600

Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded to a problematic version, upgrade master and nodes to 1.20.10-gke.301.

Available versions with fix:

  • 1.20.9-gke.210
  • 1.20.10+

Our engineering team continues to investigate the issue at the back end.

We will provide an update by Thursday, 2021-09-23 22:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts.

Workaround: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100)

To mitigate a cluster already upgraded, to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Versions above 1.20.9-gke.2100 is not impacted.

23 Sep 2021 20:26 PDT

Summary: Specific GKE Clusters upgraded from 1.19 to 1.20 (lower than 1.20.9-gke.2100) may experience pods restarting when Docker restarts.

Description: We are experiencing an issue with GKE clusters upgraded from 1.19 to 1.20(lower than 1.20.9-gke.2100). Clusters may see pods restarting when Dockers are restarted.

Problematic versions:

  • All 1.20.0.* - 1.20.8.*
  • 1.20.9.* below 1.20.9-gke.1600

Action required: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100) To mitigate a cluster already upgraded to a problematic version, upgrade master and nodes to 1.20.10-gke.301.

Available versions with fix:

  • 1.20.9-gke.210
  • 1.20.10+

Our engineering team continues to investigate the issue at the back end.

We will provide an update by Thursday, 2021-09-23 20:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may see GKE Cluster pods restarting when Docker restarts.

Workaround: Do not upgrade from 1.19 -> 1.20.x (lower than 1.20.9-gke.2100)

To mitigate a cluster already upgraded, to a problematic version, upgrade master and nodes to 1.20.10-gke.301. Versions above 1.20.9-gke.2100 is not impacted.