Service Health

GKE service is unaffected. Mitigation for issue with Anthos On-Prem admin workstation access, is underway.

Fix:

• Release 1.8.2 with the fix delivered.

ETA for the hotfix releases:

• Hotfix for 1.7.3 is expected to be released by Friday (9/03).

For workaround procedures, please refer to the previous notification down below.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

No further updates will be provided here. We thank you for your patience while we're working on resolving the issue.

Summary: Anthos On-Prem customers unable to login to admin workstation (starting 2021-08-25 for customers using Anthos OP v1.7.2)

Description: Mitigation work is currently underway by our engineering team.

ETA for the releases:

• Release 1.8.2 with the fix is expected to be next Monday (8/29).
• And hotfix for 1.7.3 is expected to be next Thursday (9/02).

We will provide an update on Thursday 2021-09-02 by 17:00 US/Pacific.

Diagnosis: Customers are unable to login to their admin workstation in both their prod and non-prod environments.

Workaround: Mitigation steps for the admin workstation: Use a temporary VM to perform the following steps. If you don’t have a VM, you can create an admin workstation at 1.7.1-gke.4 as the temporary VM.

1)Ensure the VM and the problematic 1.7.2-gke.2 admin workstation are in power off state.

2)Attach the boot disk of the problematic admin workstation to the VM. The boot disk is the one with the label Hard disk 1.

3)Mount the boot disk inside the VM. sudo mkdir -p /mnt/boot-disk sudo mount /dev/sdc1 /mnt/boot-disk, assuming the boot disk is identified as dev/sdc1.

4)Set the ubuntu user expiration date to unlimited. sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

5)Shutdown the temporary VM.

6)Power on the admin workstation. You should be able to SSH to the admin workstation as usual.

7)[Clean up] Delete the temporary VM.

Note that this fix will break the CIS benchmark rule[1] 5.4.1.1 Ensure password expiration is 365 days or less.

[1]https://cloud.google.com/anthos/clusters/docs/on-prem/1.7/concepts/cis-ubuntu-benchmark

Mitigation steps for non admin workstation nodes (including cluster nodes and Seesaw VM):

Login onto each node (or use SSH) to run the following command on the node sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

Note that if the non admin workstation nodes are not SSHable due to the error, you will have to use the mitigation step for admin workstation.

Summary: Anthos On-Prem customers unable to login to admin workstation (starting 2021-08-25 for customers using Anthos OP v1.7.2)

Description: Mitigation work is currently underway by our engineering team.

ETA for the releases:

• Release 1.8.2 with the fix is aiming next Monday (8/29).
• And hotfix for 1.7.3 is also aiming next week.

We will provide more information by Thursday, 2021-08-26 17:00 US/Pacific.

Diagnosis: Customers are unable to login to their admin workstation in both their prod and non-prod environments.

Workaround: Mitigation steps for the admin workstation: Use a temporary VM to perform the following steps. If you don’t have a VM, you can create an admin workstation at 1.7.1-gke.4 as the temporary VM.

1)Ensure the VM and the problematic 1.7.2-gke.2 admin workstation are in power off state.

2)Attach the boot disk of the problematic admin workstation to the VM. The boot disk is the one with the label Hard disk 1.

3)Mount the boot disk inside the VM. sudo mkdir -p /mnt/boot-disk sudo mount /dev/sdc1 /mnt/boot-disk, assuming the boot disk is identified as dev/sdc1.

4)Set the ubuntu user expiration date to unlimited. sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

5)Shutdown the temporary VM.

6)Power on the admin workstation. You should be able to SSH to the admin workstation as usual.

7)[Clean up] Delete the temporary VM.

Note that this fix will break the CIS benchmark rule[1] 5.4.1.1 Ensure password expiration is 365 days or less.

[1]https://cloud.google.com/anthos/clusters/docs/on-prem/1.7/concepts/cis-ubuntu-benchmark

Mitigation steps for non admin workstation nodes (including cluster nodes and Seesaw VM):

Login onto each node (or use SSH) to run the following command on the node sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

Note that if the non admin workstation nodes are not SSHable due to the error, you will have to use the mitigation step for admin workstation.

Summary: Anthos On-Prem customers unable to login to admin workstation (starting 2021-08-25 for customers using Anthos OP v1.7.2)

Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2021-08-26 07:25 US/Pacific.

Diagnosis: Customers are unable to login to their admin workstation in both their prod and non-prod environments.

Workaround: Mitigation steps for the admin workstation: Use a temporary VM to perform the following steps. If you don’t have a VM, you can create an admin workstation at 1.7.1-gke.4 as the temporary VM.

1)Ensure the VM and the problematic 1.7.2-gke.2 admin workstation are in power off state.

2)Attach the boot disk of the problematic admin workstation to the VM. The boot disk is the one with the label Hard disk 1.

3)Mount the boot disk inside the VM. sudo mkdir -p /mnt/boot-disk sudo mount /dev/sdc1 /mnt/boot-disk, assuming the boot disk is identified as dev/sdc1.

4)Set the ubuntu user expiration date to unlimited. sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

5)Shutdown the temporary VM.

6)Power on the admin workstation. You should be able to SSH to the admin workstation as usual.

7)[Clean up] Delete the temporary VM.

Note that this fix will break the CIS benchmark rule[1] 5.4.1.1 Ensure password expiration is 365 days or less.

[1]https://cloud.google.com/anthos/clusters/docs/on-prem/1.7/concepts/cis-ubuntu-benchmark

Mitigation steps for non admin workstation nodes (including cluster nodes and Seesaw VM): Login onto each node (or use SSH) to run the following command on the node sudo chroot /mnt/boot-disk chage -M 99999 ubuntu

Note that if the non admin workstation nodes are not SSHable due to the error, you will have to use the mitigation step for admin workstation.