Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Compute Engine

We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart.

Incident began at 2020-07-30 08:38 and ended at 2020-07-30 13:17 (all times are US/Pacific).

Date Time Description
30 Jul 2020 13:17 PDT

The issue with Google Compute Engine instances running RHEL and CentOS 7 and 8 is being actively investigated by Redhat.

More details on this issue are available in the following article and bugs:

Please follow our public issue tracker posting (https://issuetracker.google.com/162523000) for updates on this issue going forward. No further updates will be provided here.

30 Jul 2020 10:52 PDT

Description: We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. More details on this issue are available in the following article and bugs:

https://access.redhat.com/solutions/5272311 https://bugzilla.redhat.com/show_bug.cgi?id=1861977 (RHEL 8) https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (RHEL 7)

Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP".

This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64

Affected shim versions:

CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

We will provide an update by Thursday, 2020-07-30 14:00 US/Pacific with current details.

Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP".

This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64

Affected shim versions:

CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

30 Jul 2020 09:52 PDT

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances.

Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP".

This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64

Affected shim versions:

CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2020-07-30 11:00 US/Pacific with current details.

Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP".

This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64

Affected shim versions:

CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

30 Jul 2020 09:41 PDT

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances.

Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2020-07-30 10:30 US/Pacific with current details.

Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP".

This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64

Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run "yum downgrade shim* grub2* mokutil" to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

30 Jul 2020 09:27 PDT

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances.

Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8.

Our engineering team continues to investigate the issue. We are also preparing instructions for recovery from this issue for affected instances.

We will provide an update by Thursday, 2020-07-30 10:15 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart.

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8.

30 Jul 2020 08:49 PDT

Description: We are experiencing an issue with Google Compute Engine RHEL and CentOS 7 and 8 instances.

Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart. Customers should not update or reboot instances running RHEL or CentOS 7 and 8.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2020-07-30 09:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart.

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8.