Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud Platform. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit cloud.google.com.

Google Compute Engine Incident #20012

Elevated frequency of Host Maintenance events on GCE instances with an attached GPU(s) and SSD(s)

Incident began at 2020-11-10 13:41 and ended at 2020-11-11 11:01 (all times are US/Pacific).

Date Time Description
Nov 11, 2020 11:01

The issue with Google Compute Engine instances with an attached GPU(s) and SSD(s) is believed to be affecting a very small number of projects and our Engineering Team continues to work on it.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

No further updates will be provided here.

We thank you for your patience while we're working on resolving the issue.

The issue with Google Compute Engine instances with an attached GPU(s) and SSD(s) is believed to be affecting a very small number of projects and our Engineering Team continues to work on it.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

No further updates will be provided here.

We thank you for your patience while we're working on resolving the issue.

Nov 10, 2020 16:04

Description: Mitigation work is still underway by our engineering team. Further investigation of current impact and mitigation timeline is ongoing.

We will provide more information by Wednesday, 2020-11-11 13:00 US/Pacific.

Diagnosis: Affected customers will experience elevated frequency of Host Maintenance events on GCE instances with an attached GPU(s) and SSD(s).

Workaround: Temporarily switch to use V100 GPU's which are unaffected by this issue.
https://cloud.google.com/compute/docs/gpus

Description: Mitigation work is still underway by our engineering team. Further investigation of current impact and mitigation timeline is ongoing.

We will provide more information by Wednesday, 2020-11-11 13:00 US/Pacific.

Diagnosis: Affected customers will experience elevated frequency of Host Maintenance events on GCE instances with an attached GPU(s) and SSD(s).

Workaround: Temporarily switch to use V100 GPU's which are unaffected by this issue.
https://cloud.google.com/compute/docs/gpus

Nov 10, 2020 14:33

Description: We are experiencing an issue with Google Compute Engine beginning in 2020-08. A firmware rollout is being created that should address the issue.

The rollout is currently expected to complete next week, but mitigation efforts are still ongoing.

We will provide more information by Tuesday, 2020-11-10 16:30 US/Pacific.

Diagnosis: Affected customers will experience elevated frequency of Host Maintenance events on GCE instances with an attached GPU(s) and SSD(s).

Workaround: Temporarily switch to use V100 GPU's which are unaffected by this issue.
https://cloud.google.com/compute/docs/gpus

Description: We are experiencing an issue with Google Compute Engine beginning in 2020-08. A firmware rollout is being created that should address the issue.

The rollout is currently expected to complete next week, but mitigation efforts are still ongoing.

We will provide more information by Tuesday, 2020-11-10 16:30 US/Pacific.

Diagnosis: Affected customers will experience elevated frequency of Host Maintenance events on GCE instances with an attached GPU(s) and SSD(s).

Workaround: Temporarily switch to use V100 GPU's which are unaffected by this issue.
https://cloud.google.com/compute/docs/gpus