Google Cloud Service Health

Google Cloud Service Health
Incidents
Cloud TPU Service Activation is impacted.

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Cloud Machine Learning, Vertex AI Online Prediction, Vertex AI Training

Cloud TPU Service Activation is impacted.

Incident began at 2024-08-07 14:53 and ended at 2024-08-07 18:44 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

	Date	Time	Description
	7 Aug 2024	18:44 PDT	The issue with Cloud TPU Services, Vertex AI Online Prediction and Vertex AI Training has been resolved for all affected customers as of Wednesday, 2024-08-07 17:05 US/Pacific. We thank you for your patience while we worked on resolving the issue.
	7 Aug 2024	17:06 PDT	Summary: Cloud TPU Service Activation is impacted. Description: We are investigating issues where customers using TPUs through Vertex AI Service or Cloud TPU Service/API directly may be experiencing API activation failures. This can happen for new customers or existing customers who disable and reactivate the TPU Service beginning on Wednesday, 2024-08-07 12:00 US/Pacific. Our engineering team has rolled out a mitigation and is progressing as expected. We will provide an update by Wednesday, 2024-08-07 19:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Impacted customers would not be able to enable Cloud TPU APIs in their projects and would encounter an error message that states, e.g. "Failed to upload model. Contact Vertex AI." We continue to investigate impact to other Google Cloud Services and products. Workaround: None at this time.

Date

Time

Description

7 Aug 2024

18:44 PDT

The issue with Cloud TPU Services, Vertex AI Online Prediction and Vertex AI Training has been resolved for all affected customers as of Wednesday, 2024-08-07 17:05 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

7 Aug 2024

17:06 PDT

Summary: Cloud TPU Service Activation is impacted.

Description: We are investigating issues where customers using TPUs through Vertex AI Service or Cloud TPU Service/API directly may be experiencing API activation failures. This can happen for new customers or existing customers who disable and reactivate the TPU Service beginning on Wednesday, 2024-08-07 12:00 US/Pacific.

Our engineering team has rolled out a mitigation and is progressing as expected.

We will provide an update by Wednesday, 2024-08-07 19:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers would not be able to enable Cloud TPU APIs in their projects and would encounter an error message that states, e.g. "Failed to upload model. Contact Vertex AI."

We continue to investigate impact to other Google Cloud Services and products.

Workaround: None at this time.

All times are US/Pacific