Incident affecting Vertex AI Online Prediction, Vertex AI Batch Prediction, Cloud Machine Learning
Multi Region: Vertex AI Batch Prediction & Vertex AI Online Prediction Model Deployment Issues
Incident began at 2023-05-11 02:04 and ended at 2023-05-11 19:00 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Sydney (australia-southeast1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)
| ||11 May 2023||19:00 PDT|| |
The issue with Vertex AI Batch Prediction, Vertex AI Online Prediction has been resolved for all affected users as of Thursday, 2023-05-11 18:59 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
| ||11 May 2023||13:14 PDT|| |
Summary: Multi Region: Vertex AI Batch Prediction & Vertex AI Online Prediction Model Deployment Issues
Description: Our engineers have identified the issue and a fix is being rolled out. Online Prediction is mitigated as of 2023-05-11 12:30 US/Pacific. Batch Prediction is expected to rollout the fix by 2023-05-11 18:00 US/Pacific.
The issue is currently contained in the following four regions: us-west2, europe-west3, asia-east2, us-east1.
We will provide more information by Thursday, 2023-05-11 19:00 US/Pacific.
Diagnosis: Customers are unable to deploy new models on Online Prediction, when a new GKE cluster is created. Some customers Batch Prediction jobs are experiencing timeouts.
Workaround: None at this time.
- All times are US/Pacific