Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Vertex AI Online Prediction, Cloud Machine Learning

Global: Vertex AI Online Prediction Is Experiencing Increased Error Rates

Incident began at 2022-06-02 10:10 and ended at 2022-06-02 14:30 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)Sydney (australia-southeast1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Oregon (us-west1)Los Angeles (us-west2)

Date Time Description
3 Jun 2022 13:06 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case https://cloud.google.com/support or help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 02 June 2022 10:10 US/Pacific

Incident End: 02 June 2022 14:30 US/Pacific

Duration: 4 hours, 20 minutes

Affected Services and Features:

Vertex AI Online Prediction

Regions/Zones: Global

Description:

Vertex AI Online Prediction experienced increased error rates from 30% up to 100% per region depending on user usage patterns for a duration of 4 hours, 20 minutes. From preliminary analysis, the root cause of the issue was that Vertex Prediction Endpoints were globally marked as deleted due to faulty resource cleanup process. The service fully recovered when the Vertex Prediction Endpoints were restored.

Customer Impact:

Affected customers may have experienced:

  • All pre-existing Vertex models undeployed on Vertex AI Endpoints
  • Empty responses when listing the deployed models
  • Runtime exceptions and general errors on Predict and Explain requests
  • Quota failure when trying to re-deploy models
2 Jun 2022 15:24 PDT

The issue with Vertex AI Online Prediction has been resolved for all affected users as of Thursday, 2022-06-02 15:21 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.
2 Jun 2022 14:51 PDT

Summary: Global: Vertex AI Online Prediction Is Experiencing Increased Error Rates

Description: We believe the issue with Vertex AI Online Prediction is partially resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2022-06-02 16:10 US/Pacific with current details.

Diagnosis: For affected customers: When listing the deployed models in Endpoints, the list will be empty and Predict and Explain requests would fail.

Workaround: None at this time.
2 Jun 2022 14:50 PDT

Summary: Global: Vertex AI Online Prediction Is Experiencing Increased Error Rates

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2022-06-02 15:07 US/Pacific.

We will provide more information by Thursday, 2022-06-02 15:07 US/Pacific.

Diagnosis: For affected customers: When listing the deployed models in Endpoints, the list will be empty and Predict and Explain requests would fail.

Workaround: None at this time.
2 Jun 2022 14:06 PDT

Summary: Global: Vertex AI Online Prediction Is Experiencing Increased Error Rates

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2022-06-02 15:07 US/Pacific.

We will provide more information by Thursday, 2022-06-02 15:07 US/Pacific.

Diagnosis: Customers will experiences increased error rates.

Workaround: None at this time.
2 Jun 2022 13:58 PDT

Summary: Global: Vertex AI Online Prediction Is Experiencing Increased Error Rates

Description: We are experiencing an issue with Vertex AI Online Prediction beginning at Thursday, 2022-06-02 10:20 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2022-06-02 14:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers will experiences errors

Workaround: None at this time.