Google Cloud Service Health

Google Cloud Service Health
Incidents
Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Available
Service information
Service disruption
Service outage

Incident affecting Cloud Machine Learning, Vertex AI Online Prediction

Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5

Incident began at 2024-08-08 00:35 and ended at 2024-08-08 17:18 (all times are US/Pacific).

Date	Time	Description
8 Aug 2024	17:18 PDT	The issue with Vertex AI Online Prediction and Tensor Processing Unit (TPU) v5e workloads has been resolved for all affected users as of Thursday, 2024-08-08 17:16 US/Pacific. We thank you for your patience while we worked on resolving the issue.
8 Aug 2024	14:57 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. A limited number of customers are experiencing interruptions while operating their Tensor Processing Unit (TPU) v5e workloads. Our engineering team continues to work towards mitigation. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-08-08 18:00 US/Pacific. We apologize to all who are affected by the disruption. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: There is no workaround at this time.
8 Aug 2024	14:16 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work continues with our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-08-08 16:00 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	12:58 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work continues with our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-08-08 14:00 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	11:52 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work continues with our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-08-08 13:00 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	11:11 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-08-08 12:00 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	09:01 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to be completed by Thursday, 2024-08-08 07:30 US/Pacific. We will provide more information by Thursday, 2024-08-08 11:00 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	07:54 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to be completed by Thursday, 2024-08-08 07:30 US/Pacific. We will provide more information by Thursday, 2024-08-08 09:30 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	06:23 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: Mitigation work is currently underway by our engineering team. The mitigation is expected to be completed by Thursday, 2024-08-08 07:30 US/Pacific. We will provide more information by Thursday, 2024-08-08 08:30 US/Pacific. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	05:52 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-08-08 08:00 US/Pacific with current details. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	03:40 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-08-08 06:00 US/Pacific with current details. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	02:23 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-08-08 04:00 US/Pacific with current details. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	01:31 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet 3.5 and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-08-08 02:30 US/Pacific with current details. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet 3.5 and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.
8 Aug 2024	01:18 PDT	Summary: Vertex AI Online Prediction experiencing elevated 500 errors for Claude Sonnet and Opus models in us-east5 Description: We are experiencing an issue with Vertex AI Online Prediction beginning on Wednesday, 2024-08-07 23:55 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-08-08 02:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Some Vertex AI Online Prediction customers using Claude Sonnet and Opus may experience elevated 500 errors from the calls to the model. Please note that other Generative AI models within Vertex AI Online Prediction are functioning normally. Workaround: None at this time.

All times are US/Pacific