Service Health
-
-
-
-
Incident affecting Vertex Gemini API
We are experiencing an issue with Vertex Gemini API in Asia
Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Seoul (asia-northeast3)Mumbai (asia-south1)Singapore (asia-southeast1)
| Date | Time | Description | |
|---|---|---|---|
| | 19 Dec 2024 | 15:28 PST | Mini Incident ReportWe apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support . (All Times US/Pacific) Incident Start: 19 December, 2024 08:15 Incident End: 19 December, 2024 10:55 Duration: 2 hours, 40 minutes Affected Services and Features: Vertex Gemini API Regions/Zones: multi-region:asia Description: Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process. The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident. Customer Impact: Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%. |
| | 19 Dec 2024 | 11:05 PST | The issue with Vertex Gemini API has been resolved for all affected projects as of Thursday, 2024-12-19 10:56 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| | 19 Dec 2024 | 10:47 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: Mitigation work is still underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-12-19 12:00 US/Pacific. Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: Customers should try moving traffic to any region outside of Asia. |
| | 19 Dec 2024 | 09:55 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2024-12-19 11:00 US/Pacific. Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: Customers should try moving traffic to any region outside of Asia. |
| | 19 Dec 2024 | 09:37 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: We are experiencing an issue with Vertex Gemini API. Our engineering team has identified the cause of the issue and identifying a mitigation strategy. We will provide an update by Thursday, 2024-12-19 10:30 US/Pacific with current details. Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: Customers should try moving traffic to any region outside of Asia. |
| | 19 Dec 2024 | 09:23 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: We are experiencing an issue with Vertex Gemini API. Our engineering team continues to investigate the root cause of the issue and identify a mitigation strategy. We will provide an update by Thursday, 2024-12-19 10:00 US/Pacific with current details. Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: Customers should try moving traffic to any region outside of Asia. |
| | 19 Dec 2024 | 08:58 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: We are experiencing an issue with Vertex Gemini API. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-12-19 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: None at this time. |
| | 19 Dec 2024 | 08:57 PST | Summary: We are experiencing an issue with Vertex Gemini API in Asia Description: We are experiencing an issue with Vertex Gemini API. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2024-12-19 09:30 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers impacted by this issue may see 5xx errors. Workaround: None at this time. |
- All times are US/Pacific