Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 11 December 2024 17:15

Incident End: 13 December 2024 06:03

Duration: 1 day, 12 hours, 48 minutes

Affected Services and Features:

Vertex Gemini API

Regions/Zones: multiregion-us

Description:

The Vertex Gemini API experienced elevated latency and errors for the gemini-1.5-flash-002 model in multiregion-us for a duration of 1 day, 12 hours, and 48 minutes. From preliminary analysis, the root cause of the issue was a significant increase in decode processing load stemming from a large spike in response content heavy traffic.

Google engineers mitigated the issue by increasing processing capacity and updating a resource allocation configuration to alleviate stress on processing.

Customer Impact:

Customers would have experienced increased latency or errors for the gemini-1.5-flash-002 model in multiregion-us.

The issue with Vertex Gemini API has been resolved for all affected users as of Friday, 2024-12-13 09:00 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Vertex Gemini API serving the gemini-1.5-flash model may experience elevated latency or errors in multiregions/us

Description: We are experiencing an issue with Vertex Gemini API serving the gemini-1.5-flash model beginning on Wednesday, 2024-12-11 18:00 US/Pacific.

Our engineering team has mitigated the issue and are showing signs of recovery. We will continue to monitor the service for stability.

We will provide an update by Friday, 2024-12-13 11:00 US/Pacific with current details.

Diagnosis: Impacted customers may experience elevated latency or errors.

Workaround: None at this time.

Summary: Vertex Gemini API serving the gemini-1.5-flash model may experience elevated latency or errors in multiregions/us

Description: We are experiencing an issue with Vertex Gemini API serving the gemini-1.5-flash model beginning on Wednesday, 2024-12-11 18:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-12-13 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated latency or errors.

Workaround: None at this time.

Summary: Vertex Gemini API serving the gemini-1.5-flash model may experience elevated latency or errors in multiregions/us

Description: We are experiencing an issue with Vertex Gemini API serving the gemini-1.5-flash model beginning on Wednesday, 2024-12-11 18:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-12-13 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated latency or errors.

Workaround: None at this time.

Summary: Vertex Gemini API serving the gemini-1.5-flash model may experience elevated latency or errors in multiregions/us

Description: We are experiencing an issue with Vertex Gemini API serving the gemini-1.5-flash model beginning on Wednesday, 2024-12-11 18:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-12-13 06:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Impacted customers may experience elevated latency or errors.

Workaround: None at this time.