Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Cloud Machine Learning, Vertex AI Search

Vertex AI Search seeing elevated failures for 50% of queries for SearchService

Incident began at 2024-10-01 09:28 and ended at 2024-10-01 10:03 (all times are US/Pacific).

Previously affected location(s)

Global

Date Time Description
1 Oct 2024 14:40 PDT

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 01 October, 2024 09:28

Incident End: 01 October, 2024 10:03

Duration: 35 minutes

Affected Services and Features:

Vertex AI Search for Retail

Regions/Zones:

Global

Description:

Vertex AI Search for Retail queries on the Search API experienced elevated error rates and latencies for a duration of 35 minutes. Preliminary analysis indicates the root cause is an issue introduced by a new version release of an internal service which the Search API relies on which caused failures on our Search API.

The issue was mitigated by rolling out a patch for our internal service to resolve the disruption.

Customer Impact:

Vertex AI Search for Retail customers experienced elevated error rates and elevated latencies for a subset of queries.

Additional Information:

At the time of the outage, we believed that this issue was also impacting Recommendation AI. Upon further investigation, we concluded that this issue had no impact on Recommendation AI.
1 Oct 2024 11:14 PDT

We experienced an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific.

Our monitoring indicates that the errors have stopped occurring as of 2024-10-01 10:05 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
1 Oct 2024 10:37 PDT

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService

Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific.

Our engineering team continues to investigate the issue with the highest priority

We will provide an update by Tuesday, 2024-10-01 12:00 US/Pacific with current details.

Diagnosis: Customers may observe elevated errors while performing search for products.

Workaround: None at this time.
1 Oct 2024 10:35 PDT

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService

Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific.

Our engineering team continues to investigate the issue with the highest priority

We will provide an update by Tuesday, 2024-10-01 12:00 US/Pacific with current details.

Diagnosis: Customers may observe elevated errors while performing search for products.

Workaround: None at this time.
1 Oct 2024 10:28 PDT

Summary: Vertex AI Search seeing elevated failures for 50% of queries for SearchService

Description: We are experiencing an issue with Vertex AI Search, Recommendation AI beginning on Tuesday, 2024-10-01 09:29 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-10-01 11:45 US/Pacific with current details.

Diagnosis: Customers may observe elevated errors while performing search for products.

Workaround: None at this time.