Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Recommendation AI, Cloud Machine Learning

We are investigating an Issue with Vertex AI Search

Incident began at 2023-09-07 08:14 and ended at 2023-09-08 22:03 (all times are US/Pacific).

Previously affected location(s)

Global

Date Time Description
8 Sep 2023 22:03 PDT

The issue with Vertex AI Search has been resolved for all affected projects as of Friday, 2023-09-08 22:02 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

8 Sep 2023 19:15 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.

Engineers are now working to clean up indexes that were created for non-canonical urls.

Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Friday, 2023-09-08 19:00 US/Pacific.

We will provide an update by Friday, 2023-09-08 22:30 US/Pacific with current details.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.

8 Sep 2023 13:14 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.

Engineers are now working to clean up indexes that were created for non-canonical urls.

Mitigation work is currently underway by our engineering team. The mitigation is expected to complete by Friday, 2023-09-08 19:00 US/Pacific.

We will provide an update by Friday, 2023-09-08 19:30 US/Pacific with current details.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.

8 Sep 2023 09:16 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.

Engineers are now working to clean up indexes that were created for non-canonical urls.

The mitigation is expected to complete by Friday, 2023-09-08 19:00 US/Pacific.

We will provide an update by Friday, 2023-09-08 19:30 US/Pacific with current details.

Mitigation work is currently underway by our engineering team.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.

7 Sep 2023 12:24 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.

Engineers are now working to clean up indexes that were created for non-canonical urls.

We do not have an ETA for full resolution at this point.

We will provide an update by Friday, 2023-09-08 10:00 US/Pacific with current details.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.

7 Sep 2023 11:24 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Engineers have mitigated the issue that was causing increased indexing for Vertex AI Search.

Engineers are now working to clean up indexes that were created for non-canonical urls.

We do not have an ETA for full resolution at this point.

We will provide an update by Thursday, 2023-09-07 13:00 US/Pacific with current details.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.

7 Sep 2023 09:53 PDT

Summary: We are investigating an Issue with Vertex AI Search

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-09-07 12:00 US/Pacific.

Diagnosis: Customers might see an increase in the cost of indexing starting on 2 September 2023. Impact to customer billing related to this issue will be corrected once the underlying issue has been mitigated.

There is the potential for blocked url indexing pipelines due to quotas being reached.

Additionally, searches may return duplicate results for the same webpage with different URLs.

Workaround: Customers can increase quotas and retrigger indexing to unblock pipelines.