Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

First Impact Window Start: 20 March 2024 09:58 First Impact Window end: 20 March 2024 11:40 Duration: 1 hour, 42 minutes

Second Impact Window Start: 20 March 2024 14:15 Second Impact Window end: 20 March 2024 14:45 Duration: 30 minutes

Cumulative impact duration: 2 hours, 12 minutes

Affected Services and Features:

• Dialogflow ES
• Dialogflow CX
• Cloud Speech-to-Text
• Agent Assist

Regions/Zones:

• Global
• Regions: us-east1, us-central1, us-west1, northamerica-northeast1 [1]

Description:

Dialogflow ES, Dialogflow CX, Cloud Speech-to-Text, and Agent Assist experienced two periods of elevated errors for streaming data plane traffic for durations of 1 hours, 42 minutes and 30 minutes, respectively. From preliminary analysis, the root cause of the issue was due to a recent configuration change for an internal critical dependency that serves as a backend gateway for the affected products.

Customer Impact:

Impacted customers encountered multiple periods of internal or unavailable errors for streaming API actions (StreamingAnalyzeContent, StreamingDetectIntent).

• 20 March 2024 09:58 - 11:40 US/Pacific (1 hour, 42 minutes)
• 20 March 2024 14:15 - 14:45 US/Pacific (30 minutes)

Additional details:

On 20 March 2024, at 11:40 US/Pacific, Google engineers reversed the configuration change to the internal critical dependency, which temporarily alleviated the impact while a permanent solution was being developed.

At 14:15 US/Pacific, a previously scheduled rollout for the dependency service completed, unintentionally reverting the temporary mitigation that engineers had put in place. Engineers reapplied the temporary mitigation at 14:45 US/Pacific.

On 21 March 2024, at 16:31 US/Pacific, engineers effectively implemented a new version of the dependency service that includes the necessary mitigation measures, successfully preventing any further regressions.

Reference

• [1] https://cloud.google.com/dialogflow/cx/docs/concept/region#avail

The issue with Agent Assist, Dialogflow CX, Dialogflow ES, Speech-to-Text has been resolved for all affected users as of Thursday, 2024-03-21 16:30 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: A temporary mitigation is currently in place. Currently, our engineers are rolling out a permanent fix.

We will provide an update by Thursday, 2024-03-21 18:00 US/Pacific with current details.

We apologize for any continued inconvenience this may cause.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: A temporary mitigation is currently in place. In the meanwhile, our engineers continue to investigate a permanent fix.

We will provide an update by Thursday, 2024-03-21 15:30 US/Pacific with current details.

We apologize for any continued inconvenience this may cause.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: A temporary mitigation is currently in place. In the meanwhile, our engineers are still working on a permanent fix.

We will provide an update by Thursday, 2024-03-21 11:30 US/Pacific with current details.

We apologize for any continued inconvenience this may cause.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: A temporary mitigation is currently in place. In the meanwhile, our engineers are still working on a permanent fix.

We will provide an update by Thursday, 2024-03-21 10:00 US/Pacific with current details.

We apologize for any continued inconvenience this may cause.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: Our engineering team has implemented a temporary mitigation that will require periodic manual intervention over the next 12 hours to maintain service stability.

We will provide an update by Thursday, 2024-03-21 07:00 US/Pacific with current details.

We apologize for any continued inconvenience this may cause.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: Issues with Agent Assist, Dialogflow CX, Dialogflow ES, & Cloud Speech-to-Text

Description: Our engineering team identified the issue and implemented a fix. Error rates are down significantly, and we continue to monitor for stability.

We will provide an update by Wednesday, 2024-03-20 16:15 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.

Summary: We are experiencing issues with Agent Assist, Dialogflow CX, Dialogflow ES & Cloud Speech-to-Text

Description: We are experiencing an issue with Dialogflow CX, Dialogflow ES, Agent Assist & Cloud Speech-to-Text beginning at Wednesday, 2024-03-20 14:15 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-03-20 15:15 US/Pacific with current details.

Diagnosis: The impacted customers would encounter internal or unavailable errors while using streaming APIs (StreamingAnalyzeContent, StreamingDetectIntent).

Workaround: None at this time.