Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 29 March 2024 10:00

Incident End: 30 March 2024 05:27

Duration: 19 hours, 27 minutes

Affected Services and Features:

Chronicle Security

Regions/Zones:

Global

Description:

On 29 March 2024 at 10:00, Chronicle Security experienced delays in data processing for Entity Graph, BigQuery Export, User and Entity Behavior Analytics (UEBA) features including metrics and entity risk scores, and Log Export. From preliminary analysis, the root cause was related to a recent rollout that improperly configured an access transparency check for some of Chronicle Security’s back-end data processing pipelines.

Google Engineers completed a roll back of the change that introduced the issue at 18:09, which allowed the affected data pipelines to run. All of the affected data pipelines had successfully completed by 30 March 2024 at 05:26, fully resolving impact.

Customer Impact:

Customers experienced delays in data availability in all regions for multiple Chronicle system components and product features, including Entity Graph, BigQuery Export, UEBA Simple Analytics, and Log Export.

The issue with Chronicle Security has been resolved for all affected users as of Saturday, 2024-03-30 05:26 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Delays in Data Availability

Description: The issue with Chronicle Security is partially resolved. The issue is fully resolved for all Chronicle regions except the US where it is partially resolved.

The remaining impact includes: some legacy udm event exports to BigQuery/Looker are delayed in the US for some customers, the Entity Graph table/explore is delayed in the US.

We will provide more information by Saturday, 2024-03-30 08:00 US/Pacific.

Diagnosis: Customers may experience residual delays in data freshness in some regions for log export, Entity graph and User and Entity Behavioral Analytics(UEBA) Simple Analytics.

Workaround: None at this time.

Summary: Chronicle Delays in Data Availability

Description: We believe the issue with Chronicle Security is partially resolved.

The issue has been mitigated in most regions. Some regions have rolled back but not fully completed dependent processing for Entity Graph and some tables/explores for BigQuery and Looker.

We will provide more information by Friday, 2024-03-29 23:30 US/Pacific.

Diagnosis: Customers may experience residual delays in data freshness in some regions for log export, Entity graph and UEBA Simple Analytics.

Workaround: None at this time.

Summary: Chronicle Delays in Data Availability

Description: Mitigation work is currently underway by our engineering team.

We have mitigated UEBA metrics in all regions except Americas, Entity Graph in all regions except Americas and Europe, and BigQuery data export for UDM aggregates, IOCs, and Entity Graph in most regions. Mitigations are under way for the remaining regions.

We will provide more information by Friday, 2024-03-29 18:30 US/Pacific.

Diagnosis: Customers may experience delays in data availability in all regions for multiple Chronicle system components and product features (Entity Graph, UEBA Simple Analytics, Log Export, Indexing).

Workaround: None at this time.

Summary: Chronicle Delays in Data Availability

Description: Mitigation work is currently underway by our engineering team.

We have globally mitigated UEBA risk score. We have mitigated UEBA metrics, Entity Graph and BigQuery data export for UDM aggregates, IOCs, and Entity Graph in most regions. Mitigations are also under way for indexing metrics including prevalence and first seen (which were delayed in the Entity Graph).

We will provide more information by Friday, 2024-03-29 17:45 US/Pacific.

Diagnosis: Customers may experience delays in data availability in all regions for multiple Chronicle system components and product features (Entity Graph, UEBA Simple Analytics, Log Export, Indexing).

Workaround: None at this time.

Summary: Chronicle Delays in Data Availability

Description: Mitigation work is currently underway by our engineering team.

We have globally mitigated BigQuery data export for rule detections (for the legacy Looker connector, external Looker connector, and legacy BigQuery direct access). Mitigations are underway for additional BigQuery data exports (UDM aggregates, IOCs, Entity Graph). Mitigations are under way for Entity Graph, Log Export to GCS,

We will provide more information by Friday, 2024-03-29 17:00 US/Pacific

Diagnosis: Customers may experience delays in data availability in all regions for multiple Chronicle system components and product features (Entity Graph, UEBA Simple Analytics, Log Export).

Workaround: None at this time.

Summary: Chronicle Delays in Data Availability

Description: We are experiencing an issue with Chronicle Security beginning at Friday, 2024-03-29 10:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2024-03-29 17:35 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers may experience delays in data availability in all regions for multiple Chronicle system components and product features (Entity Graph, BigQuery Export, UEBA Simple Analytics, Log Export).

Workaround: None at this time.