Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 13 June 2024 02:36

Incident End: 13 June 2024 23:24

Duration: 20 hours, 48 minutes

Affected Services and Features:

Chronicle Security

Regions/Zones: US multi-region

Description:

Chronicle Security experienced delays with BigQuery exports, Multi event rules, and Normalization in US multi-region for a duration of 20 hours, 48 minutes.

After preliminary analysis, the root cause of the issue was identified as a temporary unavailability of a specific zone within the internal database service utilized by Chronicle. Despite Chronicle's fault-tolerant setup guaranteeing uninterrupted service availability, its performance gradually diminished over time as a result of diminished data processing capabilities.

To alleviate infrastructure load, Chronicle engineering implemented measures to delay the reprocessing of specific data. However, attempts to resume reprocessing led to further delays. Concurrently, a separate, unrelated bug caused excessive database writes, resulting in delays in data normalization. To address this issue, the data pipeline was disabled to eliminate significant delays in data normalization.

Customer Impact:

• Delays in BigQuery Export and Multi event rules processing
• Delays in late arriving enrichments and slower Unified Data Model (UDM) search.

The issue with Chronicle Security has been resolved for all affected users as of Thursday, 2024-06-13 23:24 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Security is experiencing delays with BigQuery exports, Multi event rules, and Normalization in US multi-region

Description: We have successfully mitigated the issue pertaining to delays with Normalization.

We have completed mitigations for BigQuery export delay and multi event rules, after which we have been observing a steady reduction in delays. We believe that the the issue has been mitigated for the majority of customers, and our engineering team continues working diligently to resolve any remaining issues.

We will provide more information by Friday, 2024-06-14 09:00 US/Pacific.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with BigQuery exports, Multi event rules, and Normalization in US multi-region

Description: We have successfully mitigated the issue pertaining to delays with Normalization.

We have completed mitigations for BigQuery export delay and multi event rules, after which we have been observing a steady reduction in delays. We are continuing to monitor and investigate any residual issues, and our engineering team is working diligently to resolve them.

We will provide more information by Thursday, 2024-06-13 23:30 US/Pacific.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with BigQuery exports, Multi event rules, and Normalization in US multi-region

Description: We have successfully mitigated the issue pertaining to delays with Normalization.

We have completed mitigations for BigQuery export delay and multi event rules, after which we have been observing a steady reduction in delays. We expect the any residual issues to completely subside in the next 2 hours.

We will provide more information by Thursday, 2024-06-13 20:30 US/Pacific.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with BigQuery exports, Multi event rules, and Normalization in US multi-region

Description: We have completed mitigations for BigQuery export delay and multi event rules, after which we have been observing a steady reduction in delays. We expect the delays to completely subside in the next 5 hours.

Our engineers have been able to successfully identify the reason for Normalization delays and have applied the required mitigation. We expect any related delays to be completely mitigated in the next 6 hours.

We will provide more information by Thursday, 2024-06-13 20:00 US/Pacific.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with BigQuery exports, Multi event rules, and Normalization in US multi-region

Description: We have completed mitigations for BigQuery export delay and multi event rules and we are seeing gradual reduction in delays. We expect the delays to completely subside in the next 6 hours.

The Normalization delays is being investigated by our engineering team.

We will provide more information by Thursday, 2024-06-13 15:00 US/Pacific.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with Big Query exports, Multi event rules, and Normalization in US multi-region

Description: We are experiencing an issue with Chronicle Security Big Query exports, Multi event rules, and Normalization beginning at Thursday, 2024-06-13 09:15 US/Pacific.

Our engineering team identified the root cause of the issue and are working on a mitigation.

We do not have an ETA for mitigation at this point.

We will provide an update by Thursday, 2024-06-13 13:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export, in the processing of Multi event rules, and delays in Normalization (The process of converting a raw log to a UDM record) in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.

Summary: Chronicle Security is experiencing delays with Big Query exports and Multi event rules in US multi-region

Description: We are experiencing an issue with Chronicle Security Big Query exports and Multi event rules beginning at Thursday, 2024-06-13 09:15 US/Pacific.

Our engineering team identified the root cause of the issue and are working on a mitigation.

We do not have an ETA for mitigation at this point.

We will provide an update by Thursday, 2024-06-13 12:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis:

• Chronicle security is experiencing delays in Big Query Export and in the processing of Multi event rules in the US multi region.

• Single event rules are still being processed and Unified Data Model (UDM) Search and Raw Log Search are still available.

Workaround: None at this time.