Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support (All Times US/Pacific)

Incident Start: 25 July 2023 at 06:30

Incident End: 29 July 2023 at 08:03

Duration: 4 days, 1 hour, 33 minutes

Affected Services and Features:

Chronicle Security

Regions/Zones: Multi-regions: us

Description:

Starting on Tuesday, 25 July at 06:30, Chronicle Security began experiencing a slow down in data processing in the US region. This resulted in stale data for Unified Data Model (UDM) [1] Search and delayed threat detections. Chronicle's data processing returned to normal for new events on Wednesday, 27 July at 13:45, with the last events in the incident window processed by customer rules by Saturday, 29 July at 08:03. From a preliminary analysis, the root cause was a surge in traffic from a single customer that occurred on Saturday 22 July. The traffic surge was exacerbated by a pipeline that did not have sufficient rate limiting and ultimately overloaded our persistence layer.

Google engineers mitigated the issue by temporarily limiting the traffic from the high volume customer, removing inter-dependency from as many pipelines as possible, and by disabling several non-critical pipelines.

Customer Impact:

All features in Chronicle Security were still working but the impacted users could have got stale data while searching for UDM events and may have experienced missed threat detections for recently ingested telemetry.

[1] - https://cloud.google.com/chronicle/docs/event-processing/udm-overview

The issue with Chronicle Security has been resolved for all affected users as of Saturday, 2023-07-29 08:10 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: The issue with Chronicle Security where data was stale on instances is mitigated for a majority of users.

Full resolution of data staleness and the missing detections from the impact period is expected to complete by Saturday, 2023-07-29 10:00 US/Pacific.

We will provide an update by Saturday, 2023-07-29 10:15 US/Pacific with current details.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can run raw log searches and should still be able to access results. Searches with raw log should be targeted to a log type or string/regex.

Customers can also leverage indexed UDM searches to surface up to date results. Indexed UDM searches must leverage one of the following UDM fields and one of the following UDM nouns. These searches must be an exact match on a field, which means they cannot include a REGEX expression or REF list. Basic Boolean logic that combines these UDM fields with other conditions should also surface fresh data (e.g. noun.field1 AND noun.field2)

Nouns: Principal Target

Field Names: Hostname Ip Mac Asset File.md5 File.sha1 File.sha256 process.file.md5 process.file.sha1 process.file.sha256 process.parent_process.file.md5 process.parent_process.file.sha1 process.parent_process.file.sha256

Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Our engineering team is continuing mitigation efforts.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-07-27 14:00 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can run raw log searches and should still be able to access results. Searches with raw log should be targeted to a log type or string/regex.

Customers can also leverage indexed UDM searches to surface up to date results. Indexed UDM searches must leverage one of the following UDM fields and one of the following UDM nouns. These searches must be an exact match on a field, which means they cannot include a REGEX expression or REF list. Basic Boolean logic that combines these UDM fields with other conditions should also surface fresh data (e.g. noun.field1 AND noun.field2)

Nouns: Principal Target

Field Names: Hostname Ip Mac Asset File.md5 File.sha1 File.sha256 process.file.md5 process.file.sha1 process.file.sha256 process.parent_process.file.md5 process.parent_process.file.sha1 process.parent_process.file.sha256

Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Our engineering team is continuing mitigation efforts.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-07-27 12:00 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can run raw log searches and should still be able to access results. Searches with raw log should be targeted to a log type or string/regex.

Customers can also leverage indexed UDM searches to surface up to date results. Indexed UDM searches must leverage one of the following UDM fields and one of the following UDM nouns. These searches must be an exact match on a field, which means they cannot include a REGEX expression or REF list. Basic Boolean logic that combines these UDM fields with other conditions should also surface fresh data (e.g. noun.field1 AND noun.field2)

Nouns: Principal Target

Field Names: Hostname Ip Mac Asset File.md5 File.sha1 File.sha256 process.file.md5 process.file.sha1 process.file.sha256 process.parent_process.file.md5 process.parent_process.file.sha1 process.parent_process.file.sha256

Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Our engineering team is continuing to work on mitigating this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-07-26 20:30 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can run raw log searches and should still be able to access results. Searches with raw log should be targeted to a log type or string/regex.

Customers can also leverage indexed UDM searches to surface up to date results. Indexed UDM searches must leverage one of the following UDM fields and one of the following UDM nouns. These searches must be an exact match on a field, which means they cannot include a REGEX expression or REF list. Basic Boolean logic that combines these UDM fields with other conditions should also surface fresh data (e.g. noun.field1 AND noun.field2)

Nouns: Principal Target

Field Names: Hostname Ip Mac Asset File.md5 File.sha1 File.sha256 process.file.md5 process.file.sha1 process.file.sha256 process.parent_process.file.md5 process.parent_process.file.sha1 process.parent_process.file.sha256

Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Our engineering team is continuing to work on mitigating this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-07-26 17:30 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can still leverage indexed UDM searches to surface up to date results. Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Our engineering team is continuing to work on mitigating this issue.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-07-26 12:00 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can still leverage indexed UDM searches to surface up to date results. Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-07-26 09:00 US/Pacific.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can still leverage indexed UDM searches to surface up to date results. Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2023-07-25 08:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-07-26 01:45 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can still leverage indexed UDM searches to surface up to date results. Additionally the vast majority of single event rules are still being processed.

Summary: Chronicle Security customers may experience data staleness or missed detections in US regions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2023-07-25 08:30 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2023-07-26 01:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: All features in Chronicle are still working but the impacted users may get stale data while searching for UDM events, and they may experience missed threat detections for recently ingested telemetry.

All data ingested on or before 2023-07-25 08:30 US/Pacific has been processed successfully. There are gaps in UDM search for data ingested after that time. Recent telemetry for data ingested after this time may result in missed threat detections as well.

Workaround: Customers can still leverage indexed UDM searches to surface up to date results. Additionally the vast majority of single event rules are still being processed.