Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 23 April 2024 08:46

Incident End: 24 April 2024 00:23

Duration: 15 hours, 37 minutes

Affected Services and Features: Chronicle SIEM

Regions/Zones: US multi-region

Description:

Some Chronicle customers in the US may have noticed ingestion delays for Third Party Feeds. Delays in the ingested data surfaced delays in detections and impacted other downstream product features. The impact lasted for 15 hours, 37 minutes.

From preliminary analysis, the root cause of the issue is that the service responsible for API transfers was rendered partially unavailable due to a combination of crashes and longer cache refresh times due to overload.

Customer Impact:

• Some Chronicle customers in the US may have noticed ingestion delays for Third Party API Feeds [1]
• Any systematic detections dependent on these logs would also be delayed proportionately.
• The data would automatically catch up post mitigation.

Reference(s):

[1] - https://cloud.google.com/chronicle/docs/reference/feed-management-api#api-log-types

The issue with Chronicle Security has been resolved for all affected users as of Wednesday, 2024-04-24 00:15 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Security - Service Issues

Description: Our engineering team has rolled out a mitigation to all impacted systems after which the ingestion backlog has been processed fully. Any remaining downstream impact is expected to be resolved thereafter.

The team is currently monitoring the mitigation applied to ensure extended stability.

We will provide an update by Wednesday, 2024-04-24 00:30 US/Pacific with current details.

Diagnosis: Some Chronicle customers in the US may have noticed ingestion delays for Third Party API Feeds.

The delays in ingested data may have surfaced as delays in detection and other downstream product features.

Workaround: None required.

Summary: We are experiencing an issue with Chronicle Security.

Description: Upon further investigation, our engineering team has identified a mitigation strategy which is being rolled out to impacted systems.

We will provide an update by Tuesday, 2024-04-23 16:00 US/Pacific with current details.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for Third Party API Feeds.

The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.

Summary: We are experiencing an issue with Chronicle Security.

Description: Our engineering team has determined that further investigation is required to identify the root cause and mitigate the issue.

We will provide an update by Tuesday, 2024-04-23 14:30 US/Pacific with current details.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for certain Feeds. The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.

Summary: We are experiencing an issue with Chronicle Security.

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Tuesday, 2024-04-23 14:30 US/Pacific with current details.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for certain Feeds. The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.

Summary: We are experiencing an issue with Chronicle Security.

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Tuesday, 2024-04-23 13:30 US/Pacific with current details.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for certain Feeds. The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.

Summary: We are experiencing an issue with Chronicle Security.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2024-04-23 13:09 US/Pacific.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for certain Feeds. The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.

Summary: We are experiencing an issue with Chronicle Security.

Description: We are experiencing an issue with Chronicle Security.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-04-23 12:51 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Some Chronicle customers in the US may notice ingestion delays for certain Feeds. The delays in ingested data may surface as delays in detection and other downstream product features.

Workaround: None at this time.