Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start:

25 June, 2024 11:58

Incident End:

28 June, 2024 12:22

Duration:

3 Days, 23 minutes

Affected Services and Features:

Google SecOps

Regions/Zones:

US - Multi-Region

Description:

Google SecOps experienced service degradation with multiple features in the US/multiregion for a duration of 3 days and 23 minutes.

Customer Impact:

During the incident some customers would have experienced the following issues within Google SecOps:

Parser UI: Affected users would have been unable to access parsers via the user interface.

Feeds UI: Functionality was degraded, preventing feed names from being displayed in the user interface.

Raw Log Search Timestamp Selector: The timestamp selector prevented affected users from selecting earlier dates, thus limiting search functionality.

Raw Log Search UI: The raw log search UI displayed all log types as 0kB, hindering accurate log analysis.

Raw Log Search Historic Availability: Raw log search for historic data beyond 48hrs after ingestion was unavailable.

IOC matches page: Feed Source Names were not shown correctly.

Data ingestion: Ingestion was delayed for some 3rd party API feeds.

The issue with Chronicle Security has been mitigated for all affected users as of Friday, 2024-06-28 11:00AM US/Pacific.

• Parser UI: The parser UI functionality is working as expected.
• Feeds UI: The feeds UI functionality is working as expected.
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed.
• Raw Log Search UI Display Error: Mitigated and fixed.
• Raw Log Search Historic Availability: Mitigated and fixed.
• IOC matches page: Mitigated and fixed.
• Data ingestion: Mitigated and fixed and data backfilled.

We thank you for your patience while we worked on resolving the issue.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: The Raw Log Search historic availability issue is now mitigated.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

The backfill for Proofpoint Tap Alerts for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific has been successfully completed.

Our engineering team will continue working to backfill data for log types Proofpoint On Demand for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific.

The ETA for completion of most of the backfills is Friday, 2024-06-28 13:00 US/Pacific.

We will provide more information by Friday, 2024-06-28 14:00 US/Pacific

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

Parser UI: The parser UI functionality is working as expected.

Feeds UI: The feeds UI functionality is working as expected.

Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed.

Raw Log Search UI Display Error: Mitigated and fixed.

Raw Log Search Historic Availability: Mitigated and fixed.

IOC matches page: Mitigated and fixed.

Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for log types: Proofpoint On Demand

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: The Raw Log Search historic availability issue is now mitigated.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

Our engineering team will continue working to backfill data for log types Proofpoint On Demand and Proofpoint Tap Alerts for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific.

The ETA for completion of most of the backfills is Friday, 2024-06-28 10:00 US/Pacific.

We will provide more information by Friday, 2024-06-28 05:30 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

Parser UI: The parser UI functionality is working as expected.

Feeds UI: The feeds UI functionality is working as expected.

Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed.

Raw Log Search UI Display Error: Mitigated and fixed.

Raw Log Search Historic Availability: Mitigated and fixed.

IOC matches page: Mitigated and fixed.

Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for two log types: Proofpoint On Demand and Proofpoint Tap Alerts.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: The Raw Log Search historic availability issue is now mitigated.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

Our engineering team will continue working to backfill data for log types Proofpoint On Demand and Proofpoint Tap Alerts for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific.

The ETA for completion of most of the backfills is Friday, 2024-06-28 10:00 US/Pacific.

We will provide more information by Friday, 2024-06-28 05:00 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

Parser UI: The parser UI functionality is working as expected.

Feeds UI: The feeds UI functionality is working as expected.

Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed.

Raw Log Search UI Display Error: Mitigated and fixed.

Raw Log Search Historic Availability: Mitigated and fixed.

IOC matches page: Mitigated and fixed.

Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for two log types: Proofpoint On Demand and Proofpoint Tap Alerts.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: The Raw Log Search historic availability issue is now mitigated.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

Our engineering team will continue working to backfill data for log types Proofpoint On Demand and Proofpoint Tap Alerts for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific.

The ETA for completion of most of the backfills is Friday, 2024-06-28 10:00 US/Pacific.

We will provide more information by Friday, 2024-06-28 00:00 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

Parser UI: The parser UI functionality is working as expected. Feeds UI: The feeds UI functionality is working as expected. Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed. Raw Log Search UI Display Error: Mitigated and fixed. Raw Log Search Historic Availability: Mitigated and fixed. IOC matches page: Mitigated and fixed. Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for two log types: Proofpoint On Demand and Proofpoint Tap Alerts.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: The Raw Log Search historic availability issue is now mitigated.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

Our engineering team will continue working to backfill data for log types Proofpoint On Demand and Proofpoint Tap Alerts for the timeframe of Tuesday, 2024-06-25 12:30 to Wednesday, 2024–06-26 18:30 US/Pacific.

The ETA for completion of the backfills is Friday, 2024-06-28 10:00 US/Pacific.

We will provide more information by Thursday, 2024-06-27 14:30 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI functionality is working as expected.
• Feeds UI: The feeds UI functionality is working as expected.
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed.
• Raw Log Search UI Display Error: Mitigated and fixed.
• Raw Log Search Historic Availability: Mitigated and fixed.
• IOC matches page: Mitigated and fixed.
• Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for two log types: Proofpoint On Demand and Proofpoint Tap Alerts.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with multiple features in the US/multiregion.

Description: Mitigation work is still underway by our engineering team.

Currently the primary impact is to historical raw log searches for data beyond 48 hours, which are not updated with late arriving events.

Our engineering team is actively working to address the backlog and implement a complete mitigation for users in the affected regions.

The issue of data ingestion delays is resolved for all 3rd party API sources including Mandiant Managed Defense. Backfill for most log types are complete.

Our engineer team will continue working to backfill data of log types Proofpoint On Demand and Proofpoint Tap Alerts for duration Tuesday, 2024-06-25 14:55 to Wednesday, 2024–06-26 18:02 US/Pacific with an ETA of Friday, 2024-07-05 10:00 US/Pacific.

We will provide more information by Thursday, 2024-06-27 10:00 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.

• Feeds UI: The feeds UI functionality is working as expected

• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment

• Raw Log Search UI Display Error: Mitigated and fixed in the production environment .

• Raw Log Search Historic Availability: Historic data beyond 48hrs after ingestion is available, however such data is not updated with late arriving events. Mitigation is in progress.

• IOC matches page: The issue with Feed Source Names not shown correctly is now mitigated.

• Data ingestion: Delay of > 17 hours for some 3rd party API feeds is now mitigated. Backfill pending for two log types.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregion.

Description: Mitigation work is still underway by our engineering team.

Currently the primary impact is to historical raw log searches for data beyond 48 hours, which are not updated with late arriving events.

Our engineering team is actively working to address the backlog and implement a complete mitigation for users in the affected regions.

We will provide more information by Thursday, 2024-06-27 10:00 US/Pacific.

Diagnosis: Customers across US/multiregion would experience the following issues within Chronicle security. However, the remaining features are unaffected by this issue.

Parser UI: The parser UI is now functioning and displaying active parsers. Feeds UI: The feeds UI functionality is working as expected Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment Raw Log Search UI Display Error: Mitigated and fixed in the production environment . Raw Log Search Historic Availability: Historic data beyond 48hrs after ingestion is available, however such data is not updated with late arriving events. Mitigation is in progress. IOC matches page: The issue with Feed Source Names not shown correctly is now mitigated.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: Mitigation work is still underway by our engineering team.

Currently the primary impact is to historical raw log searches for data beyond 48 hours, which are not updated with new data.

Our engineering team is actively working to address the backlog and implement a complete mitigation for users in the affected regions.

We will provide more information by Wednesday, 2024-06-26 17:00 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is working as expected
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment
• Raw Log Search UI Display Error: Mitigated and fixed in the production environment
• Raw Log Search Historic Availability: Historic data beyond 48hrs after ingestion is available, however such data is not updated with any new data. Mitigation is in progress
• **IOC matches page: The issue with Feed Source Names not shown correctly is now mitigated.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: Mitigation work is still underway by our engineering team.

System functionality has been fully restored. Our engineering team is actively working to address the backlog and implement a complete mitigation for users in the affected regions.

We will provide more information by Wednesday, 2024-06-26 12:00 US/Pacific.

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is working as expected
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment
• Raw Log Search UI Display Error: Mitigated and fixed in the production environment
• Raw Log Search Historic Availability: The issue with Raw log search for historic data beyond 48hrs after ingestion is now mitigated and available for raw log search.
• **IOC matches page: The issue with Feed Source Names not shown correctly is now mitigated.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team has determined the root cause of the issue and has prevented any impact in all regions outside the US multi-region.

Mitigation and restoration is underway to restore full system functionality. We will provide more information by Wednesday, 2024-06-26 07:00 US/Pacific

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is working as expected
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment
• Raw Log Search UI Display Error: Mitigated and fixed in the production environment
• Raw Log Search Historic Availability: The issue with Raw log search for historic data beyond 48hrs after ingestion is now mitigated and available for raw log search.
• **IOC matches page: Feed Source Names are not shown correctly. Mitigation is in progress.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team has determined the root cause of the issue and has prevented any impact in all regions outside the US multi-region.

Mitigation and restoration is underway to restore full system functionality. We will provide more information by Wednesday, 2024-06-26 05:30 US/Pacific

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is degraded in the US region (the feed name is not currently shown). The mitigation is in progress.
• Raw Log Search Timestamp Selector Malfunction: Mitigated and fixed in the production environment
• Raw Log Search UI Display Error: Mitigated and fixed in the production environment
• Raw Log Search Historic Availability: Raw log search for historic data beyond 48hrs after ingestion is currently unavailable for raw log search. There is no impact to the data integrity of raw logs in Chronicle.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team has determined the root cause of the issue and has prevented any impact in all regions outside the US multi-region.

Mitigation and restoration is underway to restore full system functionality. We will provide more information by Wednesday, 2024-06-26 02:15 US/Pacific

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is degraded in the US region (the feed name is not currently shown).
• Raw Log Search Timestamp Selector Malfunction:
  • The timestamp selector does not allow users to select earlier dates, limiting search functionality.
  • A mitigation to this issue is rolling out.
• Raw Log Search UI Display Error: The raw log search UI displays available data for all log types as 0kB.
• Raw Log Search Historic Availability: Raw log search for historic data beyond 48hrs after ingestion is currently unavailable for raw log search. There is no impact to the data integrity of raw logs in Chronicle.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team has determined the root cause of the issue and has prevented any impact in all regions outside the US multi-region.

Mitigation and restoration is underway to restore full system functionality. We will provide more information by Tuesday, 2024-06-25 22:30 US/Pacific

Diagnosis: The customers across US/multiregion would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

• Parser UI: The parser UI is now functioning and displaying active parsers.
• Feeds UI: The feeds UI functionality is degraded in the US region (the feed name is not currently shown).
• Raw Log Search Timestamp Selector Malfunction:
  • The timestamp selector does not allow users to select earlier dates, limiting search functionality.
  • A mitigation to this issue is rolling out.
• Raw Log Search UI Display Error: The raw log search UI displays available data for all log types as 0kB.
• Raw Log Search Historic Availability: Raw log search for historic data beyond 48hrs after ingestion is currently unavailable for raw log search. There is no impact to the data integrity of raw logs in Chronicle.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in the US/multiregions.

Description: We are experiencing an issue with Chronicle Security beginning on Tuesday, 2024-06-25 12:00 US/Pacific.

Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2024-06-25 21:00 US/Pacific.

We will provide more information by Tuesday, 2024-06-25 21:30 US/Pacific.

Diagnosis: The customers across US/multiregions would experience the following issues with the Raw Log search feature within Chronicle Security Console. However, the remaining features are unaffected by this issue.

1, Raw Log Search Timestamp Selector Malfunction: The timestamp selector does not allow users to select earlier dates, limiting search functionality.

2, Parser UI Failure: The parser UI is not functioning, preventing users from accessing and utilizing parsers.

3, Raw Log Search UI Display Error: The raw log search UI displays all log types as 0kB, hindering accurate log analysis.

Workaround: None at this time.

Summary: Chronicle Security is experiencing a service degradation with a few features in US multi-region.

Description: We are experiencing an issue with Chronicle Security beginning at Tuesday, 2024-06-25 12:00 US/Pacific.

Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2024-06-25 21:00 US/Pacific.

We will provide more information by Tuesday, 2024-06-25 21:30 US/Pacific.

Diagnosis: A subset of customers in the US region would experience the following issues.

1, Raw Log Search Timestamp Selector Malfunction: The timestamp selector does not allow users to select earlier dates, limiting search functionality.

2, Parser UI Failure: The parser UI is not functioning, preventing users from accessing and utilizing parsers.

3, Raw Log Search UI Display Error: The raw log search UI displays all log types as 0kB, hindering accurate log analysis.

Workaround: None at this time.

Summary: Chronicle Security customers may experience Parsers not appearing in parsers UI and raw log search has empty log types

Description: We are experiencing an issue with Chronicle Security beginning at Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team is actively investigating the issue to identify the cause and determining mitigation steps.

We will provide an update by Tuesday, 2024-06-25 16:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Parser page is empty in the UI and raw log search has empty log types

Workaround: None at this time.

Summary: Chronicle Security customers may experience Parsers not appearing in parsers UI and raw log search has empty log types

Description: We are experiencing an issue with Chronicle Security beginning at Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team is actively investigating the issue to identify the cause and determining mitigation steps.

We will provide an update by Tuesday, 2024-06-25 15:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Parser page is empty in the UI and raw log search has empty log types

Workaround: None at this time.

Summary: Chronicle Security customers may experience Parsers not showing in parsers UI and raw log search has empty log types

Description: We are experiencing an issue with Chronicle Security beginning at Tuesday, 2024-06-25 12:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2024-06-25 14:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Parser page is empty in the UI and raw log search has empty log types

Workaround: None at this time.