Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Chronicle Security, Chronicle SOAR

Chronicle Security, Chronicle SOAR. High Remote Procedure Calls (RPC) error rate in Multiregion/us

Incident began at 2024-02-28 23:50 and ended at 2024-02-29 09:50 (all times are US/Pacific).

Previously affected location(s)

Multi-region: us

Date Time Description
29 Feb 2024 12:51 PST

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 28 February 2024 23:50

Incident End: 29 February 2024 09:50

Duration: 10 hours

Affected Services and Features:

  • Chronicle Security - Security Information and Event Management (SIEM)
  • Chronicle Security Orchestration, Automation and Response (SOAR)

Regions/Zones: US/multi-region

Description:

Chronicle Security SIEM and Chronicle SOAR experienced elevated Remote Procedure Calls (RPC) error rate for Backstory API calls in US/multi-region for a total duration of 10 hours. A new service was activated which caused additional traffic to be generated and resource contention, which impacted overall RPC performance, latency, and error rate.

Our engineers identified a subset of the root cause processes and eliminated them on Thursday, 29 February 2024 06:18 US/Pacific. However, some of these processes were missed in the initial analysis, causing additional traffic and resource contention at 7:13. Engineers took additional steps, further mitigating the issue at 09:50.

At this time, we do not believe any additional actions are needed to prevent recurrence of this issue.

Customer Impact:

  • Backstory API calls would have failed with Remote Procedure Calls (RPC) errors.
29 Feb 2024 08:21 PST

The issue with Chronicle Security, Chronicle SOAR has been resolved for all affected projects as of Thursday, 2024-02-29 08:20 US/Pacific.

We thank you for your patience while we worked on resolving the issue.
29 Feb 2024 07:34 PST

Summary: Chronicle Security, Chronicle SOAR. High Remote Procedure Calls (RPC) error rate in Multiregion/us

Description: We are experiencing an issue with Chronicle Security, Chronicle SOAR.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2024-02-29 09:00 US/Pacific with current details.

Diagnosis: Impacted users may experience high failure rate for Backstory API calls

Workaround: None at this time.