Incident Report

Summary

On Tuesday, 14 April 2026, Google SecOps experienced higher latencies and query timeouts for its search and dashboard features in the multiregion: us, for a duration of 5 hours and 30 minutes. If your service or application was affected, we apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The root cause of this incident was elevated latency within the storage management service, triggered by a "hotspot" on a specific configuration key within the underlying database. This condition resulted from a parallel service disruption in the database service which caused background maintenance jobs to fail. These background jobs are responsible for consolidating small, temporary data files into larger, more efficient sets; their failure led to a significant increase in the volume of unconsolidated files required for each query.

The sequence of events leading to customer impact was as follows:

• A software update in the underlying database service led to performance degradation, preventing automated background jobs from consolidating temporary data files.
• The number of unconsolidated files increased significantly, as the system could not merge them into optimized formats during the disruption.
• Every lookup for these files required a configuration check against a single, shared configuration key in the database.
• The high volume of lookup requests created a hotspot on the specific database tablet hosting that configuration key, resulting in extreme read contention and high latency.
• The high latency in the storage layer caused Unified Data Model (UDM) search and dashboard queries to time out or experience severe performance degradation.

Remediation and Prevention

Google engineers were alerted to the issue via internal monitoring frameworks (prober failure). Subsequently, a service disruption was declared, and relevant engineering teams were engaged to identify the root cause and mitigate the issue. Immediate steps were taken by engineering teams to stabilize the service and apply mitigation:

• Engineering stabilized the storage management service by adding more capacity to specific storage management servers.
• Engineering worked to shard the impacted configuration (cause of hotspot) to permanently reduce the load on single rows.
• Engineers confirmed that latency returned to baseline levels and the issue was mitigated on 14 April at 08:00 PDT, once the configuration sharding was completed across all shards.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

• Distribute storage service configurations: We are splitting the configuration for our storage management service into multiple shards to more effectively distribute lookup requests. This change will prevent performance bottlenecks previously caused by high volumes of requests hitting a single configuration entry.
• Detect unusual data volume patterns: We are developing proactive alerting to monitor for rapid increases in the volume of data files being processed. This will allow us to identify and address potential performance triggers before they lead to customer-visible degradation.

Detailed Description of Impact

On Tuesday, 14 April 2026 from 02:30 to 08:00 US/Pacific, customers using Google SecOps in the multiregion:us experienced service degradation. Specifically, Google SecOps search and dashboard features encountered higher latencies or queries timing out. Customers trying to access these features saw queries and charts taking an unusually long time to load or failing with timeouts.

Preliminary Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. A final Incident Report with preventative actions will be posted once our investigation is complete. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

Date/Time of the Issue (All time US/Pacific)

Incident Start: 14 April 2026 02:30

Incident End: 14 April 2026 08:00

Duration: 5 hours, 30 minutes

Summary

On Tuesday, 14 April 2026, Google SecOps experienced higher latencies and query timeouts for its search and dashboard features in the multiregion:us, for a duration of 5 hours and 30 minutes. If your service or application was affected, we apologize, this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

Preliminary Root Cause

The root cause of this incident was elevated latency within the storage management service, triggered by a hotspot on a specific configuration key. This likely resulted from shifts in traffic patterns that increased the volume of data files required for each Unified Data Model (UDM) query. This situation had a direct effect on Google SecOps, leading to delays and failures within search and dashboard functionalities.

The sequence of events leading to customer impact was as follows:

• The configuration was queried for every lookup request, triggering hotspot in storage service on a single configuration row.
• This resulted in excessive load on specific storage services resulting in the high latency.
• The high latency led to significant failures and delays when accessing Google SecOps search and dashboard features.
• Customers experienced queries and charts taking an unusually long time to load or failing with timeouts.

Google engineers have initiated a comprehensive root cause analysis and will share updates as they become available. A thorough Incident Report, including preventative measures, will be published.

Remediation

Google engineers were alerted to the issue via internal monitoring frameworks (prober failure), after which a Service disruption was declared and relevant engineering teams engaged to root cause and mitigate. Immediate steps were taken by engineering teams to stabilize the service and apply mitigation:

• Engineering stabilized the storage management service by adding more capacity to specific storage management servers.
• The Eng team worked to shard the impacted configuration (cause of hotspot) to permanently reduce the load on single rows.
• Engineers confirmed latency recovered and the issue was mitigated on 14 April at 08:00 PDT, after the latest mitigation action was rolled out.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Description of Impact

On Tuesday, 14 April 2026 from 02:30 to 08:00 US/Pacific, customers using Google SecOps in the multiregion:us experienced service degradation. Specifically, Google SecOps search and dashboard features encountered higher latencies or queries timing out. Customers trying to access these features saw queries and charts taking an unusually long time to load or failing with timeouts.

Description:

The latency and time out issue with Google Secops search and dashboard has been resolved for all affected users as of Tuesday, 2026-04-14 08:00 PDT

From preliminary analysis, the issue was caused by hotspotting of the backend database. Our engineers mitigated the issue by sharding the files to reduce the load on the backend database.

We thank you for your patience while we worked on resolving the issue.

Customer Symptoms:

Google SecOps search/dashboard queries/charts took longer to load or failed to load for customers in multiregion: us.

Workaround:

The issue is now mitigated.

Description:

Engineers are currently implementing additional mitigation strategies for this issue, as earlier efforts did not resolve the situation.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2026-04-14 08:45 PDT

Customer Symptoms:

Google SecOps search/dashboard queries/charts are taking longer to load or failing to load for customers in multiregion: us.

Workaround:

None at this time.

Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Tuesday, 2026-04-14 08:00 PDT.

We will provide more information by Tuesday, 2026-04-14 08:00 PDT.

Customer Symptoms: Google SecOps search/dashboard queries/charts are taking longer to load or failing to load for customers in multiregion: us.

Workaround: None at this time.

Description: We are experiencing an issue with Google SecOps beginning on Tuesday, 2026-04-14 02:30 PDT.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2026-04-14 06:00 PDT with current details.

We apologize to all who are affected by the disruption.

Customer Symptoms: Google SecOps search/dashboard queries/charts are taking longer to load or failing to load for customers in multiregion: us.

Workaround: None at this time.