Incident Report

Summary

On 25 September 2024 from 01:27 to 12:28 US/Pacific, Mandiant Security Validation SaaS customers were unable to run or create Actions, Sequences and Evaluations for a duration of 11 hours and 01 minute.

Customers were able to login and view the UI and observe results of previously run jobs, but were unable to create and run content (Actions/Evaluations/Sequences). From 11:23 to 12:28 US/ Pacific (1 hour and 5 minutes), customers were unable to login and were redirected to a maintenance page containing a notice of the outage while engineers fixed the issue.

To our Mandiant Security Validation SaaS customers whose business was impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

This issue occurred due to an implementation issue in a backend database that limited a sequence column value. The value was incremented beyond that limit causing the database and system to encounter an error which prevented users from running Actions, Sequence and Evaluations.

Remediation and Prevention

Google engineers were alerted to the outage via a support case on 25 September 2024 at 01:41 US/Pacific and immediately started an investigation. Once the nature and scope of the issue became clear, Google engineers adjusted the column value limit that was causing the issue. This required taking Mandiant Security Validation SaaS briefly offline to perform the necessary changes needed to resolve the issue and prevent it from recurring in the future. During this time users were redirected to a maintenance page informing them of the outage. Once the system was brought back online, engineers verified the underlying issue was resolved and that no further issues were encountered.

Google is committed to preventing a repeat of this issue in the future and is completing a :

• Thorough review of the system settings to ensure no further limitations exist that would cause this or similar issues to occur.

Detailed Description of Impact

• From 1:27 to 11:22 US/ Pacific, customers were able to login and view the UI, but were unable to create and run content (Actions/Evaluations/Sequences).

• From 11:23 to 12:28 US/ Pacific, customers were redirected to a maintenance page containing a notice of the outage.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 25 September 2024 01:27

Incident End: 25 September 2024 12:28

Duration: 11 hours, 1 minute

Affected Services and Features:

Mandiant Security Validation

Regions/Zones: Global

Description:

Mandiant Security Validation SaaS customers were unable to run or create actions/sequences/evaluations for a duration of 11 hours, 1 minute. Preliminary investigation indicates the issue resulted from a maximum value limitation in the backend database. Google will complete a detailed Incident Report (IR) in the following days to provide a comprehensive root cause analysis.

Customer Impact:

From 1:27 to 11:22 US/ Pacific, customers were able to login and view the UI, but were unable to create and run content (Actions/Evaluations/Sequences).

From 11:23 to 12:28 US/ Pacific, customers were redirected to a maintenance page containing a notice of the outage.

The issue with Mandiant Security Validation has been resolved for all affected users as of Wednesday, 2024-09-25 12:30 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Mandiant Security Validation customers are unable to access the application and run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 01:27 US/Pacific.

Mitigation work is currently underway by our engineering team. Our engineers have successfully tested the mitigation in lower environments and are actively rolling out the same in production.

The revised ETA for mitigation is by Wednesday, 2024-09-25 13:00 US/Pacific.

We will provide more information by Wednesday, 2024-09-25 13:30 US/Pacific.

Diagnosis: Mandiant Security Validation is currently in maintenance mode and is not available to customers at the moment. Customers who were logged in are being redirected to the maintenance page, and no new customers will be able to login into the application.

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 01:27 US/Pacific.

Mitigation work is currently underway by our engineering team. Our engineers has successfully tested the mitigation in lower environments and are actively rolling out the same in production.

The revised ETA for mitigation is by Wednesday, 2024-09-25 12:30 US/Pacific.

We will provide more information by Wednesday, 2024-09-25 13:00 US/Pacific.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 01:27 US/Pacific.

Mitigation work is currently underway by our engineering team. Our engineers has successfully tested the mitigation in lower environments and are actively rolling out the same in production.

The mitigation is expected to complete by Wednesday, 2024-09-25 11:30 US/Pacific.

We will provide more information by Wednesday, 2024-09-25 12:00 US/Pacific.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 01:27 US/Pacific.

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2024-09-25 11:00 US/Pacific.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2024-09-25 11:00 US/Pacific.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 06:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-09-25 08:00 US/Pacific with current details.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time

Summary: Mandiant Security Validation customers are unable to run jobs

Description: We are experiencing an issue with Mandiant Security Validation beginning on Wednesday, 2024-09-25 01:27 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2024-09-25 07:15 US/Pacific with current details.

Diagnosis: Mandiant Security Validation customers are unable to run jobs

Workaround: None at this time