Service Health

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Detection Time: 24 August 2022 07:59

Incident End: 25 August 2022 17:16

Duration: 1 day 9 hours, 17 minutes

Affected Services and Features:

Cloud Security Command Center (SCC) - Container Threat Detection (KTD) was not monitoring some customer clusters.

Regions/Zones: Global

Description:

Google Cloud Security Command Center (SCC) experienced an issue with enabling Container Threat Detection(KTD) during the SCC Premium onboarding process. During the onboarding process if a customer accepted the default service settings, it would show all detection services as Enabled, but the SCC User Interface (UI) would fail to save the settings for Container Threat Detection(KTD) causing this detection to be in disabled state unless the customer explicitly enabled it at a later point.

The issue was first detected on 24 August 2022 at 07:59 US/Pacific while Google Engineers were working on analyzing a future code release. From preliminary analysis, the root cause of the issue is a latent bug in the SCC UI. The issue was resolved on 25 August 2022 at 17:16 US/Pacific after a mitigation was rolled out to the SCC Settings backend that changed the default state of Container Threat Detection to Enabled.

Customer Impact:

Affected customers failed to receive Container Threat Detection (KTD) coverage.

The issue with Cloud Security Command Center has been resolved for all affected users as of Thursday, 2022-08-25 17:16 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is still underway by our engineering team.

The mitigation is expected to complete by Thursday, 2022-08-25 18:10 US/Pacific.

We will provide more information by Thursday, 2022-08-25 18:15 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is still underway by our engineering team and is taking longer than expected.

We do not have a new ETA for mitigation at this point.

We will provide more information by Thursday, 2022-08-25 16:30 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is still underway by our engineering team.

We will provide more information by Thursday, 2022-08-25 14:30 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is currently underway by our engineering team.

The mitigation is expected to complete by Thursday, 2022-08-25 13:30 US/Pacific.

We will provide more information by Thursday, 2022-08-25 13:30 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2022-08-25 10:30 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-08-24 17:00 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-08-24 16:00 US/Pacific.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: We are experiencing an issue with Cloud Security Command Center

Our engineering team continues to investigate the issue and are working on a mitigation strategy.

We will provide an update by Wednesday, 2022-08-24 13:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: We are experiencing an issue with Cloud Security Command Center

Our engineering team continues to investigate the issue and are working on a mitigation strategy.

We will provide an update by Wednesday, 2022-08-24 11:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: We are experiencing an issue with Cloud Security Command Center

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-08-24 10:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings.

Workaround: Customers can enable KTD in SCC settings.

Summary: SCC onboarding failed to enable Container Threat Detection (KTD)

Description: We are experiencing an issue with Cloud Security Command Center

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-08-24 10:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Container Threat Detection is not monitoring customers cluster in organizations that have not explicitly enabled Container Threat Detection in their SCC settings

Workaround: Customers can enable KTD in SCC settings