This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.
Available
Service information
Service disruption
Service outage
Incident affecting Google Compute Engine
Windows VMs using CrowdStrike are crashing.
Incident began at 2024-07-18 23:48 and ended at 2024-07-19 16:32(all times are US/Pacific).
Previously affected location(s)
Johannesburg (africa-south1)Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Berlin (europe-west10)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Dammam (me-central2)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
Date
Time
Description
19 Jul 2024
16:32 PDT
Beginning July 19th at 04:09 UTC, Google Cloud detected some customer Windows VMs experiencing Blue Screen of Death (BSOD) and crash loops. These Windows VMs running CrowdStrike Falcon began to fail after a CrowdStrike software update.
Crowdstrike quickly deployed a fix, however some customer impact remained. While Google Cloud services were not directly impacted, Google Cloud continues to work with CrowdStrike to help our customers recover from any remaining impact.
If your Windows VM continues to experience issues after a reboot, manual patching. Please contact Google Cloud Customer Support.
19 Jul 2024
14:46 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
Google teams are continuing to work with CrowdStrike on helping customers recover their VMs and proactively reaching out to affected customers.
We will provide an update by Friday, 2024-07-19 17:00 US/Pacific with current details.
If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround:
Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
13:14 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
Google teams are continuing to work with CrowdStrike on helping customers recover their VMs and proactively reaching out to affected customers.
We will provide an update by Friday, 2024-07-19 15:00 US/Pacific with current details.
If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Reboot the host to give it an opportunity to download the reverted channel file. If the host crashes again, then:
Boot Windows into Safe Mode or the Windows Recovery Environment
NOTE: Putting the host on a wired network (as opposed to WiFi) and using Safe Mode with Networking can help remediation.
Navigate to the %WINDIR%\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Boot the host normally.
Note: Bitlocker-encrypted hosts may require a recovery key.
Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
10:51 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CrowdStrike, Windows VMs may crash and might not be able to reboot. Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
Google teams are continuing to work with CrowdStrike on recovery efforts and proactively reaching out to affected customers to mitigate
We will provide an update by Friday, 2024-07-19 13:00 US/Pacific with current details.
If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
08:59 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CrowdStrike, Windows VMs crash and might not be able to reboot.
Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
Google teams are continuing to work with CrowdStrike on recovery efforts and proactively reaching out to affected customers to mitigate.
We will provide an update by Friday, 2024-07-19 11:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
07:05 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CrowdStrike on Google Compute Engine. CrowdStrike has confirmed that a faulty update to the CrowdStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CrowdStrike, Windows VMs crash and might not be able to reboot.
Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
06:54 PDT
Summary: Windows VMs using CrowdStrike are crashing.
Description: We are experiencing an issue with Windows VMs running CloudStrike on Google Compute Engine. CloudStrike has confirmed that a faulty update to the CloudStrike Falcon agent was deployed beginning at 04:09 UTC July 19.
After having automatically received a defective patch from CloudStrike, Windows VMs crash and might not be able to reboot.
Windows VMs that are currently up and running should no longer be impacted.
According to CrowdStrike, 80% of Windows VMs experiencing this issue will self-heal during a reboot.
We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
04:56 PDT
Summary: Windows VMs using Crowdstrike’s csagent.sys are crashing and going into unexpected reboot
Description: We are experiencing an issue with Windows VMs running Cloudstrike on Google Compute Engine.
After having automatically received a defective patch from Cloudstrike, Windows VMs crash and will not be able to reboot.
Windows VMs that are currently up and running should no longer be impacted.
We will provide an update by Friday, 2024-07-19 09:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Use rescue VM (Test VM), attach boot disk of the affected VM as a secondary disk
**NOTE: Ensure that the boot disk image of the recovery VM differs from the boot disk that is being repaired; failure to do so may result in duplicate disk or partition GUID and unpredictable results as confirmed by Microsoft.
Navigate to the D:\Windows\System32\drivers\CrowdStrike directory
Locate the file matching “C-00000291*.sys”, and delete it.
Power down the Rescue VM detach the secondary disk
Re-attach the VM to the original VM and boot
19 Jul 2024
02:17 PDT
Summary: Windows VMs using Crowdstrike’s csagent.sys are crashing and going into unexpected reboot
Description: We are experiencing an issue with Google Compute Engine.
Our engineering team continues to investigate the issue.
We will provide an update by Friday, 2024-07-19 05:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Windows VM are crashing and going into an unexpected reboot.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
Identifying Faulty "C-00000291*.sys"
Channel file "C-00000291*.sys" with timestamp of 0527 UTC or later is the reverted (good) version.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
Workaround: Crowdstrike has pushed an update which should replace the agent "C-00000291*.sys”. The windows VMs that are currently running should no longer be impacted.
If your VMs are affected, please follow the Workaround Steps to fix the issue.
