Service Health

(All Times US/Pacific)

Incident Start: 19 June 2021 23:00

Incident End: 23 June 2021 09:06

Duration: 3 days, 10 hours, 6 minutes

Affected Services and Features:

• Google Compute Engine (GCE) - New Virtual Machine (VM) creations using PD SSD.
• Google Kubernetes Engine (GKE) - New cluster creations, node pools creation, scaling GKE node pools
• Cloud Composer - Composer environment creations
• Cloud SQL - Cluster creations
• Dataproc - Cluster creations
• Dataflow - Job creations
• Cloud Build - Builds using custom size VMs or workerpools
• Persistent Disk (PD) - New PD-SSD device creation

Regions/Zones: us-east1-{a,b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a

Description:

Multiple Google Cloud services experienced intermittent problems with creating new Persistent Disk Solid-State Drive (PD-SSD) devices. This issue impacted creation of resources that depend on PD-SSD. The root cause of the issue was an unexpected increase in utilization of PD-SSD, which led to unavailability of SSD capacity for some users in the impacted zones.

Customer Impact:

• Google Compute Engine (GCE) :

  • Issues with new VM creations as boot disk creations failed.
  • Auto-scaling of VMs was impacted due to failure with new VM creations.
  • Intermittent reduction in write throughput.
  • Issues with expanding SSD-backed device sizes.

• Cloud Composer:

  • Failed Cloud Composer environment creations.

• Cloud SQL:

  • Intermittent create operation failures.
  • PD-SSD disk size increases failures.

• Cloud Build:

  • Cloud Builds using custom size VMs or using workerpools experienced failures.

• Google Kubernetes Engine:

  • Intermittent issues with cluster creation.
  • Intermittent issues with creation and scaling of node pools with PD-SSD nodes.

• Dataproc/Dataflow:

  • Dataproc cluster creation and Dataflow jobs may have failed due to underlying errors regarding disk creation failures.

Additional details:

To prevent unexpected spike in utilization from impacting PD-SSD capacity availability, the following actions and safety measures were implemented:

• Engineering team has identified and throttled the source of the unexpected spike in PD-SSD utilization.
• Additional SSD capacity added for higher tolerance to spikes in utilization.
• Increased rebalancing to make additional SSD capacity available.

This issue has been fully mitigated by our product team. For any ongoing issues, please open a Support case for further troubleshooting.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: The issue with creating SSD PDs is now mitigated in all affected zones for all affected customers.

We are currently monitoring the environment performance and will provide a final status update by Wednesday, 2021-06-23 17:00 US/Pacific.

Diagnosis: Affected users might be unable to create new VMs in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Compute Engine, such as Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc and Apigee X in those regions.

Workaround: Customers may attempt to create resources in unaffected zones or regions.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: The issue with creating SSD PDs is now mitigated in all affected zones for most affected customers. Some customers are still unable to create VMs in affected zones.

We are still working to fully mitigate this issue for all customers, and estimate this to be complete around Wednesday, 2021-06-23 12:00 US/Pacific.

We will provide more information by Wednesday, 2021-06-23 13:00 US/Pacific.

Diagnosis: Affected users might be unable to create new VMs in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Compute Engine, such as Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc and Apigee X in those regions.

Workaround: Customers may attempt to create resources in unaffected zones or regions.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. In those zone the issue is resolved for the most of customers and we are still working to mitigate the rest of the impact, ETA for full resolution is for by end of day 2021-06-23 US/Pacific hours.

We will provide more information by Wednesday, 2021-06-23 04:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. In those zone the issue is resolved for the most of customers and we are still working to mitigate the rest of the impact, ETA for full resolution is for by end of day 2021-06-23 US/Pacific hours.

We will provide more information by Wednesday, 2021-06-23 02:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. We are still working to mitigate impact in these two remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 23:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. We are still working to mitigate impact in these two remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 20:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL, Cloud Dataproc, and Apigee X in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. We are still working to mitigate impact in these two remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 20:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. We are still working to mitigate impact in these two remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 17:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,}. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}. We are still working to mitigate impact in these two remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 17:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}, us-east4-a. We are still working to mitigate impact in these three remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 16:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}, us-east4-a. We are still working to mitigate impact in these three remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 15:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}, us-east4-a. We are still working to mitigate impact in these three remaining zones, ETA by end of day 2021-06-22 US/Pacific hours.

We will provide more information by Tuesday, 2021-06-22 13:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-{b,c}, us-east4-a. We are still working to mitigate impact in these three remaining zones.

We will provide more information by Tuesday, 2021-06-22 11:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-b and us-east4-a. We are still working to mitigate impact in these two remaining zones.

We will provide more information by Tuesday, 2021-06-22 09:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-b and us-east4-a. We are still working to mitigate impact in these two remaining zones.

We will provide more information by Tuesday, 2021-06-22 07:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-b and us-east4-a. We are still working to mitigate impact in these two remaining zones.

We will provide more information by Tuesday, 2021-06-22 05:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation is mitigated for all affected zones except us-east1-b and us-east4-a. We are still working to mitigate impact in these two remaining zones.

We will provide more information by Tuesday, 2021-06-22 03:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: Mitigation work is still underway by our engineering team.

PD creation should be working currently in us-east1-c and us-west1-c. We are working on mitigating impact in other affected zones.

We will provide more information by Tuesday, 2021-06-22 02:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: A partial mitigation has been identified by the engineering team and is currently underway. We do not have an ETA for the partial mitigation at this point.

We will provide more information by Tuesday, 2021-06-22 01:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Affected users may experience errors when attempting to create a new GKE Clusters or Node pools in us-east1.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

For Users experiencing errors when attempting to create a new GKE Clusters or Node pools -Creating a node pool or cluster should be successful in other zones or regions may help. -Using Standard PDD or Local SSD should be successful.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: A partial mitigation has been identified by the engineering team and is currently underway. We do not have an ETA for the partial mitigation at this point.

We will provide more information by Monday, 2021-06-21 22:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: A partial mitigation has been identified by the engineering team and is currently underway. We do not have an ETA for the partial mitigation at this point.

We will provide more information by Monday, 2021-06-21 20:30 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: A partial mitigation is identified by the engineering team and is currently underway. We do not have an ETA for the same at this point.

We will provide more information by Monday, 2021-06-21 20:00 US/Pacific.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.

Summary: Persistent Disk SSD creation is failing in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Description: We are still experiencing an issue with Google Compute Engine Persistent Disk SSD creation in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a beginning on Monday, 2021-06-20 20:00 US/Pacific. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Our engineering team continues to investigate the issue.

We will provide an update by Monday, 2021-06-21 18:30 US/Pacific with current details.

Diagnosis: Affected users might be unable to create new Persistent Disks (SSD) in us-east1-{b,c,d}, us-east4-{a,b}, us-west1-c and us-west4-a. This may impact creation of resources which depend on Persistent Disk SSD such as Google Compute Engine, Google Kubernetes Engine, Cloud Composer, Cloud SQL and Cloud Dataproc in those same regions.

Workaround: Standard Persistent Disks are not impacted, if workloads are not disk dependent, creating or scaling them with Standard PD may unblock workloads currently using PD backed by SSD. Creation of new Persistent Disks (SSD) in other regions are unaffected.