Service Health

INCIDENT REPORT

Summary:

On Thursday, May 20, 2021 at 05:25 US/Pacific, Google Cloud products experienced elevated latency and/or errors due to an issue with Access Control Lists (ACLs) intermittently for a duration of 10 hours and 45 minutes. The affected ACLs in this instance were internal that are used to define permissions for Google’s internal production resources. This prevented some internal service accounts from accessing various production jobs, which led to the downstream service impact. We apologize to our Cloud customers whose businesses were impacted during this disruption, and we are taking immediate action to improve the resiliency, performance, and availability of our services to avoid a recurrence.

Root Cause:

Google’s datacenters rely on Access Control Lists (ACLs) stored in a highly available lock service to perform operations, including validating permissions, activating new APIs, or creating new cloud resources. The access distribution manager, which maintains ACLs across Google production, encompasses both the control plane systems that allow for users to edit groups and the jobs which write the intended current state into the data plane, which is stored in a lock service as a set of signed ACL files.

The incident was triggered by a latent concurrency issue in a component of the production ACL system combined with a missing safety check, which led to truncation of ACLs across a subset of production. The safety mechanism for the legacy ACL components was inadvertently disabled by a change to update to the latest version in a subset of datacenters. The disabled safety checks in the legacy ACL component jobs allowed the invalid ACLs to be processed. The ACL component identified membership removals in the data snapshot and proceeded to remove ACL files from a backend locking service and passed the invalid ACLs to the backend stack, which correctly failed the safety checks.

Remediation and Prevention:

Google engineers were alerted to the issue by automated alerting on 20 May 2021 at 05:55 US/Pacific and immediately started an investigation. At 06:07, engineers identified an incomplete data snapshot being written, and a safety check mechanism was unexpectedly removing group memberships. At 06:21, engineers began mitigation efforts to terminate ACL component jobs globally to prevent further impact. To identify the root cause, engineers compared jobs serving affected datacenters and verified they were served by legacy ACL components, while those which were not affected were served using the latest version of safety check flags. The next phase of the mitigation strategy was to restore most critical group memberships that were removed. Simultaneously, the failing ACL component jobs were being configured to apply the correct safety checks and prepared to restart to restore healthy memberships. By 07:32, engineers worked to repair and restore the legacy stack jobs to write the correct ACLs in impacted datacenters. By 07:44, we began manually restoring critical ACLs. By 15:09, most of the production fleet was mitigated and engineers continued to restore the remaining ACLs. At 16:10, the ACL issues across production were fully mitigated.

We apologize for the length and severity of this incident. We are taking immediate steps to prevent a recurrence and improve reliability in the future.

Detailed Description of Impact:

On Thursday, May 20, 2021 from 05:25 to 16:10 US/Pacific, Google Cloud Infrastructure components experienced a service disruption which had varying degrees of impact to downstream services as described in detail below:

BigQuery

Experienced elevated error rates running queries and streaming in europe-west2. Around 5% of projects in the region were impacted with 0.63% failed query jobs and 3% of failed requests for streaming inserts from 06:10 to 07:30 PT.

Cloud Data Catalog

Experienced elevated error rates (30-40%) with permissions when reading backend databases in europe-west2 from 10:15 PT to 11:35 PT.

Cloud Logging

Experienced delayed logs in Google Cloud Storage (GCS) export pipeline in europe-west2 from 07:45 to 11:31 PT.

Cloud SQL

Experienced elevated error rates and latency on Admin APIs in the following regions from 06:51 to 09:44 PT: asia-south2, asia-southeast1, australia-southeast1, europe-north1, europe-west2, europe-west4, europe-west6, northamerica-northeast1, southamerica-east1, us-central1, us-central2, us-east1, us-east4, us-west1 and europe-west1. Among most impacted operations, instance creation (instances.insert) had an error ratio of 69%. Additionally, 10% of successful responses to instance.insert later experienced downstream creation failures. Changes to instance configs (instances.update and instances.patch) had 96% and 87% error rates respectively. Existing instances had no significant connectivity impact. Customers using gcloud sql to connect from new IP addresses that had not yet been added to the ACL, would have been unable to connect.

Cloud Bigtable

Data API experienced elevated latency and error rates up to 10% in europe-west2 from 06:30 to 09:33 PT.

Cloud Dataproc

Experienced elevated latency of 95% with cluster creation and deletion in europe-west2.

Cloud Vertex AI

Experienced service degradation in europe-west2 which caused 15% of API requests to fail from 06:57 to 10:28 PT.

Cloud AI Platform

Experienced up to 100% widespread backend error rates in europe-west2 from 06:15 to 12:40 PT.

Cloud Dataflow

Experienced jobs stuck in a pending state; jobs were unable to be terminated or cancelled; jobs were unable to migrate backends which resulted in stuck progress; and streaming jobs experienced delays in europe-west2. 35% of projects were impacted in the region from 06:06 to 12:37 PT.

Cloud Data Loss Prevention (DLP)

Up to 75% of DLP worker tasks were down in europe-west2. 25% error rate averaged across all method types from 10:28 to 11:36 PT.

Google Compute Engine (GCE)

Some GCE APIs experienced degraded availability and elevated error rates up to 40% in europe-west2 and europe-west3. Customers also experienced delays in VM creation and deletion operations in us-west2-a from 06:50 to 11:48 PT.

Cloud Composer

Experienced composer environment creation and update operation failures up to 80% globally from 06:08 to 11:07 PT. Existing environments may have experienced service errors.

AppEngine Flexible

Experienced deployment failures with timeouts or reporting success globally. Approximately 21% of affected projects experienced 503 errors after deploying a new version, and approximately 79% of affected projects experienced deployment deadline exceeded errors from 05:45 to 09:30 PT.

Cloud Networking

Customer initiated control plane config changes related to load balancing products were delayed globally from 05:28 PT to 09:15 PT. Creation or changes to existing Network Load Balancers were delayed in europe-west2 (from 06:20 to 10:05 PT) and europe-west3 (from 07:19 to 10:05 PT). Customers who didn't command config changes during the incident were not affected.

Cloud Armor

New Cloud Armor configs were delayed globally from 05:35 to 09:15 PT.

Cloud Console

Experienced elevated error rates up to 100% while accessing some pages of the console. 2.82% of users were impacted globally from 07:03 to 10:00 PT.

Artifact Registry

Experienced errors for several HTTP-Maven/Docker and ScottyAgent requests in europe-west1 and australia-southeast1 from 06:20 - 09:45 PT.

Google Cloud Storage (GCS)

Experienced elevated “400” errors up to 95% which were mostly deadline exceeded errors for 3 hours in europe-west2 from 07:25 to 10:22 PT.

Google Kubernetes Engine (GKE)

Customers were unable to create new clusters for around 2 hours in europe-west2 and europe-west3 from 08:20 to 10:10 PT.

Identity and Access Management (IAM)

Europe-west2: Checkpolicy <0.01% error rate (2-3 qps) from 6:02 to 11:33 PT Globalization Expand RPC served elevated latency everywhere from 6:47 to 10:02 PT.

Persistent Disk (PD)

Up to 40% of device creation operations failed in us-west2, while approximately 1% of snapshot operations were delayed or failed in us-central1, us-east1, us-west-1, and northamerica-northeast2. Existing Persistent Disk devices were not impacted.

In addition to fixing the underlying cause, we will be implementing changes to prevent, reduce the impact of, and better communicate about this type of failure in several ways:

• Enhancing safety checks of our underlying systems.
• Adding additional mechanisms that allow Google engineers to rapidly determine and fix the root cause of the service disruption.
• Improve regression testing for ACL components for error conditions.

Mini Incident Report while full Incident Report is prepared

(All Times US/Pacific)

Incident Start: 20 May 2021 05:25

Incident End: 20 May 2021 16:10

Duration: 10 hours, 45 minutes

Affected Services and Features:

Google Compute Engine - VM creations and deletions

Persistent Disk - Lifecycle Operations

Google Cloud Networking - Load Balancers

Cloud Armor - New configs

Cloud Bigtable - Data API

Cloud SQL - Instance creation, gcloud sql connect

AppEngine Flexible - Deployments

Dataproc - Cluster Creation and Deletion

BigQuery, Cloud Vertex AI, Cloud AI Platform , IAM , Cloud Console, Artifact Registry, Google Cloud Storage, Cloud Logs, Dataflow, Chemist, Cloud Composer, Cloud Data Catalog, Cloud DLP

Regions/Zones: Multi-Region

Description:

Multiple Google Cloud Products experienced elevated latencies and/or errors due to an issue with Access Control Lists (ACLs) intermittently for a duration of 10 hours and 45 minutes. Preliminary analysis revealed the cause was related to an issue impacting how ACLs are propagated. The issue was mitigated by correcting the behavior of the system responsible for distributing ACLs.

Customer Impact:

europe-west2:

BigQuery- Streaming and Query fully stuck primarily in europe-west2.

Cloud Data Catalog - Permission errors when reading from backend databases.

Cloud Logs - Delays in exporting log data.

Cloud SQL - SQL operations like Instance creations, gcloud sql connect commands either failed or experienced delays.

Cloud Bigtable - Data API experienced elevated latencies and error rates

Dataproc- Increased cluster creation and deletion latencies in europe-west2.

Cloud Vertex AI - Service availability degraded for around 10 minutes in europe-west2

Cloud AI Platform - Widespread backend errors in europe-west2.

Dataflow- Jobs were stuck in pending state. Customers were unable to cancel or terminate jobs. Jobs unable to migrate backends which resulted in stuck progress. Streaming Jobs experienced delays.

Cloud DLP - Some DLP worker tasks were down.

us-west2-a:

Google Compute Engine - Some GCE APIs experienced degraded availability. Delays in VM creation and deletion operations.

Multi-Region:

Cloud Composer - Composer environment creation and update operations failed. Existing environments may have experienced service errors.

AppEngine Flexible - “major version" (the usual way to deploy, "not in place") deployments failed with timeouts or reporting success then the deployed version served 503 errors.

Cloud Networking - Any customer initiated config change during the outage that is related to load balancing products were delayed over 3 hours. Customers that did not initiate config changes during the outage period were not affected.

Cloud Armor - New Cloud Armor configs were delayed.

Persistent Disk - Lifecycle operations such as device creations, snapshot creations etc. experienced delays or failures. No impact to live Persistent Disk devices.

Cloud Console - Customers may have experienced errors while accessing some pages on the console.

Artifact Registry - Errors for several HTTP-Maven/Docker and ScottyAgent requests for a little over 3 hours.

Google Cloud Storage - Elevated 400 errors among which most are deadline exceeded errors.

Google Kubernetes Engine - Customers were unable to create new clusters for 2 hours.

The issue with Google Cloud infrastructure components is believed to be affecting a very small number of projects and our Engineering Team is working on it.

If you have questions or are impacted, please open a case with the Support Team and we will work with you until this issue is resolved.

We thank you for your patience while we're working on resolving the issue.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: The issue with BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs, PD Lifecycle Operations, AppEngine Flexible Deployments, Cloud Logs, Cloud DLP, CMLE, Cloud SQL,Cloud Console, Artifact Registry, Dataflow, Cloud Composer, Dataproc, Cloud Data Catalog have been partially resolved.

Full resolution is expected to complete by Thursday, 2021-05-20 17:30 US/Pacific.

We will provide more information by Thursday, 2021-05-20 17:30 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: The issue with BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs, PD Lifecycle Operations, AppEngine Flexible Deployments, Cloud Logs, Cloud DLP, CMLE, Cloud SQL,Cloud Console, Artifact Registry, Dataflow, Cloud Composer, Dataproc, Cloud Data Catalog have been partially resolved.

We will provide more information by Thursday, 2021-05-20 17:00 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs, PD Lifecycle Operations, AppEngine Flexible Deployments, Cloud Logs, Cloud DLP, CMLE, Cloud SQL,Cloud Console, Artifact Registry, Dataflow, Cloud Composer, Dataproc, Cloud Data Catalog have been mitigated.

There could be some minor residual impacts pending to be resolved.

The Full resolution is still expected to complete by Thursday, 2021-05-20 16:30 US/Pacific

We will provide more information by Thursday, 2021-05-20 16:30 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs, PD Lifecycle Operations, AppEngine Flexible Deployments, Cloud Logs, Cloud DLP, CMLE, Cloud SQL,Cloud Console, Artifact Registry, Dataflow, Cloud Composer, Dataproc, Cloud Data Catalog have been mitigated.

There could be some minor residual impacts pending to be resolved.

The Full resolution is expected to complete by Thursday, 2021-05-20 16:30 US/Pacific

We will provide more information by Thursday, 2021-05-20 16:00 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs, PD Lifecycle Operations, AppEngine Flexible Deployments, Cloud Logs, Cloud DLP have been restored.

Currently known affected products: CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, Artifact Registry (europe-west1, australia-southest1), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 15:30 US/Pacific

We will provide more information by Thursday, 2021-05-20 15:30 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs have been restored.

Currently known affected products: PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 15:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 15:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs have been restored.

Currently known affected products: PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 15:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 14:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs have been restored.

Currently known affected products: PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 15:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 14:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs have been restored.

Currently known affected products: PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 14:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 14:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services, GCE APIs have been restored.

Currently known affected products: PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 14:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 13:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services have been restored.

Currently known affected products: GCE APIs (including VM creation and deletion in us-west2-a and slow VM deletion in europe-west2, europe-west3), PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

The mitigation is expected to complete by Thursday, 2021-05-20 14:00 US/Pacific

We will provide more information by Thursday, 2021-05-20 13:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services have been restored.

Currently known affected products: GCE APIs (including VM creation and deletion in us-west2-a and slow VM deletion in europe-west2, europe-west3), PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2), Cloud DLP (europe-west2)

We will provide more information by Thursday, 2021-05-20 12:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner, Bigtable, Blobstore, Spanner services have been restored.

Currently known affected products: GCE APIs (including VM creation and deletion in us-west2-a and slow VM deletion in europe-west2, europe-west3), PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Cloud Data Catalog (europe-west2)

We will provide more information by Thursday, 2021-05-20 12:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM, Spanner services have been restored. Engineering team is currently obtaining confirmation on Cloud Composer and Dataproc service status post mitigation steps.

Currently known affected products: GCE APIs (including VM creation and deletion in us-west2-a and slow VM deletion in europe-west2, europe-west3), PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry (europe-west1, australia-southest1), Cloud Logs (GCS Exports in europe-west2), Dataflow (europe-west2), Cloud Composer, Dataproc (europe-west2), Ads, Bigtable (largely recovered), Blobstore (mitigated), Colossus (mostly restored)

We will provide more information by Thursday, 2021-05-20 11:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM services have been restored. Engineering team is currently obtaining confirmation on Cloud Composer and Dataproc service status post mitigation steps.

Currently known affected products: GCE APIs (including VM creation), PD Lifecycle Operations, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry, Cloud Logs Exports, Dataflow (europe-west2), Cloud Composer, Dataproc, Ads, Bigtable, Blobstore, Colossus, Spanner

We will provide more information by Thursday, 2021-05-20 11:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. BigQuery, Cloud Networking, Cloud Armor, IAM services have been restored. Engineering team is currently obtaining confirmation on Cloud Composer and Dataproc service status post mitigation steps.

Currently known affected products: GCE APIs (including VM creation), PD Lifecycle Operations, Cloud Bigtable, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry, Cloud Logs Exports, Dataflow (europe-west2), Ads, Bigtable, Blobstore, Colossus, Spanner.

We will provide more information by Thursday, 2021-05-20 10:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. IAM service has been recovered globally. Engineering team is currently working on rolling out the potential fix through the ACL manager and will confirm the results on completion. ETA for resolution for other affected services is unknown at this time.

Currently known affected products: GCE APIs (including VM creation), PD Lifecycle Operations, Cloud Bigtable, CMLE (europe-west2), Cloud SQL (instance creations), Cloud Console, AppEngine Flexible Deployments, Artifact Registry, Cloud Logs Exports, Dataflow (europe-west2), Ads, Bigtable, Blobstore, Colossus, Spanner.

We will provide more information by Thursday, 2021-05-20 10:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. IAM service has been recovered globally. Engineering team is currently working on rolling out the potential fix through the ACL manager and will confirm the results on completion. ETA for resolution for other affected services is unknown at this time.

Currently known affected products: Bigtable, Cloud networking, Ads, Persistent Disk, Blobstore, BigQuery, GCE APIs, Cloud Composer, Cloud Console, Dataflow, Dataproc, App Engine, Control Plane.

We will provide more information by Thursday, 2021-05-20 10:15 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Mitigation work is currently underway by our engineering team. Permissions that allow services in the cloud to communicate to other cloud services have apparently been removed.

Currently known affected products: Bigtable, Cloud networking, Ads, Persistent Disk, Blobstore, BigQuery, GCE APIs, Cloud Composer, IAM, Cloud Console, Dataflow, Dataproc, App Engine, Control Plane.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2021-05-20 09:45 US/Pacific.

Diagnosis: Specific Cloud console pages are currently inaccessible for some customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Permissions that allow services in the cloud to communicate to other cloud services have apparently been removed.

Currently known affected products: Bigtable, Cloud networking, Ads, Persistent Disk, Blobstore, BigQuery, GCE APIs, Cloud Composer, IAM, Cloud Console, Dataflow, Dataproc, App Engine, Control Plane.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2021-05-20 09:15 US/Pacific with current details.

Diagnosis: Cloud console is currently inaccessible for all customers. Customers may experience issues with creating new backend services and observe latency in Cloud admin operations. Control plane operations may fail or have degraded performance.

Workaround: None at this time.

Summary: Issue with multiple Google Cloud infrastructure components.

Description: Permissions that allow services in the cloud to communicate to other cloud services have apparently been removed.

Currently known affected products: Spanner, Bigtable, IAM, Networking, GFE, Compute Engine, Big Query

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2021-05-20 08:45 US/Pacific with current details.

Diagnosis: Increased error rates and delayed operations.

Workaround: None at this time.

Summary: We are experiencing elevated error rates and delayed operations

Description: We are experiencing an issue with Google Cloud infrastructure components. You will see elevated error rates and delayed operations across multiple products.

Our engineering team continues to investigate the issue.

We will provide an update by Thursday, 2021-05-20 08:00 US/Pacific with current details.

Diagnosis: Increased error rates and delayed operations.

Workaround: None at this time.