Service Health

The issue with Cloud Build, Google Cloud Networking, Google Kubernetes Engine has been resolved for all affected users as of Wednesday, 2022-12-21 19:24 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Summary: us-central1: Degraded performance for Cloud Networking

Description: At this time we believe that impact to Google Kubernetes Engine and Cloud Build have been mitigated.

Mitigation work for Cloud Networking remains underway by our engineering team.

We do not have an ETA for full mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 20:30 US/Pacific.

Diagnosis: Cloud Networking: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Google Kubernetes Engine: Some GKE operations are experiencing issues with cluster creation, node pool creation, cluster autoscaling, node auto repair and node pool upgrades. New node creation during upgrades will fail and can cause upgrade operation failure however, it will not cause any disruption to running nodes.

Cloud Build: Customers using private pools, or nonstandard VM shapes in the shared pool, are observing high queue latencies.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Affected Cloud Build customers may build in a different region, or use a standard-sized VM in the shared pool.

Summary: us-central1: Degraded performance for Cloud Networking

Description: At this time we believe that impact to Google Kubernetes Engine and Cloud Build have been mitigated.

Mitigation work for Cloud Networking remains underway by our engineering team.

We do not have an ETA for full mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 19:30 US/Pacific.

Diagnosis: Cloud Networking: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Google Kubernetes Engine: Some GKE operations are experiencing issues with cluster creation, node pool creation, cluster autoscaling, node auto repair and node pool upgrades. New node creation during upgrades will fail and can cause upgrade operation failure however, it will not cause any disruption to running nodes.

Cloud Build: Customers using private pools, or nonstandard VM shapes in the shared pool, are observing high queue latencies.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Affected Cloud Build customers may build in a different region, or use a standard-sized VM in the shared pool.

Summary: us-central1: Degraded performance for Cloud Networking

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 17:30 US/Pacific.

Diagnosis: Cloud Networking: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Google Kubernetes Engine: Some GKE operations are experiencing issues with cluster creation, node pool creation, cluster autoscaling, node auto repair and node pool upgrades. New node creation during upgrades will fail and can cause upgrade operation failure however, it will not cause any disruption to running nodes.

Cloud Build: Customers using private pools, or nonstandard VM shapes in the shared pool, are observing high queue latencies.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Affected Cloud Build customers may build in a different region, or use a standard-sized VM in the shared pool.

Summary: us-central1: Degraded performance for Cloud Networking

Description: Some GKE operations are experiencing issues with cluster creation, node pool creation, cluster autoscaling, node auto repair and node pool upgrades.

New node creation during upgrades will fail and can cause upgrade operation failure however, it will not cause any disruption to running nodes.

Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 17:30 US/Pacific.

Diagnosis: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Summary: us-central1: Degraded performance for Cloud Networking

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 17:00 US/Pacific.

Diagnosis: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Summary: us-central1: Degraded performance for Cloud Networking

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2022-12-21 16:00 US/Pacific.

Diagnosis: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Summary: us-central1: Degraded performance for Cloud Networking

Description: We are experiencing an issue with Google Cloud Networking for the region us-central1.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-21 15:00 US/Pacific with current details.

Diagnosis: Creations/deletions/modifications cannot be completed. Eg: New networking endpoints creations, VM auto-scaling, etc will be affected. End users would see packet drops or connectivity failures for new VMs and other networking endpoints.

Workaround: Avoid VM migrations and consider failing over the affected traffic to others regions.

Summary: us-central1: Degraded performance for Cloud Networking

Description: We are experiencing an issue with Google Cloud Networking.

Our engineering team continues to investigate the issue.

We will provide an update by Wednesday, 2022-12-21 14:30 US/Pacific with current details.

Diagnosis: Customers will observe that new endpoints or modifications to existing endpoints are not propagated. And users will observe packet drops.

Workaround: None at this time.