Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Anthos Service Mesh

MDP (an istio component) evicts pods continuously when the image URL is overwritten by Kubernetes-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24

Incident began at 2023-11-21 07:43 and ended at 2023-11-21 15:33 (all times are US/Pacific).

Previously affected location(s)

Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Doha (me-central1)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)

Date Time Description
21 Nov 2023 15:33 PST

The issue with Anthos Service Mesh interacting with k8s-digester has an impact on a very small number of projects having a fully managed GCP service.

We sincerely apologize for any inconvenience this may cause. Please rest assured that our engineering teams are working diligently to mitigate all known impacts.

In the meantime, if you have any questions or are impacted by this issue, please open a case with our support team, and we will work with you until this issue is resolved.

** Please go to https://console.cloud.google.com/support/cases for more information on how to open a support case.**

We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible.
21 Nov 2023 11:33 PST

Summary: MDP (an istio component) evicts pods continuously when the image URL is overwritten by Kubernetes-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24

Description: We are experiencing an issue with Anthos Service Mesh beginning at Tuesday, 2023-11-21 06:39 US/Pacific.

Our engineering team continues to investigate the issue. we sincerely apologize for any inconvenience this may cause. Please rest assured that our engineering teams are working diligently to mitigate all known impacts.

We will provide an update by Tuesday, 2023-11-21 20:00 US/Pacific with current details.

Diagnosis: Customers may experience their Anthos Service Mesh (ASM) pods being continuously (and sometimes simultaneously) evicted when the proxy image URL is overwritten by Kubernetes-digester.

Workaround: As a workaround, we suggest that:-

  • Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:

https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional

** Customers disable Managed ASM individually on each namespace with the following commands:

kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'
21 Nov 2023 08:18 PST

Summary: MDP (an istio component) evicts pods continuously if image url is overwritten by k8s-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24

Description: We've received a report of an issue with Anthos Service Mesh as of Tuesday, 2023-11-21 06:39 US/Pacific.

We will provide more information by Tuesday, 2023-11-21 12:00 US/Pacific.

Diagnosis: Anthos Service Mesh (ASM) pods are continuously (and sometimes simultaneously) evicted if the proxy image url is overwritten by k8s-digester.

Workaround: Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:

https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional

Please disable Managed ASM individually on each namespace with the following commands:

kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'
21 Nov 2023 07:53 PST

Summary: MDP (an istio component) evicts pods continuously if image url is overwritten by k8s-digester for versions of istio: 1.14.6-asm.23 & 1.14.6-asm.24

Description: We've received a report of an issue with Anthos Service Mesh as of Tuesday, 2023-11-21 06:39 US/Pacific.

We will provide more information by Wednesday, 2023-11-29 10:00 US/Pacific.

Diagnosis: Anthos Service Mesh (ASM) pods are continuously (and sometimes simultaneously) evicted if the proxy image url is overwritten by k8s-digester.

Workaround: Customers should disable MDP if they are impacted. Instruction how to disable MDP is available here:

https://cloud.google.com/service-mesh/docs/managed/provision-managed-anthos-service-mesh#disable_the_managed_data_plane_optional

Please disable Managed ASM individually on each namespace with the following commands:

kubectl annotate --overwrite controlplanerevision -n istio-system
REVISION_LABEL mesh.cloud.google.com/proxy='{"managed":"false"}'