Incident Report

Summary

On 1 December 2022 at 17:30 PT, Google Compute Engine (GCE) customers experienced issues connecting to instances via Secure Shell Protocol (SSH) through Identity Aware Proxy (IAP) [1] for a total duration of 22 hours and 46 minutes. To our GCE customers whose instances were impacted during this outage, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability. We have conducted an internal investigation and are taking steps to improve our service.

[1] https://cloud.google.com/iap

Background

IAP assists in securing user access to registered web applications and cloud resources by providing centralized access control. IAP is the default method of connectivity when customers SSH through the Google Cloud Console or gcloud tool to GCE instances which do not have external IP addresses. The IAP connection method also can be specified by organization policy or gcloud invocation flags.

Root Cause

Before allowing a connection through SSH, IAP validates the user and connects to a backend system database. That database is used to collect instance and networking information for GCE instances for validation that the instance exists. This database is updated in real-time, as GCE instances are added, changed, or removed.

Due to an issue with a global index associated with this backend system database, writes to the database were unable to keep up with instance updates from GCE and eventually caused the updates of GCE IAP information to lag. This meant that as new GCE instances were created, they were not being added to the database that IAP accessed. Therefore, IAP was unaware of the GCE instances and thus denied access to them when customers attempted to use SSH through the IAP Proxy.

Remediation and Prevention

Google engineers were initially notified of the issue through a support case on 1 December 2022 at 22:44 PT and immediately started an investigation. At 01:55 PT on 2 December, engineers determined the problem to be an internal system backend database used for instance and networking information for GCE, and the issue was escalated to our database team. Database engineers attempted several mitigations starting at 03:01 PT but experienced limited success.

At 11:01 PT, engineers redirected IAP traffic to a standby replica of the system database, which temporarily mitigated the issue at 13:38 PT. At 17:45 PT, the issue recurred, as an existing rollout mistakenly redirected traffic back to the primary database, overriding the non-standard configuration being used to mitigate the issue at the time. Engineers again redirected traffic to the database replica, which fully mitigated and resolved the issue at 20:50 PT.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions:

1. Make modifications to our alerting practices for IAP SSH to closely monitor the backlog of unprocessed instance creations and updates, enabling us to take corrective actions to mitigate any customer impact.

2. Refactor our rollout vehicles to provide more fine-grained change deployment, so that unrelated configuration changes do not also deploy core service configuration or binaries for the IAP infrastructure. This will prevent the secondary issue recurrence, which occurred at 17:45 PT.

3. Improve the performance of concurrent reads and writes to the same database rows by applying exclusive locks, which will serialize those transactions. This will effectively help eliminate transactions from becoming deadlocked and rejected.

Detailed Description of Impact

On 1 December 2022 from 17:30 PT to 2 December 2022 13:11 PT and again from 17:45 PT to 20:50 PT, GCE customers experienced issues connecting to instances via SSH through IAP.

Customers attempting to connect to GCE instances created or updated after 1 December 2022 at 17:30 PT using SSH through IAP would have experienced connection failures with “ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance'].”

This issue only impacted SSH access, and workloads continued to run correctly. Additionally, connections without IAP and connection to instances that were not modified after 1 December 2022 at 17:30 PT would not have been affected.

Mini Incident Report

We apologize for the inconvenience this service disruption may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

First Incident Start: 01 December 2022 17:30

First Incident End: 02 December 2022 13:11

First Duration: 19 hours, 41 minutes

Second Incident Start: 02 December 2022 17:45

Second Incident End: 02 December 2022 20:50

Second Duration: 3 hours, 5 minutes

Affected Services and Features: Google Compute Engine

Regions/Zones: Global

Description:

Google Compute Engine experienced issues connecting to instances via SSH through Identity Aware Proxy (IAP) for 19 hours and 41 minutes and 3 hours and 5 minutes. This issue only affected instances created or updated after 1 December 2022 at 17:30 PT. From preliminary analysis, the root cause of the issue was an internal distributed database that contained data critical to IAP access that was not updating correctly. We will be providing a full root cause analysis in an Incident Report that will be published in the next several days.

Customer Impact:

Customers attempting to connect to GCE instances created or updated after 1 December 2022 at 17:30 PT using SSH through IAP would have experienced connection failures with “ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance'].”

This issue only impacted SSH access, and workloads continued to run correctly. Additionally, connections without IAP and connection to instances that were not modified after 1 December 2022 at 17:30 PT would not have been affected.

The issue with Google Compute Engine has been resolved for all affected projects as of Friday, 2022-12-02 13:11 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: We believe that this incident has been mitigated as of 13:11 US/Pacific; however we are continuing to monitor.

We will provide an update by Friday, 2022-12-02 16:30 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy and connection to the VMs not modified recently should continue to work.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method: https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

For more information about Identity Aware Proxy (IAP), please see: https://cloud.google.com/iap/docs/tcp-forwarding-overview

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team continues to identify a mitigation strategy and determined that further investigation is required.

We will provide an update by Friday, 2022-12-02 14:30 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method: https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

For more information about Identity Aware Proxy (IAP), please see: https://cloud.google.com/iap/docs/tcp-forwarding-overview

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team continues to identify a mitigation strategy and determined that further investigation is required.

We will provide an update by Friday, 2022-12-02 12:30 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Friday, 2022-12-02 10:30 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Friday, 2022-12-02 09:30 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 09:00 US/Pacific.

Diagnosis: UPDATE: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:00 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 08:20 US/Pacific.

Diagnosis: UPDATE: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:00 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Friday, 2022-12-02 07:30 US/Pacific with current details.

Diagnosis: UPDATE: We believe that this is only impacting VMs which were created or modified since 2022-12-01 17:30 PST, the VMs who were stably running longer than that should work fine.

The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Friday, 2022-12-02 07:00 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine UPDATE: The VMs are not accessible via SSH after creation/deletion since the start of the issue. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Friday, 2022-12-02 06:56 US/Pacific with current details.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible via SSH for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is still underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 06:00 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible via SSH for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 05:00 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible via SSH for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 04:30 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible via SSH for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 04:00 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible via SSH for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working.

Also, we believe this is only impacting the SSH access, and the workloads continue to running correctly

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

The VMs are not accessible for some time after creation/deletion, but the proxy is making progress to overtime catchup. In the meantime, connections without the proxy as well as connection to VMs not modified recently should continue working

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 03:40 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 03:10 US/Pacific.

Diagnosis: We believe that this is only impacting VMs which were created or modified within last 6 hours. the VMs who were stably running longer than that should work fine

ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

For internal IP, use bastion host connection method. https://cloud.google.com/compute/docs/instances/connecting-advanced#bastion_host

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2022-12-02 03:10 US/Pacific.

Diagnosis: ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: For instances with external IPs, update the SSH URL in Cloud Console to have useAdminProxy=false. For gcloud CLI, don't use the --tunnel-through-iap flag.

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: We are experiencing an issue with Google Compute Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-12-02 03:10 US/Pacific with current details.

Diagnosis: ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: Try SSH access to instances with useAdminProxy=false in gCloud CLI. Try using SSH to public IPs with useAdminProxy=false

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: We are experiencing an issue with Google Compute Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-12-02 02:40 US/Pacific with current details.

Diagnosis: ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: Try SSH access to instances with useAdminProxy=false in gCloud CLI

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: We are experiencing an issue with Google Compute Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-12-02 02:30 US/Pacific with current details.

Diagnosis: ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: None at this time.

Summary: Google Compute Engine. ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Description: We are experiencing an issue with Google Compute Engine.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2022-12-02 02:20 US/Pacific with current details.

Diagnosis: ssh fails with ERROR: (gcloud.compute.start-iap-tunnel) Error while connecting [4047: 'Failed to lookup instance']

Workaround: None at this time.