Service Health
-
-
-
-
Incident affecting Hybrid Connectivity, Google Cloud Networking
Cloud Interconnect users may have been impacted in us-east4
Incident began at 2023-11-30 05:42 and ended at 2023-11-30 12:44 (all times are US/Pacific).
Previously affected location(s)
Northern Virginia (us-east4)
| Date | Time | Description | |
|---|---|---|---|
| | 30 Nov 2023 | 12:44 PST | The issue with Cloud Interconnect has been resolved as of Thursday, 2023-11-30 12:41 US/Pacific. We understand that this issue had impacted your ability to access and use our services, we deeply appreciate your patience and cooperation while we worked to mitigate the issue. We take pride in providing our customers with a reliable and highly available service, Please rest assured of the fact that we are committed to doing everything we can to prevent issues like this from happening in the future. Thank you for choosing us. |
| | 30 Nov 2023 | 11:54 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: Our engineering team has identified a mitigation for the issue and are currently working on implementing it. At this time, we do not have a specific ETA for the mitigation to be completed. We will provide more information by Thursday, 2023-11-30 13:30 US/Pacific. We sincerely appreciate your patience and understanding as we work to resolve it as quickly as possible. Diagnosis: Customers may experience problems creating, updating or deleting Cloud Interconnect attachments in us-east4 HA1. Existing attachments are believed to be unaffected. Between approximately 05:09 and 07:03 PST, customers with Cloud Interconnect attachments in us-east4 trying to communicate with endpoints in other GCP regions may have experienced traffic imbalances in the GCP-onprem direction, as traffic may have concentrated on HA0 attachments. Workaround: There is no workaround for the management plane issue at this time. In regard to existing attachments, customers not experiencing connectivity problems are advised to take no action. Customers experiencing problems reaching GCP destinations outside us-east4 via HA1 attachments in us-east4 should consider manually failing over to HA0 attachments. |
| | 30 Nov 2023 | 08:50 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. Please rest assured that our engineering teams are working diligently to mitigate all known impacts. We will provide an update by Thursday, 2023-11-30 12:00 US/Pacific with current details. We sincerely appreciate your patience and understanding as we work to resolve this issue as quickly as possible. Diagnosis: Customers may experience problems creating, updating or deleting Cloud Interconnect attachments in us-east4 HA1. Existing attachments are believed to be unaffected. Between approximately 05:09 and 07:03 PST, customers with Cloud Interconnect attachments in us-east4 trying to communicate with endpoints in other GCP regions may have experienced traffic imbalances in the GCP-onprem direction, as traffic may have concentrated on HA0 attachments. Workaround: There is no workaround for the management plane issue at this time. In regard to existing attachments, customers not experiencing connectivity problems are advised to take no action. Customers experiencing problems reaching GCP destinations outside us-east4 via HA1 attachments in us-east4 should consider manually failing over to HA0 attachments. |
| | 30 Nov 2023 | 07:53 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 09:00 US/Pacific with current details. Diagnosis: Customers may experience problems creating, updating or deleting Cloud Interconnect attachments in us-east4 HA1. Existing attachments are believed to be unaffected. Between approximately 05:09 and 07:03 PST, customers with Cloud Interconnect attachments in us-east4 trying to communicate with endpoints in other GCP regions may have experienced traffic imbalances in the GCP-onprem direction, as traffic may have concentrated on HA0 attachments. Workaround: There is no workaround for the management plane issue at this time. In regard to existing attachments, customers not experiencing connectivity problems are advised to take no action. Customers experiencing problems reaching GCP destinations outside us-east4 via HA1 attachments in us-east4 should consider manually failing over to HA0 attachments. |
| | 30 Nov 2023 | 07:26 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 08:00 US/Pacific with current details. Diagnosis: Customers may experience problems creating, updating or deleting Cloud Interconnect attachments in us-east4 HA1. Existing attachments are believed to be unaffected. Between approximately 05:09 and 07:03 PST, customers with Cloud Interconnect attachments in us-east4 trying to communicate with endpoints in other GCP regions may have experienced traffic imbalances in the GCP-onprem direction, as traffic may have concentrated on HA0 attachments. Workaround: There is no workaround for the management plane issue at this time. In regard to existing attachments, customers not experiencing connectivity problems are advised to take no action. Customers experiencing problems reaching GCP destinations outside us-east4 via HA1 attachments in us-east4 should consider manually failing over to HA0 attachments. |
| | 30 Nov 2023 | 06:59 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 07:30 US/Pacific with current details. Diagnosis: Customers with Cloud Interconnect attachments in us-east4 trying to communicate with endpoints in other GCP regions may experience traffic imbalances in the GCP-onprem direction, as traffic may concentrate on HA0 attachments. Workaround: Customers not experiencing connectivity problems are advised to take no action. Customers experiencing problems reaching GCP destinations outside us-east4 via HA1 attachments in us-east4 should consider manually failing over to HA0 attachments. |
| | 30 Nov 2023 | 06:54 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 07:30 US/Pacific with current details. Diagnosis: Customer may may observe significant data plane loss on Cloud Interconnect and/or Cloud VPN Workaround: The problem only affects HA1 so customers should be able to fail over to HA0 |
| | 30 Nov 2023 | 06:21 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 07:00 US/Pacific with current details. Diagnosis: Customer may may observe significant data plane loss on Cloud Interconnect and/or Cloud VPN Workaround: The problem only affects HA1 so customers should be able to fail over to HA0 |
| | 30 Nov 2023 | 06:05 PST | Summary: Cloud Interconnect users may be impacted in us-east4 Description: We are experiencing an issue with Hybrid Connectivity beginning on Thursday, 2023-11-30 03:25 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Thursday, 2023-11-30 06:30 US/Pacific with current details. Diagnosis: Customer may may observe significant data plane loss on Cloud Interconnect and/or Cloud VPN Workaround: The problem only affects HA1 so customers should be able to fail over to HA0 |
- All times are US/Pacific