Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Dataproc

Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Incident began at 2021-07-28 19:24 and ended at 2021-07-30 18:52 (all times are US/Pacific).

Date Time Description
2 Aug 2021 11:45 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 28 July 2021 19:24

Incident End: 30 July 2021 18:52

Duration: 1 day, 23 hours and 28 minutes

Affected Services and Features:

Google Cloud Dataproc - Ability to view and access Component Gateway URLs [1]

Regions/Zones:

asia-east1, asia-east2, asia-northeast1, asia-northeast3, asia-south1, asia-southeast1, asia-southeast2, australia-southeast1, europe-north1, europe-west1, europe-west2, europe-west4, europe-west6, northamerica-northeast1, southamerica-east1, us-central1, us-east1, us-east4, us-west1

Description:

Google Cloud Dataproc experienced elevated 400 errors indicating “Bad Requests” when accessing the component gateway URLs[1] for a duration of around 2 days. From preliminary analysis, the root cause of the issue was a rollout that started on 28 July 2021 at 12:45. The rollout was paused preventing further regions from being affected and a rollback started on 30 July 2021 at 9:18 to mitigate the issue in affected regions . During the incident, a workaround was provided which was to use the SSH SOCKS proxy as described in doc [2] to access the component gateway URLs.

Customer Impact:

Google Cloud Dataproc- Accessing component gateway URLs failed with 400 errors.

References:

[1] https://cloud.google.com/dataproc/docs/concepts/accessing/dataproc-gateways#viewing_and_accessing_component_gateway_urls

[2] https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 19:04 PDT

The issue with Cloud Dataproc has been resolved for all affected projects as of Friday, 2021-07-30 19:03 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

30 Jul 2021 18:18 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: We have observed reduction in the error rate and the issue is currently intermittent.

Engineering team continues to work on the mitigation. The mitigation is in progress and estimated to complete by Friday, 2021-07-30 19:00 US/Pacific.

Action: Utilize the SOCKS proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 19:00 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the SOCKS proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 15:59 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: We have observed reduction in the error rate and the issue is currently intermittent.

Engineering team continues to work on the mitigation. The mitigation is estimated to complete by Friday, 2021-07-30 18:00 US/Pacific.

Action: Utilize the SOCKS proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 18:00 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the SOCKS proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 15:59 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: We have observed reduction in the error rate and the issue is currently intermittent.

Engineering team continues to work on the mitigation. The mitigation is estimated to complete by Friday, 2021-07-30 16:00 US/Pacific.

Action: Utilize the SOCKS proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 18:00 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the SOCKS proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 13:17 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: We have observed reduction in the error rate and the issue is currently intermittent.

Engineering team continues to work on the mitigation. The mitigation is estimated to complete by Friday, 2021-07-30 16:00 US/Pacific.

Action: Utilize socks proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 16:00 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the socks proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 12:27 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: Mitigation work is still underway by our engineering team.

We do not have an ETA for mitigation at this point.

Action: Utilize socks proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 13:30 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the socks proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 11:07 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

Action: Utilize socks proxy to access their UIs as a workaround. Please refer to workaround section for more details.

We will provide more information by Friday, 2021-07-30 12:30 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: Customers can use the socks proxy to access their UIs while component gateway is not working. Please refer to the link for more details: https://cloud.google.com/dataproc/docs/concepts/accessing/cluster-web-interfaces

30 Jul 2021 10:12 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request). Unable to access clusters details page after cluster creation.

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2021-07-30 11:30 US/Pacific.

Diagnosis: Unable to access the cluster details page after cluster creation.

Workaround: None at this time.

30 Jul 2021 09:47 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request)

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Friday, 2021-07-30 11:00 US/Pacific.

Diagnosis: Increased failures with 400 bad request error

Workaround: None at this time.

30 Jul 2021 09:27 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request)

Description: We are experiencing an issue with Cloud Dataproc.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2021-07-30 10:16 US/Pacific with current details.

Diagnosis: Increased failures with 400 bad request error

Workaround: None at this time.

30 Jul 2021 08:56 PDT

Summary: Multi-region: Accessing Component Gateway fails with 400 (Bad Request)

Description: We are experiencing an issue with Cloud Dataproc.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2021-07-30 09:30 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Increased failures with 400 bad request error

Workaround: None at this time.

30 Jul 2021 08:26 PDT

Summary: Accessing Component Gateway fails with 400 (Bad Request)

Description: We are experiencing an issue with Cloud Dataproc.

Our engineering team continues to investigate the issue.

We will provide an update by Friday, 2021-07-30 09:00 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Increased failures with 400 bad request error

Workaround: None at this time.