Google Cloud Status Dashboard

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google Cloud Scheduler

Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Incident began at 2021-07-21 19:22 and ended at 2021-07-27 15:28 (all times are US/Pacific).

Date Time Description
28 Jul 2021 13:32 PDT

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Support by opening a case using https://cloud.google.com/support

(All Times US/Pacific)

Incident Start: 21 July 2021 19:22

Incident End: 27 July 2021 15:28

Duration: 5 days, 20 hours, 6 minutes

Affected Services and Features:

Google Cloud Scheduler Pub/Sub

Regions/Zones: All Regions

Description:

Google Cloud Scheduler jobs experienced increased errors globally when publishing messages to Pub/Sub topics for a duration of 5 days, 20 hours, 6 minutes. From preliminary analysis, the root cause of the issue is due to a configuration change that updated the service agent used when publishing to Pub/Sub. Projects using the new service agent without the correct permissions resulted in PERMISSION_DENIED errors for tasks that required publishing to Pub/Sub.

Customer Impact:

All customers with Cloud Scheduler jobs with a Pub/Sub topic as a target that did not grant the Cloud Scheduler Google-managed service account access to that Pub/Sub topic saw PERMISSION_DENIED errors.

Additional details:

The issue was fully resolved on 27 July 2021 at 15:28 US/Pacific after a rollback of the change was completed.

27 Jul 2021 15:46 PDT

The issue with Cloud Scheduler has been resolved for all affected projects as of Tuesday, 2021-07-27 15:43 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

27 Jul 2021 15:23 PDT

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Description: We believe the issue with Cloud Scheduler is partially resolved.

We do not have an ETA for full resolution at this point.

We will provide an update by Tuesday, 2021-07-27 16:01 US/Pacific with current details.

Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED

Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

27 Jul 2021 13:20 PDT

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Description: We believe the issue with Cloud Scheduler is partially resolved and there is no further impact observed.

Action:

  • Customers should utilize the Cloud Services Robot account for authentication.

We will provide an update by Tuesday, 2021-07-27 15:30 US/Pacific with current details.

Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED

Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

27 Jul 2021 12:28 PDT

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Description: We believe the issue with Cloud Scheduler is partially resolved and there is no further impact observed.

Action:

  • Customers should utilize the Cloud Services Robot account for authentication.

We will provide an update by Tuesday, 2021-07-27 13:30 US/Pacific with current details.

Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED

Workaround: Add the permission pubsub.topics.publish to Cloud Scheduler service account (service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com).

27 Jul 2021 11:03 PDT

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Description: Mitigation work is still underway by our engineering team.

The rollback activity is currently 50% complete and ongoing.

We will provide more information by Tuesday, 2021-07-27 13:00 US/Pacific.

Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED

Workaround: Add publisher role to Cloud Scheduler service account. The service account has the form service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com

27 Jul 2021 09:10 PDT

Summary: Global: Cloud Scheduler Pub/Sub jobs fail with permission denied

Description: This is a continuation of the previous post for incident "Global: Cloud Scheduler Pub/Sub jobs fail with permission denied" that was closed as resolved.

We have received updates from our engineering team that the Mitigation work is still underway for some regions and are currently waiting for an ETA.

We will provide more information by Tuesday, 2021-07-27 11:00 US/Pacific.

Diagnosis: Receiving Cloud Scheduler PERMISSION_DENIED

Workaround: Add publisher role to Cloud Scheduler service account. The service account has the form service-PROJECT_NUMBER@gcp-sa-cloudscheduler.iam.gserviceaccount.com