Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Apigee

Multi-Region: Inconsistent Verify JWT policy execution

Incident began at 2023-03-14 16:35 and ended at 2023-03-17 02:19 (all times are US/Pacific).

Previously affected location(s)

Finland (europe-north1)South Carolina (us-east1)Los Angeles (us-west2)

Date Time Description
17 Mar 2023 02:19 PDT

The issue with Apigee has been resolved for all affected users as of Friday, 2023-03-17 02:15 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

16 Mar 2023 18:10 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2.

We do not have an ETA for mitigation at this point for us-east1.

We will provide more information by Friday, 2023-03-17 12:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

16 Mar 2023 09:43 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2.

We do not have an ETA for mitigation at this point for us-east1.

We will provide more information by Thursday, 2023-03-16 18:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

16 Mar 2023 08:36 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

16 Mar 2023 03:58 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 17:59 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected.

We do not have an ETA for mitigation at this point.

We will provide more information by Thursday, 2023-03-16 06:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 12:07 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Our engineering team has determined that further investigation is required to mitigate the issue.

We will provide an update by Wednesday, 2023-03-15 18:00 US/Pacific with current details.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 07:22 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 12:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 05:39 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 08:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 04:59 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 06:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

15 Mar 2023 01:35 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 05:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 23:32 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 03:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 20:40 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Wednesday, 2023-03-15 01:00 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 19:17 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-03-14 21:30 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 18:13 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-03-14 19:30 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 17:18 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: Mitigation work is currently underway by our engineering team.

We do not have an ETA for mitigation at this point.

We will provide more information by Tuesday, 2023-03-14 18:30 US/Pacific.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.

14 Mar 2023 16:49 PDT

Summary: Multi-Region: Inconsistent Verify JWT policy execution

Description: We are experiencing an issue with Apigee beginning at Tuesday, 2023-03-14 00:00 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Tuesday, 2023-03-14 17:20 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances.

Workaround: None at this time.