Service Health
Incident affecting Apigee
Multi-Region: Inconsistent Verify JWT policy execution
Incident began at 2023-03-14 16:35 and ended at 2023-03-17 02:19 (all times are US/Pacific).
Previously affected location(s)
Finland (europe-north1)South Carolina (us-east1)Los Angeles (us-west2)
Date | Time | Description | |
---|---|---|---|
| 17 Mar 2023 | 02:19 PDT | The issue with Apigee has been resolved for all affected users as of Friday, 2023-03-17 02:15 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| 16 Mar 2023 | 18:10 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point for us-east1. We will provide more information by Friday, 2023-03-17 12:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 16 Mar 2023 | 09:43 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point for us-east1. We will provide more information by Thursday, 2023-03-16 18:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 16 Mar 2023 | 08:36 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is still in progress in us-east1. Rollback has been completed and an issue is mitigated in: europe-north1 and us-west2. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 16 Mar 2023 | 03:58 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 10:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 17:59 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. A rollback is in progress and engineers have completed rollbacks for 90% of the orgs affected. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2023-03-16 06:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 12:07 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Our engineering team has determined that further investigation is required to mitigate the issue. We will provide an update by Wednesday, 2023-03-15 18:00 US/Pacific with current details. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 07:22 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 12:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 05:39 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 08:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 04:59 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 06:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 15 Mar 2023 | 01:35 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 05:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 23:32 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 03:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 20:40 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2023-03-15 01:00 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 19:17 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 21:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 18:13 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team and a rollback is currently in progress. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 19:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 17:18 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Tuesday, 2023-03-14 18:30 US/Pacific. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
| 14 Mar 2023 | 16:49 PDT | Summary: Multi-Region: Inconsistent Verify JWT policy execution Description: We are experiencing an issue with Apigee beginning at Tuesday, 2023-03-14 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Tuesday, 2023-03-14 17:20 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Customers are experiencing inconsistent execution of the Verify JWT policy in instances. Workaround: None at this time. |
- All times are US/Pacific