Service Health
Incident affecting Cloud Memorystore, Memorystore for Redis
Issue with ‘Cloud Memorystore for Redis’ instances
Incident began at 2023-10-02 19:55 and ended at 2023-10-08 21:40 (all times are US/Pacific).
Previously affected location(s)
Taiwan (asia-east1)Hong Kong (asia-east2)Tokyo (asia-northeast1)Osaka (asia-northeast2)Seoul (asia-northeast3)Mumbai (asia-south1)Delhi (asia-south2)Singapore (asia-southeast1)Jakarta (asia-southeast2)Sydney (australia-southeast1)Melbourne (australia-southeast2)Warsaw (europe-central2)Finland (europe-north1)Madrid (europe-southwest1)Belgium (europe-west1)Turin (europe-west12)London (europe-west2)Frankfurt (europe-west3)Netherlands (europe-west4)Zurich (europe-west6)Milan (europe-west8)Paris (europe-west9)Dammam (me-central2)Tel Aviv (me-west1)Montréal (northamerica-northeast1)Toronto (northamerica-northeast2)São Paulo (southamerica-east1)Santiago (southamerica-west1)Iowa (us-central1)South Carolina (us-east1)Northern Virginia (us-east4)Columbus (us-east5)Dallas (us-south1)Oregon (us-west1)Los Angeles (us-west2)Salt Lake City (us-west3)Las Vegas (us-west4)
Date | Time | Description | |
---|---|---|---|
| 8 Oct 2023 | 21:40 PDT | The issue with Cloud Memorystore has been resolved for all affected users as of Sunday, 2023-10-08 21:35 US/Pacific. We thank you for your patience while we worked on resolving the issue. |
| 3 Oct 2023 | 12:12 PDT | Summary: We are investigating an issue with some ‘Cloud Memorystore for Redis’ instances that fail health checks and are not being repaired correctly, leaving them in an unavailable state. Description: A fix rollout is currently underway and our engineering team will continue to mitigate the unhealthy nodes as they appear until the fix is rolled out. We expect the fix roll out to complete by Monday, 2023-10-09 We will provide more information by Monday, 2023-10-09 14:00 US/Pacific. Diagnosis: Impacted ‘Memorystore for Redis’ customers may observe unresponsive nodes. Please note that ‘Memorystore for Memcached’ and ‘Memorystore for Redis Cluster’ are not impacted by this issue Workaround: None at this time. |
| 3 Oct 2023 | 09:54 PDT | Summary: We are investigating an issue with some ‘Cloud Memorystore for Redis’ instances that fail health checks and are not being repaired correctly, leaving them in an unavailable state. Description: Engineering continues mitigating the unhealthy nodes, and a fix rollout is currently underway. We do not have an ETA for full mitigation at this point. We will provide more information by Tuesday, 2023-10-03 12:00 US/Pacific. Diagnosis: Impacted ‘Memorystore for Redis’ customers may observe unresponsive nodes. Please note that ‘Memorystore for Memcached’ and ‘Memorystore for Redis Cluster’ are not impacted by this issue Workaround: None at this time. |
| 3 Oct 2023 | 00:29 PDT | Summary: We are investigating an issue with some ‘Cloud Memorystore for Redis’ instances that fail health checks and are not being repaired correctly, leaving them in an unavailable state. Description: Engineering continues mitigating the unhealthy nodes, and a fix rollout is currently underway. We do not have an ETA for full mitigation at this point. We will provide more information by Tuesday, 2023-10-03 10:00 US/Pacific. Diagnosis: Impacted ‘Memorystore for Redis’ customers may observe unresponsive nodes. Please note that ‘Memorystore for Memcached’ and ‘Memorystore for Redis Cluster’ are not impacted by this issue Workaround: None at this time. |
| 2 Oct 2023 | 21:35 PDT | Summary: We are investigating an issue with some Cloud Memorystore instances that fail health checks are not being repaired correctly, leaving them in an unavailable state. Description: Engineering is mitigating the unhealthy nodes and a fix roll out is under progress. We do not have an ETA for full mitigation at this point. We will provide more information by Tuesday, 2023-10-03 10:00 US/Pacific. Diagnosis: Impacted customers may observe unresponsive nodes Workaround: None at this time. |
| 2 Oct 2023 | 20:59 PDT | Summary: We've received a report of an issue with Cloud Memorystore. Description: We are experiencing an issue with Cloud Memorystore beginning at Monday, 2023-10-02 00:00 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Monday, 2023-10-02 22:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Impacted customers may observe unresponsive nodes Workaround: None at this time. |
- All times are US/Pacific