Service Health

The issue with Healthcare and Life Sciences is believed to have affected a very small number of projects and our Engineering Team is working to recover replay notifications that were not processed between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.

If you are impacted, the team will reach out with individual communications. If you have questions, please open a case with the Support Team and we will work with you until this issue is resolved.

No further updates will be provided here.

We thank you for your patience while we're working on resolving the issue.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.

The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.

We do not have an ETA for replay of dropped notifications at this point.

We will provide more information by Tuesday, 2023-07-18 15:00 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.

The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.

We do not have an ETA for replay of dropped notifications at this point.

We will provide more information by Monday, 2023-07-17 14:00 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team mitigated any further dropped notifications and resolved the delays as of July 15, 2023 9:41 AM Pacific Time. The issue is fully mitigated for ongoing traffic.

The engineering team is working to replay notifications that were not processed, which impacted less than 5% of traffic between 2023-07-11 08:30 US/Pacific and 2023-07-13 14:00 US/Pacific.

We do not have an ETA for replay of dropped notifications at this point.

We will provide more information by Monday, 2023-07-17 13:15 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team has mitigated any further dropped notifications and resolved the delays. The issue is fully mitigated for ongoing traffic.

The engineering team is working to recover notifications that were previously dropped, which was a small percentage of overall traffic.

We do not have an ETA for replay of dropped notifications at this point.

We will provide more information by Sunday, 2023-07-16 13:00 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team has mitigated any further dropped notifications and resolved the delays. The issue is fully mitigated for ongoing traffic.

The engineering team is working to recover notifications that were previously dropped, which was a small percentage of overall traffic.

We do not have an ETA for mitigation at this point.

We will provide more information by Sunday, 2023-07-16 13:00 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) in the us-meta region are being delayed. This impacts Pub/Sub notifications, streaming to BigQuery, streaming de-identification, and could delay search indexing

Description: The team has mitigated any further dropped notifications and is working to recover notifications that were previously dropped, which was a small percentage of overall traffic.

The engineering team continues to investigate the core issue of delayed notifications and mitigation work is underway.

We do not have an ETA for mitigation at this point.

We will provide more information by Saturday, 2023-07-15 13:15 US/Pacific.

Diagnosis: Data is being successfully written to the Healthcare API but may be delayed or missing from streaming and notifications to other services. Search results within the Healthcare API may be stale.

Workaround: None at this time.

Summary: Change notifications for Healthcare API services (FHIR, DICOM, HL7v2) are being delayed or dropped. This impacts search indexing, Pub/Sub notifications, streaming to BigQuery, and streaming de-identification.

Description: We are experiencing an issue with Healthcare and Life Sciences beginning at Saturday, 2023-07-12 10:13 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-07-15 10:55 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: None at this time.

Workaround: None at this time.

Summary: We are investigating an issue with Healthcare and Life Sciences.

Description: We are experiencing an issue with Healthcare and Life Sciences beginning at Saturday, 2023-07-12 10:13 US/Pacific.

Our engineering team continues to investigate the issue.

We will provide an update by Saturday, 2023-07-15 08:55 US/Pacific with current details.

We apologize to all who are affected by the disruption.

Diagnosis: None at this time.

Workaround: None at this time.