Incident Report

Summary

Starting on 4 December 2024 at 14:30 US/Pacific, Google BigQuery experienced elevated invalid value and internal system errors globally for traffic related to BigQuery and Google Drive integration for 3 hours and 25 minutes. The incident affected users and tasks attempting to export data to Google Drive, resulting in failed export jobs.

The impacted users would have encountered “API key not valid” and “Failed to read the spreadsheet” errors for export jobs when accessing Google Drive. This resulted in service unavailability or failing jobs for the duration of this disruption for Google BigQuery.

To our BigQuery customers whose business analytics were impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

This disruption of data export functionality was triggered by the deletion of an internal project containing essential API keys. This deletion was an unintended consequence of several contributing factors:

• An internally used API key was flagged for a Google policy non-compliance and deemed no longer in use which led to the deleting of the API key.
• Unclear Internal Google Project Ownership: The project ownership was not clearly recorded, and thus incorrectly associated with a deprecated service.
• Outdated Information: The combination of the perceived lack of recent activity and the incorrect ownership led to the project being mistakenly classified as abandoned and deprecated.

Remediation and Prevention

Google engineers were alerted to the service degrading via a support case on 4 December 2024, at approximately 14:21 US/Pacific when users began experiencing failures in data export operations. Google Engineers were alerted to the service disruption through internal monitoring systems and user reports. Upon investigation, the deletion of the project was identified as the root cause.

To mitigate the impact, the project was restored at approximately 15:45 US/Pacific. This action successfully recovered the API keys and over time restored the data export functionality for all affected users. The final error related to this incident was observed at approximately 17:55 US/Pacific, indicating full service recovery.

Google is committed to continually improving our technology and operations to prevent service disruptions. We apologize for any inconvenience this incident may have caused and appreciate your understanding.

Google is committed to preventing a repeat of this issue in the future and is completing the following actions.

• Remove dependency on API keys for BigQuery integrations with other Google services: This will eliminate the entire failure mode.

• Implement accidental deletion protection for critical internal resources: Use mechanisms like project liens to ensure that a critical resource cannot be deleted accidentally.

• Enhance Project Metadata: We are implementing a process for regular review and validation of project ownership and metadata. This will ensure that critical information about project usage and status is accurate and up-to-date, reducing the risk of incorrect assumptions about project status.

• Strengthen Internal Processes and Access Controls: We are strengthening our processes for deprecating and deleting projects, including mandatory reviews, impact assessments, and stakeholder approvals. This will prevent accidental deletion of critical projects and ensure that all potential impacts are thoroughly evaluated before any action is taken. We are also strengthening access controls for project deletion, ensuring that only authorized personnel with appropriate approvals can perform this action. This will add an additional layer of protection against unintended project deletion.

Detailed Description of Impact

Starting on 4 December 2024, Google BigQuery experienced elevated error rates for data export operations to Google Drive globally. Between approximately 14:21 and 18:04 US/Pacific, users attempting to export data from BigQuery to Google Drive encountered failures, resulting in service disruption for this specific functionality.

The incident affected all regions and impacted users encountered errors such as "API key not valid," "Failed to read the spreadsheet," or "[Error: 80324028]". Internal error messages further specified the issue as "Dremel returned third-party error from GDRIVE: FAILED_PRECONDITION: Encountered an error while creating temporary directory" with an underlying status of "Http(400) Bad Request, API key not valid. Please pass a valid API key."

Google BigQuery

This disruption specifically impacted users and automated tasks relying on the BigQuery to Google Drive export functionality. Export jobs initiated during this period failed to complete, preventing data transfer and potentially impacting downstream processes and workflows dependent on this data.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support or to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 4 December, 2024 14:30

Incident End: 4 December, 2024 17:55

Duration: 3 hours, 25 minutes

Affected Services and Features:

Google BigQuery

Regions/Zones: Global

Description:

Google BigQuery experienced elevated invalid value and internal system errors globally for traffic related to BigQuery and Google Drive integration for 3 hours and 25 minutes. From preliminary analysis, the root cause has been identified as a misconfiguration in BigQuery's Google Drive integration. Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

The impacted users would have encountered “API key not valid” and “Failed to read the spreadsheet” errors for export jobs when accessing Google Drive. This resulted in service unavailability or failing jobs for the duration of this disruption for Google BigQuery.

The issue with Google BigQuery accessing Google Drive has been resolved for all affected projects as of Wednesday, 2024-12-04 18:10 US/Pacific.

We will publish an analysis of this incident once we have completed our internal investigation.

We thank you for your patience while we worked on resolving the issue.

Summary: We are working on mitigating elevated errors with BigQuery accessing Google Drive.

Description: We are working on a mitigation for the issue that is causing elevated error rates for query and export jobs that access Google Drive starting Wednesday 2024-12-04 14:30 US/Pacific. The root cause of the issue is understood, and our engineering team is working on mitigating the issue. As of 17:30 US/Pacific, the majority of the customers are no longer impacted.

We will provide more information by Wednesday, 2024-12-04 18:30 US/Pacific.

We apologize to all those affected by the issue.

Diagnosis: Impacted customers would encounter elevated errors or invalid values with “Error: 80324028” while accessing the Google Drive and these jobs would fail.

Workaround: There is no workaround at this time.

Summary: We are experiencing elevated errors with BigQuery.

Description: As of Wednesday 2024-12-04 14:30 US/Pacific, we are experiencing elevated errors with BigQuery related to query and export jobs that access Google Drive.

Mitigation work is currently underway by our engineering team

The mitigation is expected to complete by Wednesday, 2024-12-04 17:30 US/Pacific.

We will provide the next update by Wednesday, 2024-12-04 18:00 US/Pacific.

Diagnosis: Impacted customers would encounter elevated errors or invalid values with “Error: 80324028” while accessing the Google Drive and these jobs would fail.

Workaround: There is no workaround at this time.