Console

Service Health

This page provides status information on the services that are part of Google Cloud. Check back here to view the current status of the services listed below. If you are experiencing an issue not listed here, please contact Support. Learn more about what's posted on the dashboard in this FAQ. For additional information on these services, please visit https://cloud.google.com/.

Incident affecting Google App Engine

Stale composite indexes on Google App Engine

Incident began at 2015-06-02 10:15 and ended at 2015-06-02 19:52 (all times are US/Pacific).

Date Time Description
9 Jun 2015 05:56 PDT

Approximately 0.01% of Google App Engine apps are experiencing stale composite indexes for recently updated Datastore entities. Partial remediation has been applied but some Datastore queries may still get old results. Google Engineers are working to complete the remediation.

To test if you are affected, use the App Engine Datastore Viewer to query for a suspected stale entity both by it's key and it's indexed value. If you cannot find the entity by index but can see the entity by key you may be affected. Example queries to do this are:

SELECT key FROM YourKind WHERE key__=Key('YourKind', 'id_string')

and

SELECT __key FROM YourKind WHERE indexed_col1='value1' and indexed_col_2='value2'

For those affected, we apologize for any inconvenience you may be experiencing. We will provide an update by 14:00 US/Pacific time with current details.
2 Jun 2015 20:05 PDT

The remediation of missing index entries has been completed. We apologize for the inconvenience and thank you for your patience and continued support.
2 Jun 2015 18:05 PDT

Some of the missing index entries have been fixed and the remaining ones will require additional time. We will provide another status update by 20:00 US/Pacific with current details.
2 Jun 2015 15:58 PDT

Google Engineers have identified the issue with Google App Engine Datastore composite indexes missing entries. Remediation has been applied so that all future writes will correctly update indexes. Over the next several hours all remaining missing index entries will be fixed. We will provide another status update by 18:00 US/Pacific with current details.
2 Jun 2015 15:02 PDT

The investigation into Google App Engine stale composite indexes is ongoing. Current data indicates the issue is isolated to approximately 0.01% of applications running in the US. We will provide another status update by 16:00 US/Pacific with current details.
2 Jun 2015 14:03 PDT

We are still investigating the issue with a small number of Google App Engine apps experiencing stale composite indexes for recently updated Datastore entities. Current data indicates the issue only affects apps located in the US. Approximately 0.01% of apps are affected by this issue. We will provide another status update by 15:00 US/Pacific with current details.