Data processing delay
Incident Report for Skylight
Postmortem

We have published a post-mortem for this incident on our blog.

Posted 8 days ago. Dec 04, 2018 - 22:00 PST

Resolved
We have finished processing the backlog and returned to normal data processing capacity.
Posted 12 days ago. Nov 30, 2018 - 16:39 PST
Update
We've deployed additional resources to resolve the final backlog.
Posted 12 days ago. Nov 30, 2018 - 13:02 PST
Update
We are continuing to monitor for any further issues.
Posted 12 days ago. Nov 30, 2018 - 10:42 PST
Update
The backlog has been resolved for most applications. We anticipate that the remaining processing will be resolved within the next day.
Posted 12 days ago. Nov 30, 2018 - 07:15 PST
Update
We have finished processing the backlog for the majority of our customers. We expect to be fully caught up in the next few hours and return to the normal data processing capacity for Friday daytime (U.S.) traffic.
Posted 13 days ago. Nov 30, 2018 - 01:14 PST
Update
Data processing has caught up for some applications. We continue to work on the backlog for the remaining applications.
Posted 13 days ago. Nov 29, 2018 - 17:20 PST
Update
We are continuing to progress and slowly catch up with the backlog. In the meantime, we're investigating ways to process the data more quickly.
Posted 13 days ago. Nov 29, 2018 - 12:44 PST
Update
Note, the uptick "Data Processing Delay" is due to raising the maximum value displayed. Previously, 360 was the highest value we would show. We are now displaying the true value.
Posted 13 days ago. Nov 29, 2018 - 09:13 PST
Update
Our workers continue to process data for all applications. We still anticipate a full recovery without data loss.
Posted 13 days ago. Nov 29, 2018 - 07:58 PST
Monitoring
We've deployed the hotfix to all of our workers and are processing data for all applications. It may take some time before processing is caught up, but we do not anticipate data loss. We continue to work towards a more permanent resolution to the issue.
Posted 14 days ago. Nov 28, 2018 - 17:41 PST
Identified
We've identified the issue and have a hotfix partially deployed. Data processing has resumed for some applications and we anticipate that it will resume for others soon. At this point, we do not believe that any data has been lost.
Posted 14 days ago. Nov 28, 2018 - 16:17 PST
Investigating
We are currently investigating this issue.
Posted 15 days ago. Nov 28, 2018 - 02:37 PST
This incident affected: Data Processing (Application).