We’ve caught up with all data processing. We’re continuing to investigate internally to avoid future issues.
Posted Oct 11, 2019 - 07:47 PDT
Update
Nearly all backlogged data has been processed. We are still continuing to process the backlog on the original 3% that was not being correctly handled. We anticipate resolution within the next 12 hours. Normal usage of Skylight should not be affected as 97% of the data for all apps will be present in our UI.
Posted Oct 10, 2019 - 11:23 PDT
Monitoring
We've identified the issue and deployed a hot fix. We're now processing all backlogged data.
Posted Oct 10, 2019 - 09:28 PDT
Investigating
An unanticipated side-effect from our fix has caused data to stop processing entirely for some workers. We are investigating.
Posted Oct 10, 2019 - 08:37 PDT
Identified
We've identified the cause and have resumed processing of the data. We'll update again once we've caught up.
Posted Oct 10, 2019 - 08:12 PDT
Investigating
As of approximately 3:20 UTC we’ve been unable to process ~3% of all inbound data due to a queuing issue. While the inbound data is still present in the queue, it is not yet certain whether we will be able to process it after the fact. However, we believe that some delay in resolution will not further worsen the situation or harm our chances of data recovery. As it is late in the night for many of us we will get some rest and investigate further in the morning. Our apologies for any inconvenience this may cause you.
Posted Oct 09, 2019 - 23:07 PDT
This incident affected: Data Processing (Application).