Data Processing Delay

Incident Report for Skylight

Resolved

We’ve caught up with all data processing. We’re continuing to investigate internally to avoid future issues.
Posted Oct 11, 2019 - 07:47 PDT

Update

Nearly all backlogged data has been processed. We are still continuing to process the backlog on the original 3% that was not being correctly handled. We anticipate resolution within the next 12 hours. Normal usage of Skylight should not be affected as 97% of the data for all apps will be present in our UI.
Posted Oct 10, 2019 - 11:23 PDT

Monitoring

We've identified the issue and deployed a hot fix. We're now processing all backlogged data.
Posted Oct 10, 2019 - 09:28 PDT

Investigating

An unanticipated side-effect from our fix has caused data to stop processing entirely for some workers. We are investigating.
Posted Oct 10, 2019 - 08:37 PDT

Identified

We've identified the cause and have resumed processing of the data. We'll update again once we've caught up.
Posted Oct 10, 2019 - 08:12 PDT

Investigating

As of approximately 3:20 UTC we’ve been unable to process ~3% of all inbound data due to a queuing issue. While the inbound data is still present in the queue, it is not yet certain whether we will be able to process it after the fact. However, we believe that some delay in resolution will not further worsen the situation or harm our chances of data recovery. As it is late in the night for many of us we will get some rest and investigate further in the morning. Our apologies for any inconvenience this may cause you.
Posted Oct 09, 2019 - 23:07 PDT
This incident affected: Data Processing (Application).