Data Processing Incident
Incident Report for Skylight
Resolved
We have fully recovered from the incident and will continue to process new data normally. Unfortunately, we are not able to recover any lost data between 1:50 PM and 2:50 PM (Pacific Time). As a result, you may see small "gaps" in the Skylight UI during that time period. We are sorry for the inconvenience. We will resume the hardware upgrade tomorrow, but we will put in additional safe guards to prevent the same issue.
Posted Dec 18, 2018 - 16:12 PST
Monitoring
We have stabilized the system and are processing new data normally.
Posted Dec 18, 2018 - 15:06 PST
Update
We are observing increased error rates on the agent data collection endpoints (i.e. where the agent sends performance data to). This will not prevent your apps from functioning correctly, but we may not be able to process all the data sent during this time correctly and you may see 'gaps' in the Skylight UI. This is expected behavior at this point, we will post another update when we have determined the next steps. We are sorry for the inconvenience.
Posted Dec 18, 2018 - 14:48 PST
Investigating
We are performing some hardware upgrades and have encountered some unexpected instability. At the moment we still expect some gaps in data between 1:50pm and 2:50pm PST.
Posted Dec 18, 2018 - 14:37 PST
This incident affected: Data Processing (Application).