We are continuing to work on a fix for this issue.
Posted Oct 18, 2022 - 19:29 CEST
Identified
On October 17th 2022, at around 5pm GMT, The queuing system in charge of handling Github Check Runs and Real Time Monitoring accumulated tasks that were slow to process and impacted the whole system. The size of the task queue started increasing for several hours straight, delaying processing for all customers. until it started decreasing again and went back to normal the day after at around 1am GMT, where processing time was back to normal.
This event occurred again on October 18th 2022, at around 4pm GMT, but has been detected earlier and have been solved at 5pm GMT.
Actions were taken to avoid this issue to happen again: It includes a better monitoring of those queues to improve time to detect an issue. The montoring of processing latencies on a end-user point of view will be improved too.
More long terms actions are also planned, including implementing mechanisms to avoid accumulating latency in case of slow tasks accumulations.
Posted Oct 18, 2022 - 19:29 CEST
This incident affected: Internal Repositories Monitoring (Webhooks).