Resolved -
On June 28th, 2024, at 16:06 UTC, a backend update by GitHub triggered a significant number of long-running Organization membership update jobs in our job processing system. The job queue depth rose as these update jobs consumed most of our job worker capacity. This resulted in delays for other jobs across services such as Pull Requests and PR-related Actions workflows. We mitigated the impact to Pull Requests and Actions at 19:32 UTC by pausing all Organization membership update jobs. We deployed a code change at 22:30 UTC to skip over the jobs queued by the backend change and re-enabled Organization membership update jobs. We restored the Organization membership update functionality at 22:52 UTC, including all membership changes queued during the incident.
During the incident, about 15% of Action workflow runs experienced a delay of more than five minutes. In addition, Pull Requests had delays in determining merge eligibility and starting associated Action workflows for the duration of the incident. Organization membership updates saw delays for upwards of five hours.
To prevent a similar event in the future from impacting our users, we are working to: improve our job management system to better manage our job worker capacity; add more precise monitoring for job delays; and strengthen our testing practices to prevent future recurrences.
Jun 28, 22:51 UTC
Update -
We are continuing to work to mitigate delays in organization membership changes.
Jun 28, 22:18 UTC
Update -
We are still actively working to mitigate delays in organization membership changes.
Jun 28, 21:45 UTC
Update -
We are actively working to mitigate delays in organization membership changes. Actions and Pull Requests are both functioning normally now.
Jun 28, 20:46 UTC
Update -
Actions is operating normally.
Jun 28, 20:00 UTC
Update -
Pull Requests is operating normally.
Jun 28, 19:59 UTC
Update -
We are continuing to apply mitigations and are seeing improvement in creating pull request merge commits and Actions runs for pull request events. Applying changes to organization members remains delayed.
Jun 28, 19:51 UTC
Update -
We are continuing to work on mitigating delays creating pull request merge commits, Actions runs for pull request events, and changes to organization members.
Jun 28, 19:03 UTC
Update -
Actions runs triggered by pull requests are experiencing start delays. We have engaged the appropriate teams and are investigating the issue.
Jun 28, 17:59 UTC
Update -
Pull Requests is experiencing degraded performance. We are continuing to investigate.
Jun 28, 17:58 UTC
Investigating -
We are investigating reports of degraded performance for Actions
Jun 28, 17:34 UTC
