Our fix has been deployed and we are monitoring the results. Initial metrics show that the poorly performing task is now completing consistently within 1-2s rather than up to 100s.
Posted Nov 11, 2019 - 20:52 UTC
Update
Worker queues have drained and tasks should be running in a reasonable amount of time. We have identified the issue and are working to deploy and verify a fix.
Posted Nov 11, 2019 - 20:45 UTC
Identified
We have identified a specific asynchronous task for purging a cache internal to PyPI which is performing poorly. This is impacting the rate at which all jobs are processed.
For the moment, we have provisioned additional worker capacity to catch up while we work to investigate and resolve the issue.
Posted Nov 11, 2019 - 19:59 UTC
Investigating
Asynchronous jobs on PyPI such as email sending and cache invalidation are currently delayed.