This incident has been resolved.

Our fix has been deployed and we are monitoring the results. Initial metrics show that the poorly performing task is now completing consistently within 1-2s rather than up to 100s.

Worker queues have drained and tasks should be running in a reasonable amount of time. We have identified the issue and are working to deploy and verify a fix.

We have identified a specific asynchronous task for purging a cache internal to PyPI which is performing poorly. This is impacting the rate at which all jobs are processed.

For the moment, we have provisioned additional worker capacity to catch up while we work to investigate and resolve the issue.

Asynchronous jobs on PyPI such as email sending and cache invalidation are currently delayed.