PyPI Cache Invalidation Delays

Incident Report for Python Infrastructure

Resolved

This incident has been resolved.

Posted Nov 11, 2019 - 22:21 UTC

Monitoring

Our fix has been deployed and we are monitoring the results. Initial metrics show that the poorly performing task is now completing consistently within 1-2s rather than up to 100s.

Posted Nov 11, 2019 - 20:52 UTC

Update

Worker queues have drained and tasks should be running in a reasonable amount of time. We have identified the issue and are working to deploy and verify a fix.

Posted Nov 11, 2019 - 20:45 UTC

Identified

We have identified a specific asynchronous task for purging a cache internal to PyPI which is performing poorly. This is impacting the rate at which all jobs are processed.

For the moment, we have provisioned additional worker capacity to catch up while we work to investigate and resolve the issue.

Posted Nov 11, 2019 - 19:59 UTC

Investigating

Asynchronous jobs on PyPI such as email sending and cache invalidation are currently delayed.

Posted Nov 11, 2019 - 19:55 UTC

This incident affected: PyPI (pypi.org - CDN).