PyPI: Elevated Error Rates for /packages

Incident Report for Python Infrastructure

Resolved

Errors for /packages urls have subsided. During the course of responding to this incident, we determined that the mirror that is intended to act as a failover in situations like this had run out of disk space.

We have begun creating a new mirror with the necessary storage and verifying that failover can take place.

Additionally, we've added monitoring to alert when mirror space is low which should avoid surprises like this in the future.
Posted Dec 26, 2017 - 20:30 UTC

Monitoring

Errors have subsided, we're continuing to monitor and take other preventative measures.
Posted Dec 26, 2017 - 19:56 UTC

Identified

The object store that hosts PyPI's files is experiencing intermittent issues serving packages. The error rate is minimal, but we are continuing to work to resolve the issue.
Posted Dec 26, 2017 - 19:47 UTC

Investigating

Our automated systems detected an elevated error rate for `/packages` urls. We are investigating the issue and will update shortly.
Posted Dec 26, 2017 - 19:39 UTC