python.org unavailable

Incident Report for Python Infrastructure

Resolved

This incident has been resolved.

We removed two deprecated service backends recently but failed to update the configuration of the inbound HTTP load balancing infrastructure. As a result, when the services finally aged out of internal service discovery the templates that are rendered to configure the HTTP load balancer were not valid. This caused the load balancers to fail and our backends became unavailable.

Additionally caching caused our monitoring to fail to detect this outage until about 20 minutes into the incident.

We are working to resolve the template rendering issue and improve the time to alert for outages of python.org.

Posted Aug 10, 2018 - 18:19 UTC

Monitoring

We identified the issue and have put a fix in place. We're monitoring to ensure things are stable.

Posted Aug 10, 2018 - 17:55 UTC

Investigating

We are investigating an outage of https://python.org

Posted Aug 10, 2018 - 17:45 UTC

This incident affected: python.org (python.org - CDN).