We removed two deprecated service backends recently but failed to update the configuration of the inbound HTTP load balancing infrastructure. As a result, when the services finally aged out of internal service discovery the templates that are rendered to configure the HTTP load balancer were not valid. This caused the load balancers to fail and our backends became unavailable.
Additionally caching caused our monitoring to fail to detect this outage until about 20 minutes into the incident.
We are working to resolve the template rendering issue and improve the time to alert for outages of python.org.
Posted 9 months ago. Aug 10, 2018 - 18:19 UTC
We identified the issue and have put a fix in place. We're monitoring to ensure things are stable.