PSF Infra Incident - 2017-10-20

Beginning at 2017-10-20 15:55 UTC and lasting until 2017-10-20 18:08 UTC the following PSF Hosted services experienced increased latency and elevated error rates:

• www.python.org
• docs.python.org

The incident began when a Purge All was performed in the CDN for www.python.org. This operation invalidated the cache for all pages on the domain and caused a rush of requests from our CDN's edge nodes as they began rebuilding their caches.

After the Purge All, the influx of requests had the following impacts:

• Overloading the server which hosts release files
  • Many large files related to Python source code and releases are hosted on www.python.org, specifically under the /ftp prefix.
• Saturating the network to our load balancer tier
  • Our load balancer servers are maxed out on the provisioned bandwidth available, at nearly 2x of our normal peak bandwidth.

This directly impacted requests for download files and caused degraded service for requests which traverse the load balancer tier.

Caches were slow to refill leading to a compounding effect when the release file hosting server began to fail due to the overload.

Once the CDN's caches has rebuilt services returned to their previous performance.

Mitigation

During the incident the following actions were taken by the PSF Infrastructure Team:

• Improve tuning for the release hosting server
• Implement cache "shielding" within our CDN to reduce the traffic needed to our backend load balancers

Future Work

The primary take aways here:

• Educate users with the appropriate privileges on the risks and consequences of Purge All for our high traffic services.
• Ensure shielding is enabled for remaining high traffic services to reduce the time to recovery if this occurs in the future.