Hosting Provider Reboots

The infrastructure which runs PyPI was scheduled by our hosting provider for rolling reboots of all hosts beginning on 2014-09-28 at 1100 UTC and ending at 2014-09-29 at 1101 UTC.

Ultimately, due to the lack of specific timing from our provider the PSF Infrastructure team chose to follow the lowest risk path and take down the active backends ahead of these rolling reboots in order to ensure a trouble free return to 100% availability.

The PyPI Infrastructure is currently comprised of:

• 2x GlusterFS nodes which store and serve packages in a an active-active pair.
• 2x PostgreSQL Servers in a primary-hot-standby cluster.
• 2x Redis servers for download counts and caching.
• 3x Python Web servers
• 3x Geo-Distributed mirrors which offer the full /simple, /packages, and /serversig trees.
• Various administrative/backup servers.
• A stellar Global CDN.

The PSF Infrastructure team has tooling in place to easily switch from the active web backends which provide 100% of PyPI's functionality to the static mirror network, behind our CDN.

In static mode, all package installers using the /simple index are fully supported. This service is considered the #1 priority of PyPI.

Ultimately three factors contributed to our choice in going static for the lengthy scheduled maintenance window:

• Possible unforeseen complexities that could have arisen with an unmanaged reboot in the GlusterFS and PostgreSQL clusters, however unlikely.
• Static mirror network meets the #1 Priority of the Python Package Index, without needing active backends.
• The lead on the PyPI Infrastructure had been managing similar maintenance windows with other providers, for other services and would have been approaching 28 hours awake at the start of PyPI's scheduled maintenance.

We felt that leaving PyPI vulnerable to intermittent or difficult to recover from outages due to rolling reboots would have been regrettable. Instead, our choice was to go static.