PyPI: Object Storage Migration
Scheduled Maintenance Report for Python Infrastructure
Completed
We have completed this maintenance! As far as we are aware no user impact was reported at any phase aside from the temporary upload suspension. If you do experience issues with files.pythonhosted.org in the coming weeks that may be related, you can report to https://github.com/pypi/warehouse/issues/13403.
Posted Apr 12, 2023 - 13:59 UTC
Verifying
Our migration has completed. We are closely monitoring and completing some final tasks before closing this maintenance notice.
Posted Apr 12, 2023 - 11:40 UTC
Update
The second phase of our migration is nearing completion. Our new buckets are finalizing being verified against the old. We're seeing only a handful of fallback requests, which indicates that our new provider performing well and serving its files dutifully.

Next phase will be ensuring our new archival buckets are fully validated as well, then finally we will make our fallback point to the archival buckets rather than the previous generation.

We anticipate that these changes will be made over the next 24-48 hours and should have no impact on services for files.pythonhosted.org or test-files.pythonhosted.org.
Posted Apr 11, 2023 - 19:15 UTC
Update
Uploads are fully re-enabled. We have validated our current stop-point in the migration.

Next steps are to begin a monitor a backfill operation from our old primary bucket into our new primary bucket. We expect this to take 3-6 hours. No further impact is expected.

We will continue to update here as needed.
Posted Apr 11, 2023 - 12:02 UTC
Update
We have completed the momentary switchover. We will now assess to ensure everything is stable before moving to finalize the migration.
Posted Apr 11, 2023 - 11:55 UTC
Update
We will have one additional 1-2 minute window where uploads are disabled.
Posted Apr 11, 2023 - 11:47 UTC
Update
All uploads will temporarily be suspended for the momentary switchover.
Posted Apr 11, 2023 - 11:09 UTC
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Apr 11, 2023 - 09:31 UTC
Scheduled
The object storage for PyPI (pypi.org) and Test PyPI (test.pypi.org) are being migrated to new redundant storage. We anticipate that this maintenance will about two days to fully complete, but have worked to ensure that it should have minimal, if any perceptible impact on users.

During the migration, files not found in our new bucket will be fetched from the bucket we are migrating away from. This is the only *expected* impact and may incur additional latency for misses. We will be monitoring for this latency.

While we do live in the blessed days of error code correction ram being the status quo, we cannot rule out that some objects may be inconsistent. The tools used for migrating objects perform checksumming, but we will be doing a full reconciliation sweep after the migration has completed to ensure that all files currently known to the databases were stored with the correct checksum.

If you experience any unexpected behavior with file downloads from files.pythonhosted.org during this migration, please file an issue at https://github.com/pypi/warehouse/issues/13403.
Posted Apr 11, 2023 - 09:26 UTC
This scheduled maintenance affected: PyPI (files.pythonhosted.org - Files).