PyPI Database Upgrade

Scheduled Maintenance Report for Python Infrastructure

Postmortem

While successful and without error, this database maintenance took a little over 200% of the time we expected and communicated.

In analyzing the performance of the dump/restore process necessary for upgrading major versions of our database, we noted that one table in particular (our largest) took nearly four times longer than in initial testing of the procedure while all other tables performed as observed in tests.

We identified two differences between our trial runs and the actual final migration:

PyPI’s codebase being in read-only mode
Upgrades to the database client utilities

We have nothing to indicate that changing our codebase to stop database writes would have impacted performance, and are reasonably confident that using different versions of the dump/restore utilities may have impacted our estimates.

We will consider other methods for this kind of maintenance in the future and apologize for the extended duration of this maintenance. After our previous major database upgrade in December 2021, we are satisfied with the minimized impact on installers in this maintenance window but have a goal of reducing the duration that PyPI needs to be in read-only in the future.

Posted Oct 28, 2022 - 17:37 UTC

Completed

All systems are online and we are seeing successful operations and good performance from the backends. This maintenance is complete.

Posted Oct 28, 2022 - 16:12 UTC

Verifying

Our validation steps have completed and we are currently rolling out a deployment to move to the new database server. This will slowly bring PyPI out of read-only mode. We are monitoring to ensure systems are online and stable.

Posted Oct 28, 2022 - 16:06 UTC

Update

The database restore has finished. Indexes are being built and constraints applied which will take 10-15 minutes. After that we will begin validating consistency, updating database planner statistics, and preparing to switch to the newly upgraded database.

Posted Oct 28, 2022 - 15:49 UTC

Update

The largest portion of the data migration has completed and we are working on estimating the remaining time before we can begin validation and final steps to bring PyPI out of read-only mode.

Posted Oct 28, 2022 - 15:21 UTC

Update

Based on current estimate for completion we have extended this maintenance window by an additional hour. The data migration is progressing and we hope that after the current table completes the remainder will proceed at the expected rate.

Posted Oct 28, 2022 - 14:46 UTC

Update

We are continuing to monitor the progress of the database migration and will update after its next milestone.

Posted Oct 28, 2022 - 14:05 UTC

Update

Based on current estimate for completion we have extended this maintenance window by one hour.

Posted Oct 28, 2022 - 13:28 UTC

Update

The data migration is still in progress, but progressing at a slower rate than expected based on testing. We will have an update on ETA soon but expect to use the entire maintenance window at this time.

Posted Oct 28, 2022 - 13:09 UTC

Update

PyPI has been placed in read-only mode and the database upgrade is now in process. We expect this phase to take 60-90 minutes.

Posted Oct 28, 2022 - 12:02 UTC

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.

Posted Oct 28, 2022 - 12:00 UTC

Scheduled

PyPI's databases will be undergoing a major version upgrade during this period. At the beginning of the maintenance window, PyPI will be set to a read only mode to ensure database consistency as data is migrated to a new database instance running the upgraded version.

We anticipate that PyPI will remain in read only mode for approximately 60-90 minutes. During this time, uploads will not be allowed and changes that would result in any database write will not succeed via the web UI. All client APIs (simple, JSON, XMLRPC) will remain online and fully available during the maintenance.

Posted Oct 26, 2022 - 20:05 UTC

This scheduled maintenance affected: PyPI (pypi.org - Backends).