PyPI mailing errors
Incident Report for Python Infrastructure
Postmortem

The outage affecting email delivery from the PyPI infrastructure for password reset and registration emails has been resolved.

Root Cause:

  • PyPI was sending email from cheeseshop@python.org with reply-to cheeseshop@python.org while signing with the domain key for pypi-mail.python.org
  • cheesehop@python.org was not a valid reply-to address for the signed domain.
  • Domain validation settings which not fully configured.

Effect:

  • A significant portion of recipient mail servers rejected PyPI emails.

Resolution:

  • A domain was configured specifically for PyPI email (mail.pypi.python.org) with proper domain validation settings.
  • PyPI was configured to send from an address within the configured domain (PyPI Admin <admin@mail.pypi.python.org>)
  • A valid reply-to address (admin@mail.pypi.python.org) was configured which will be accessible to the PyPI administration team.

Validation:

  • All unsent emails were resent via the new configuration.
  • Delivery was verified with the logging available from the delivery service.
Posted Jan 27, 2014 - 06:20 UTC

Resolved
We have verified the new configuration and resent all failed messages.
Posted Jan 27, 2014 - 05:29 UTC
Update
Monitoring of our new mail configuration will continue through January 27th.

We have not seen any issues yet, but also have not seen enough volume or variety in delivery domains to call this resolved.
Posted Jan 27, 2014 - 04:26 UTC
Monitoring
We've updated our mail configuration and are monitoring for validation and delivery issues.
Posted Jan 27, 2014 - 01:13 UTC
Identified
We've traced the validation errors back to bad configuration in our new mail setup and are working to resolve it now.
Posted Jan 27, 2014 - 00:33 UTC
Investigating
PyPI emails are experiencing a significant volume of delivery errors.
Posted Jan 26, 2014 - 22:36 UTC