Spam on PyPI

Incident Report for Python Infrastructure

Resolved

While we can't say with certainty that they won't return, the spammers have at least shifted their focus away from PyPI at this time. We're going to continue working on future mitigation tools and procedures and will update if necessary at a later date.
Posted 7 years ago. Feb 22, 2018 - 22:13 UTC

Update

There have been a few attempts at submitting new spam to PyPI over the last 36 hours. We are monitoring and using some rapid-response tooling to remove spam as it arrives. This seems to be reducing the frequency of attempts.

Development has begun on our longer term strategies and we'll update here as those progress through the week.
Posted 7 years ago. Feb 20, 2018 - 16:27 UTC

Update

No new spam has arrived on PyPI since reenabling new Project registration. We are monitoring new project registration closely and planning our next steps. Based on information from the team behind https://www.npmjs.com, these spammers are rather ruthless and we expect them to return.

Here's a quick summary of what's been done so far:

- Require at least one verified email address per user in order to register new projects
- Admin feature flags to allow for quickly disabling new project registration and new user registration
- Admin features for spam User and Project cleanup
- Disabled User registration on https://pypi.python.org and direct users to register at https://pypi.org

We're working on automated spam classification of Projects and Releases along with Admin features for us to train the model taking into account User reports of spam as well.
Posted 7 years ago. Feb 19, 2018 - 13:12 UTC

Monitoring

We've completed initial cleanup and are preparing for next phase of features related to automated classification, community reporting, and admin verification of spam.

New Project registrations are re-enabled for the time being to determine the efficacy of our existing prevention methods.
Posted 7 years ago. Feb 19, 2018 - 05:08 UTC

Update

We've begun cleanup of spam from the first volley. Once that is complete, we'll assess and consider re-enabling new Project registration. Thanks for your patience as we work through this unfortunate situation.
Posted 7 years ago. Feb 19, 2018 - 04:02 UTC

Update

The PyPI Administrators have decided to temporarily disable new Project registration on PyPI.

We understand that this will be disruptive for some users, but have decided to take this action to allow for two things:
- A brief rest for the PyPI administrators
- Perhaps convince the spammers to leave PyPI alone

See https://pypi.org/help/#admin-intervention for more details.

Note: Existing Projects will still be able to upload new releases.

We have a few more mitigations in the works, but have decided that protecting our contributors from burnout is of higher importance than resolving this issue in a haphazard or partial manner.
Posted 7 years ago. Feb 18, 2018 - 18:24 UTC

Update

We've taken additional action to dissuade spammers by disabling registration via https://pypi.python.org and forcing user registration to take place via https://pypi.org (the new codebase) which has a CAPTCHA.

We'll begin disabling attacker accounts after this change goes live to see if they return.
Posted 7 years ago. Feb 18, 2018 - 18:01 UTC

Update

We have shipped the first of a handful of spam mitigations, PyPI users are now required to have a verified email address in order to register new projects.

See https://pypi.org/help/#verified-email for details on the change and how you can retroactively verify your email address.

We're still waiting to determine the efficacy of our spam prevention techniques before undertaking cleanup, as it is costly in time for the administrators of PyPI and best done in large batches rather than repeating the process multiple times.
Posted 7 years ago. Feb 18, 2018 - 15:05 UTC

Update

First pass of our initial spam prevention measures are feature complete, but require review. We'll still be waiting to clean up the spam until after they ship and we're able to reduce the rate of incoming spam, which should occur tomorrow.
Posted 7 years ago. Feb 18, 2018 - 01:44 UTC

Identified

Two spam prevention measures are nearly ready for review and deploy. After those are out, we'll begin analysis and cleanup of today's volley of Spam.
Posted 7 years ago. Feb 18, 2018 - 00:25 UTC

Investigating

PyPI is currently being spammed in a relatively well designed way. We are working to put measures in place to stop new spam before beginning cleanup efforts.
Posted 7 years ago. Feb 17, 2018 - 19:22 UTC