PyPI XMLRPC Search Disabled

Incident Report for Python Infrastructure

Resolved

XMLRPC Search has been permanently disabled.
Posted 3 years ago. Jan 03, 2022 - 20:32 UTC

Update

We are now at 100 days since the decision to disable the XMLRPC search endpoint.

Traffic to the endpoint has not subsided in any substantial way and we have not heard from any of the parties who continue to issue hundreds of thousands of search calls per hour.

As such, XMLRPC search will be permanently disabled.
Posted 4 years ago. Mar 24, 2021 - 14:22 UTC

Update

The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Posted 4 years ago. Jan 12, 2021 - 16:04 UTC

Update

We are continuing to monitor for any further issues.
Posted 4 years ago. Dec 28, 2020 - 13:51 UTC

Update

The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Posted 4 years ago. Dec 28, 2020 - 13:50 UTC

Update

The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Posted 4 years ago. Dec 23, 2020 - 14:54 UTC

Update

The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Posted 4 years ago. Dec 15, 2020 - 20:59 UTC

Monitoring

With the temporary disabling of XMLRPC we are hoping that the mass consumer that is causing us trouble will make contact. Due to the huge swath of IPs we were unable to make a more targeted block without risking more severe disruption, and were not able to receive a response from their abuse contact or direct outreach in an actionable time frame.
Posted 4 years ago. Dec 14, 2020 - 17:46 UTC

Update

Due to the overwhelming surges of inbound XMLRPC search requests (and growing) we will be temporarily disabling the XMLRPC search endpoint until further notice.
Posted 4 years ago. Dec 14, 2020 - 17:30 UTC

Identified

We've identified that the issue is with excess volume to our XLMRPC search endpoint that powers `pip search` among other tools. We are working to try to identify patterns and prohibit abusive clients to retain service health.
Posted 4 years ago. Dec 14, 2020 - 15:09 UTC

Investigating

PyPI's search backends are experiencing an outage causing the backends to timeout and fail, leading to degradation of service for the web app. Uploads and installs are currently unaffected but logged in actions and search via the web app and API access via XMLRPC are currently experiencing partial outages.
Posted 4 years ago. Dec 14, 2020 - 09:41 UTC
This incident affected: PyPI (pypi.org - CDN, pypi.org - Backends).