Python Infrastructure
Update - We are now at 100 days since the decision to disable the XMLRPC search endpoint.

Traffic to the endpoint has not subsided in any substantial way and we have not heard from any of the parties who continue to issue hundreds of thousands of search calls per hour.

As such, XMLRPC search will be permanently disabled.
Mar 24, 14:22 UTC
Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Jan 12, 16:04 UTC
Update - We are continuing to monitor for any further issues.
Dec 28, 13:51 UTC
Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Dec 28, 13:50 UTC
Update - The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Dec 23, 14:54 UTC
Update - The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Dec 15, 20:59 UTC
Monitoring - With the temporary disabling of XMLRPC we are hoping that the mass consumer that is causing us trouble will make contact. Due to the huge swath of IPs we were unable to make a more targeted block without risking more severe disruption, and were not able to receive a response from their abuse contact or direct outreach in an actionable time frame.
Dec 14, 17:46 UTC
Update - Due to the overwhelming surges of inbound XMLRPC search requests (and growing) we will be temporarily disabling the XMLRPC search endpoint until further notice.
Dec 14, 17:30 UTC
Identified - We've identified that the issue is with excess volume to our XLMRPC search endpoint that powers `pip search` among other tools. We are working to try to identify patterns and prohibit abusive clients to retain service health.
Dec 14, 15:09 UTC
Investigating - PyPI's search backends are experiencing an outage causing the backends to timeout and fail, leading to degradation of service for the web app. Uploads and installs are currently unaffected but logged in actions and search via the web app and API access via XMLRPC are currently experiencing partial outages.
Dec 14, 09:41 UTC
python.org ? Operational
python.org - CDN ? Operational
python.org - Backends ? Operational
PyPI Operational
pypi.org - CDN ? Operational
pypi.org - Backends ? Operational
files.pythonhosted.org - Files ? Operational
files.pythonhosted.org - Redirects ? Operational
files.pythonhosted.org - Redirects Backends ? Operational
PyPI Hosting Platforms Operational
AWS elasticache-us-east-2 Operational
AWS elb-us-east-2 Operational
AWS ec2-us-east-2 Operational
AWS rds-us-east-2 Operational
Google Cloud Platform Google Cloud Storage Operational
docs.python.org Operational
docs.python.org - Backends ? Operational
docs.python.org - CDN Operational
bugs.python.org Operational
wiki.python.org Operational
mail.python.org Operational
Message Handling Services Operational
Mailing Lists and Archives - Mailman Operational
Mailing Lists and Archives - Mailman 3 Operational
psfmember.org Operational
us.pycon.org Operational
PyPy Operational
pypy.org Operational
speed.pypy.org Operational
Content Delivery Network ? Operational
Fastly US East (JFK) Operational
Fastly Asia/Pacific (HK) Operational
Fastly US East (IAD) Operational
Fastly US East (ATL) Operational
Fastly US East (MIA) Operational
Fastly US Central (DEN) Operational
Fastly US Central (DFW) Operational
Fastly US Central (ORD) Operational
Fastly US West (LAX) Operational
Fastly US West (SEA) Operational
Fastly US West (SJC) Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Fastly Europe (LCY) Operational
Fastly Europe (LHR) Operational
Fastly Asia/Pacific (SYD) Operational
Fastly Asia/Pacific (SIN) Operational
Fastly Asia/Pacific (NZ) Operational
Fastly Brisbane (BNE) Operational
Fastly Dubai (FJR) Operational
Fastly Melbourne (MEL) Operational
Fastly Osaka (ITM) Operational
Fastly Perth (PER) Operational
Fastly Singapore (QPG) Operational
Fastly Singapore (XSP) Operational
Fastly Tokyo (HND) Operational
Fastly Tokyo (TYO) Operational
Fastly Wellington (WLG) Operational
Fastly Dublin (DUB) Operational
Fastly Copenhagen (CPH) Operational
Fastly Frankfurt (HHN) Operational
Fastly Helsinki (HEL) Operational
Fastly London (LON) Operational
Fastly Madrid (MAD) Operational
Fastly Manchester (MAN) Operational
Fastly Milan (MXP) Operational
Fastly Oslo (OSL) Operational
Fastly Buenos Aires (EZE) Operational
Fastly Bogota (BOG) Operational
Fastly Curitiba (CWB) Operational
Fastly Rio de Janeiro (GIG) Operational
Fastly Santiago (SCL) Operational
Fastly Sāo Paulo (GRU) Operational
Fastly Johannesburg (JNB) Operational
Fastly Cape Town (CPT) Operational
Fastly Vancouver (YVR) Operational
Fastly Toronto (YYZ) Operational
Fastly St. Louis (STL) Operational
Fastly Palo Alto (PAO) Operational
Fastly Newark (EWR) Operational
Fastly New York (LGA) Operational
Fastly Montreal (YUL) Operational
Fastly Minneapolis (STP) Operational
Fastly Minneapolis (MSP) Operational
Fastly Los Angeles (BUR) Operational
Fastly Kansas City (MCI) Operational
Fastly Jacksonville (JAX) Operational
Fastly Houston (IAH) Operational
Fastly Dallas (DAL) Operational
Fastly Columbus (CMH) Operational
Fastly Chicago (PWK) Operational
Fastly Chicago (MDW) Operational
Fastly Chicago (CHI) Operational
Fastly Boston (BOS) Operational
Fastly Atlanta (PDK) Operational
Fastly Atlanta (FTY) Operational
Fastly Ashburn (WDC) Operational
Fastly Ashburn (DCA) Operational
Fastly Ashburn (BWI) Operational
Fastly New Delhi (DEL) Operational
Fastly Mumbai (BOM) Operational
Fastly Chennai (MAA) Operational
Fastly Vienna (VIE) Operational
Fastly Stockholm (BMA) Operational
Fastly Paris (CDG) Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
PyPI CDN Edge Errors ?
Fetching
PyPI Files CDN Edge Errors ?
Fetching
PyPI CDN Miss Times
Fetching
PyPI Files CDN Miss Times
Fetching
Past Incidents
Aug 3, 2021

No incidents reported today.

Aug 2, 2021

No incidents reported.

Aug 1, 2021

No incidents reported.

Jul 31, 2021

No incidents reported.

Jul 30, 2021

No incidents reported.

Jul 29, 2021

No incidents reported.

Jul 28, 2021
Resolved - This incident has been resolved.
Jul 28, 16:42 UTC
Monitoring - Services have been restored and our provider has indicated that network connectivity should be restored to the affected region. We are continuing to monitor to ensure stability.
Jul 28, 16:28 UTC
Update - We are unable to access any hosts in the region and are currently unable to reach the control plane as well. We will be watching for updates from the provider.
Jul 28, 16:09 UTC
Investigating - We are currently investigating a major outage impacting all services hosted out of one of our providers regions.
Jul 28, 16:08 UTC
Jul 27, 2021

No incidents reported.

Jul 26, 2021

No incidents reported.

Jul 25, 2021

No incidents reported.

Jul 24, 2021

No incidents reported.

Jul 23, 2021

No incidents reported.

Jul 22, 2021

No incidents reported.

Jul 21, 2021

No incidents reported.

Jul 20, 2021

No incidents reported.