Python Infrastructure
Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Jan 12, 16:04 UTC
Update - We are continuing to monitor for any further issues.
Dec 28, 13:51 UTC
Update - The XMLRPC Search endpoint remains disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again.
Dec 28, 13:50 UTC
Update - The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Dec 23, 14:54 UTC
Update - The XMLRPC Search endpoint is still disabled due to ongoing request volume. As of this update, there has been no reduction in inbound traffic to the endpoint from abusive IPs and we are unable to re-enable the endpoint, as it would immediately cause PyPI service to degrade again. We are working with the abuse contact at the owner of the IPs and trying to make contact with the maintainers of whatever tool is flooding us via other channels.
Dec 15, 20:59 UTC
Monitoring - With the temporary disabling of XMLRPC we are hoping that the mass consumer that is causing us trouble will make contact. Due to the huge swath of IPs we were unable to make a more targeted block without risking more severe disruption, and were not able to receive a response from their abuse contact or direct outreach in an actionable time frame.
Dec 14, 17:46 UTC
Update - Due to the overwhelming surges of inbound XMLRPC search requests (and growing) we will be temporarily disabling the XMLRPC search endpoint until further notice.
Dec 14, 17:30 UTC
Identified - We've identified that the issue is with excess volume to our XLMRPC search endpoint that powers `pip search` among other tools. We are working to try to identify patterns and prohibit abusive clients to retain service health.
Dec 14, 15:09 UTC
Investigating - PyPI's search backends are experiencing an outage causing the backends to timeout and fail, leading to degradation of service for the web app. Uploads and installs are currently unaffected but logged in actions and search via the web app and API access via XMLRPC are currently experiencing partial outages.
Dec 14, 09:41 UTC
python.org ? Operational
python.org - CDN ? Operational
python.org - Backends ? Operational
PyPI Partial Outage
pypi.org - CDN ? Operational
pypi.org - Backends ? Partial Outage
files.pythonhosted.org - Files ? Operational
files.pythonhosted.org - Redirects ? Operational
files.pythonhosted.org - Redirects Backends ? Operational
PyPI Hosting Platforms Operational
AWS elasticache-us-east-2 Operational
AWS elb-us-east-2 Operational
AWS ec2-us-east-2 Operational
AWS rds-us-east-2 Operational
Google Cloud Platform Google Cloud Storage Operational
docs.python.org Operational
docs.python.org - Backends ? Operational
docs.python.org - CDN Operational
bugs.python.org Operational
wiki.python.org Operational
mail.python.org Operational
Message Handling Services Operational
Mailing Lists and Archives - Mailman Operational
Mailing Lists and Archives - Mailman 3 Operational
psfmember.org Operational
us.pycon.org Operational
PyPy Operational
pypy.org Operational
speed.pypy.org Operational
Content Delivery Network ? Operational
Fastly US East (JFK) Operational
Fastly Asia/Pacific (HK) Operational
Fastly US East (IAD) Operational
Fastly US East (ATL) Operational
Fastly US East (MIA) Operational
Fastly US Central (DEN) Operational
Fastly US Central (DFW) Operational
Fastly US Central (ORD) Operational
Fastly US West (LAX) Operational
Fastly US West (SEA) Operational
Fastly US West (SJC) Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Fastly Europe (LCY) Operational
Fastly Europe (LHR) Operational
Fastly Asia/Pacific (SYD) Operational
Fastly Asia/Pacific (SIN) Operational
Fastly Asia/Pacific (NZ) Operational
Fastly Brisbane (BNE) Operational
Fastly Dubai (FJR) Operational
Fastly Melbourne (MEL) Operational
Fastly Osaka (ITM) Operational
Fastly Perth (PER) Operational
Fastly Singapore (QPG) Operational
Fastly Singapore (XSP) Operational
Fastly Tokyo (HND) Operational
Fastly Tokyo (TYO) Operational
Fastly Wellington (WLG) Operational
Fastly Dublin (DUB) Operational
Fastly Copenhagen (CPH) Operational
Fastly Frankfurt (HHN) Operational
Fastly Helsinki (HEL) Operational
Fastly London (LON) Operational
Fastly Madrid (MAD) Operational
Fastly Manchester (MAN) Operational
Fastly Milan (MXP) Operational
Fastly Oslo (OSL) Operational
Fastly Buenos Aires (EZE) Operational
Fastly Bogota (BOG) Operational
Fastly Curitiba (CWB) Operational
Fastly Rio de Janeiro (GIG) Operational
Fastly Santiago (SCL) Operational
Fastly Sāo Paulo (GRU) Operational
Fastly Johannesburg (JNB) Operational
Fastly Cape Town (CPT) Operational
Fastly Vancouver (YVR) Operational
Fastly Toronto (YYZ) Operational
Fastly St. Louis (STL) Operational
Fastly Palo Alto (PAO) Operational
Fastly Newark (EWR) Operational
Fastly New York (LGA) Operational
Fastly Montreal (YUL) Operational
Fastly Minneapolis (STP) Operational
Fastly Minneapolis (MSP) Operational
Fastly Los Angeles (BUR) Operational
Fastly Kansas City (MCI) Operational
Fastly Jacksonville (JAX) Operational
Fastly Houston (IAH) Operational
Fastly Dallas (DAL) Operational
Fastly Columbus (CMH) Operational
Fastly Chicago (PWK) Operational
Fastly Chicago (MDW) Operational
Fastly Chicago (CHI) Operational
Fastly Boston (BOS) Operational
Fastly Atlanta (PDK) Operational
Fastly Atlanta (FTY) Operational
Fastly Ashburn (WDC) Operational
Fastly Ashburn (DCA) Operational
Fastly Ashburn (BWI) Operational
Fastly New Delhi (DEL) Operational
Fastly Mumbai (BOM) Operational
Fastly Chennai (MAA) Operational
Fastly Vienna (VIE) Operational
Fastly Stockholm (BMA) Operational
Fastly Paris (CDG) Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
PyPI CDN Edge Errors ?
Fetching
PyPI Files CDN Edge Errors ?
Fetching
PyPI CDN Miss Times
Fetching
PyPI Files CDN Miss Times
Fetching
Past Incidents
Jan 22, 2021

No incidents reported today.

Jan 21, 2021

No incidents reported.

Jan 20, 2021

No incidents reported.

Jan 19, 2021

No incidents reported.

Jan 18, 2021

No incidents reported.

Jan 17, 2021

No incidents reported.

Jan 16, 2021

No incidents reported.

Jan 15, 2021
Resolved - This incident has been resolved.
Jan 15, 14:31 UTC
Monitoring - We have restored the service and are monitoring for problematic traffic.
Jan 15, 11:09 UTC
Identified - wiki.python.org has been shut off temporarily in the face of a distributed denial of service attack. thousands of IP addresses are flooding the host with valid but garbage requests and attempts to block them are not keeping up with the influx.

the host will be offline overnight until we are able to assess and address this issue.
Jan 15, 02:53 UTC
Jan 14, 2021

No incidents reported.

Jan 13, 2021

No incidents reported.

Jan 12, 2021
Resolved - This incident has been resolved.
Jan 12, 16:52 UTC
Monitoring - Our CDN provider is experiencing a global instability issue causing a mixture of errors for users of many Python Infrastructure services. We are monitoring their status and awaiting resolution.
Jan 12, 16:06 UTC
Jan 11, 2021

No incidents reported.

Jan 10, 2021

No incidents reported.

Jan 9, 2021

No incidents reported.

Jan 8, 2021

No incidents reported.