Incident Report for Python Infrastructure

This outage was caused when our systems automatically updated to the latest version of the configuration management tooling that orchestrates the PSF infrastructure.

Due to a misconfiguration, the automated update jumped a major version which broke the tooling that issues internal TLS certificates that secure traffic within our [network.’s backend host as well as load balancers were the first to experience an expired certificate.

We have updated our configuration to pin to a major version of the tooling, updated the monitoring and alerting for docs to automatically update the status on our status page, and will investigate ways to monitor for internal certificates nearing expiration.

Posted Feb 26, 2019 - 02:18 UTC

This incident has been resolved.
Posted Feb 26, 2019 - 02:14 UTC
We are currently investigating this issue.
Posted Feb 26, 2019 - 01:37 UTC
This incident affected: ( - Backends).