OpenAI System Design: Why They Ran Python at 20 Million Requests/Sec

OpenAI’s online storage platform, Habitat, handles over 70 million requests per second across 40 geographic regions and 500 petabytes of data to power ChatGPT, the Responses API, and Codex. While tech headlines focused on OpenAI rewriting its core data service from Python to Rust using just two engineers and Codex, the real architectural masterclass is how OpenAI deliberately ran Python at 20 million requests per second at peak while scaling an unsharded, single-primary PostgreSQL database to 800 million users. 🔔 Subscribe: https://www.youtube.com/channel/UC0DZj1PNa_Fp0MU6uPSKv5w?sub_confirmation=1 💙 Become a Member: https://www.youtube.com/channel/UC0DZj1PNa_Fp0MU6uPSKv5w/join 🐦 Twitter/X: https://x.com/cloud_codes 💬 Discord: https://discord.gg/mhgXXmhrh In this deep dive, Cloud Codes deconstructs OpenAI’s internal distributed systems architecture: how they defended a single Postgres primary using nearly 50 read replicas, MVCC write-amplification mitigations, PgBouncer pooling (dropping connection latency from 50ms to 5ms), and a strict 5-second schema timeout. We trace why Habitat evolved from an unversioned client library into an independent service, examine the Python runtime limits—including asyncio event-loop scheduling delays on fast storage returns, Statsig 60-second synchronous config parsing lockups, and aiohttp LIFO connection-pool metastable failure loops—and unpack why OpenAI adopted Facebook TAO’s NoSQL object-and-edge data model to physically outlaw expensive SQL joins before executing a 6x CPU / 15x memory Rust rewrite in a single quarter. If this breakdown helped you understand high-scale system design, distributed database architecture, and production engineering tradeoffs, subscribe to Cloud Codes for a new deep dive every week. Build, solve, deploy. If you learned something new here, subscribe for your daily dose. 🔗 Resources Mentioned: • OpenAI Engineering: Rapidly Scaling Online Storage (Habitat Part 1): https://openai.com/index/scaling-storage-one-billion-users-part-one/ • OpenAI Engineering: Scaling PostgreSQL to Power 800M Users: https://openai.com/index/scaling-postgresql/ • OpenAI Developers Official Habitat Architecture Post (@OpenAIDevs): https://x.com/OpenAIDevs/status/2098502006935814272 • TAO: Facebook's Distributed Data Store for the Social Graph (Bronson et al.): https://www.usenix.org/conference/atc13/technical-sessions/presentation/bronson • Metastable Failures in Distributed Systems (HotOS '21 Paper): https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf • OpenAI Acquisition of Rockset (Analytical Retrieval Engine): https://openai.com/index/openai-acquires-rockset/ • OpenAI Careers: Software Engineer, Habitat Database Platform: https://openai.com/careers/ ⏱️ Chapters: 0:00 - The Hidden Layer Between You and ChatGPT 0:59 - Nov 2022 to DevDay 2023: The Birth of Habitat 2:39 - Wall 1: How OpenAI Scaled an Unsharded Postgres Primary 4:54 - The 12-Table Join, PgBouncer, & The 5-Second Rule 6:28 - Wall 2: Why Client Libraries Cause Outages 8:18 - The Python Wager: Deliberate Technical Debt 9:35 - Wall 3: The GIL Tax & Measuring asyncio Delays 10:49 - The $1.1B Config Stall: Statsig Fleet Lockups 12:05 - Wall 4: The aiohttp LIFO Connection Pool Trap 14:26 - The Real Fix: Facebook TAO Objects & Edges API 16:57 - The Rust Rewrite: 2 Engineers, Codex, & GPT-5.5 18:49 - Final Verdict: 4 Lessons from OpenAI's System Design #openai #systemdesign #softwareengineering #rust #python #postgresql #distributeddatabase #cloudcodes #devops #backend User Queries: openai system design explained openai quietly deleted its python backend openai habitat storage platform architecture how chatgpt scales postgresql to 800m users openai python to rust rewrite codex openai 70 million requests per second asyncio event loop scheduling delay tail latency statsig feature flag stall openai python aiohttp lifo connection pool bug metastable failure facebook tao object and edge api openai why openai did not shard postgres pgbouncer connection pooling openai openai habitat vs cosmos db nathan bronson openai tao paper openai 12 table join outage deliberate technical debt openai python how openai uses rust in production openai 5 second schema migration timeout chatgpt imagegen viral traffic sev 0 outage cloud codes