The Golden Rule: Measure First
Every optimization in this collection is worthless if you apply it to the wrong bottleneck. The #1
mistake under load is guessing. Adding 200 threads to a CPU-bound service makes it
slower. Rewriting to async when your DB is the limit changes nothing.
Always answer this first: is the system CPU-bound (maxing cores),
I/O-bound (waiting on network/disk/DB), or memory-bound (paging,
GC thrash, OOM)? The answer dictates the entire strategy — and it's cheap to find out with a profiler.
Know Your Bottleneck → Know Your Tool
| Bottleneck | Symptom under load | Primary weapon |
|---|
| I/O-bound | Low CPU, high wait, requests pile up | Async or threads + pooling, batching, concurrency caps |
| CPU-bound | All cores at 100%, GIL contention | multiprocessing / C extensions / chunking |
| Memory-bound | RSS climbs, GC pauses, OOM kills | Streaming, generators, bounded queues, __slots__ |
| Downstream-bound | A slow dependency (DB, API) saturates | Rate limiting, circuit breakers, caching, backpressure |
The three models, in one line each: Threads — overlap blocking I/O
with existing sync libraries.
Async — overlap tens of thousands of I/O tasks cheaply
on one core.
Multiprocessing — the only one that gives real CPU parallelism in Python.
Full comparison here.
Throughput vs Latency (They Fight)
Under load these two pull in opposite directions, and optimizing blindly for one wrecks the other.
- Throughput — requests handled per second (QPS). Batching and high concurrency raise it.
- Latency — time for one request. Batching and deep queues raise it.
- Tail latency (p99/p99.9) — the slowest 1%. This is what users actually feel, and it explodes first under load. A p50 of 20ms with a p99 of 3s is a broken system.
flowchart LR
L["Rising load"] --> Q["Queues fill up"]
Q --> T["Throughput plateaus
(at max capacity)"]
Q --> P["p99 latency spikes
(waiting in queue)"]
P --> D["Timeouts & retries
→ MORE load"]
D --> COL["💥 congestion collapse"]
Little's Law: concurrency = throughput × latency. If each request takes
100ms and you want 1000 req/s, you need ~100 in flight at once. This one formula sizes your pools,
your semaphores, and your queues.
The Ceiling: Amdahl's Law
Parallelism has a hard limit. If 10% of your work is inherently serial, then even with infinite cores
you can never go more than 10× faster. Chasing more workers past that point just adds
coordination overhead.
# Speedup ceiling with a serial fraction s and N workers:
# speedup(N) = 1 / (s + (1 - s) / N)
# s = 0.10 (10% serial), N -> infinity => max speedup = 1 / 0.10 = 10x
# The lesson: kill the serial bottleneck (locks, single DB, GIL) FIRST.
The Load-Optimization Playbook
Every page in this collection is one move in this loop. Run it in order:
flowchart TB
M["1. Measure
(profile, load-test)"] --> B["2. Find the bottleneck
(CPU / IO / mem / downstream)"]
B --> F["3. Fix the biggest one
(pool, batch, cap, cache, parallelize)"]
F --> P["4. Protect it
(timeouts, backpressure, limits)"]
P --> M
- Reuse, don't recreate — connections, clients, processes (pooling).
- Do less work — cache and batch.
- Bound everything — concurrency caps, bounded queues, rate limits.
- Parallelize the CPU part — multiprocessing.
- Survive failure — timeouts, retries, circuit breakers.
- Watch memory — stream, don't buffer.
- Verify — profile and benchmark the change.
The mantra: Reuse → Do less → Bound it → Parallelize → Protect it → Measure again.
Optimization without measurement is just superstition.