⚖️ Choosing the Model Under Load

Step Zero: CPU-Bound or I/O-Bound?

Before you type a single import, classify the workload. Everything downstream flows from this one question, and getting it wrong means the fastest model you can pick will still be the slowest thing in your stack.

• I/O-bound — the process spends most of its time waiting: network calls, DB queries, disk reads, downstream APIs. CPU sits near idle while requests pile up. This is 90% of web services.
• CPU-bound — the process spends most of its time computing: parsing, serialization, image resizing, hashing, ML inference, number crunching. Cores pin at 100%.

The Big Comparison

The GIL Under Load: Threads Don't Parallelize CPU

CPython's Global Interpreter Lock allows only one thread to execute Python bytecode at a time. For CPU-bound work this means 8 threads on 8 cores run no faster than 1 thread — they just take turns holding the GIL, and the lock hand-off adds overhead, so it can be slower.

# ❌ Threads for CPU-bound work — the GIL serializes them
import time
from concurrent.futures import ThreadPoolExecutor

def crunch(n):
    total = 0
    for i in range(n):
        total += i * i        # pure Python bytecode → holds the GIL
    return total

N = 20_000_000
t0 = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:
    list(pool.map(crunch, [N] * 8))
print(f"8 threads: {time.perf_counter() - t0:.2f}s")   # ≈ same as running 8x serially

Two important caveats: (1) the GIL is released around blocking I/O syscalls and inside many C extensions (NumPy, hashlib, compression), so threads do overlap I/O and native compute. (2) Free-threaded CPython (the experimental 3.13t no-GIL build) changes this story — but on standard CPython in production today, assume the GIL is there.

The Decision Flowchart

One API, Three Models: concurrent.futures

The stdlib concurrent.futures module hides threads and processes behind the same Executor interface — submit(), map(), as_completed(). Swapping ThreadPoolExecutor ↔ ProcessPoolExecutor is often a one-line change, which makes it the perfect harness for A/B benchmarking a workload across models.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def run(executor_cls, fn, items, workers):
    with executor_cls(max_workers=workers) as ex:
        return list(ex.map(fn, items))    # identical call site for threads OR processes

# threads → run(ThreadPoolExecutor, task, items, 32)
# procs   → run(ProcessPoolExecutor, task, items, 8)

Benchmark: The Same Workload, Three Ways

Talk is cheap — time it. Below we run one I/O-bound task and one CPU-bound task under all three models. Fill in a real endpoint / real numbers, but the relative results below are what you should expect on standard CPython.

In Threading

import time
from concurrent.futures import ThreadPoolExecutor

def io_task(_):
    import time
    time.sleep(0.1)          # stand-in for a network/DB call (GIL released while sleeping)
    return 1

def cpu_task(_):
    total = 0
    for i in range(5_000_000):
        total += i * i       # pure Python → holds the GIL
    return total

def timed(label, fn, executor, workers, n):
    t0 = time.perf_counter()
    with executor(max_workers=workers) as ex:
        list(ex.map(fn, range(n)))
    print(f"{label:>14}: {time.perf_counter() - t0:.2f}s")

timed("threads I/O", io_task,  ThreadPoolExecutor, workers=100, n=100)   # ≈ 0.1s  ✅ overlaps
timed("threads CPU", cpu_task, ThreadPoolExecutor, workers=8,   n=8)     # ≈ serial ❌ GIL

In Multiprocessing

from concurrent.futures import ProcessPoolExecutor
# reuse the SAME io_task / cpu_task / timed defined above (must be module-level to pickle)

timed("procs   CPU", cpu_task, ProcessPoolExecutor, workers=8,  n=8)     # ≈ 8x faster ✅ real cores
timed("procs   I/O", io_task,  ProcessPoolExecutor, workers=8,  n=100)   # ok, but wasteful for I/O
# CPU work now spreads across 8 interpreters → wall time ≈ single-task time, not 8x it.

In Asyncio

import asyncio, time

async def io_task():
    await asyncio.sleep(0.1)     # non-blocking wait — the loop runs other tasks meanwhile
    return 1

async def main(n):
    t0 = time.perf_counter()
    await asyncio.gather(*(io_task() for _ in range(n)))
    print(f"   async I/O: {time.perf_counter() - t0:.2f}s")

asyncio.run(main(10_000))        # ≈ 0.1s for 10,000 tasks — one thread, tiny memory footprint

Where To Go Next

Once you've picked a model, the rest of this collection shows how to make it fast and safe under load — and most techniques apply across all three. Dive deeper into each model:

• Async patterns — event loop, gather, semaphores, timeouts: async guides →
• Threading patterns — pools, locks, thread-safe queues, the GIL in depth: threading guides →