🎚️ Thread Pool Tuning

The Default — and Why It's Only a Starting Point

Since Python 3.8, if you don't pass max_workers, ThreadPoolExecutor defaults to min(32, os.cpu_count() + 4). On an 8-core box that's 12; on a 64-core box it's capped at 32. That formula is a compromise — enough threads to overlap some I/O without spawning hundreds by accident. It is almost never the right number for your workload.

import os
from concurrent.futures import ThreadPoolExecutor

# What the default resolves to on this machine:
default_workers = min(32, (os.cpu_count() or 1) + 4)
print(default_workers)   # e.g. 12 on an 8-core host

with ThreadPoolExecutor() as pool:   # uses the default above
    ...

The Sizing Model

I/O-bound: size to concurrency, not to cores

An I/O-bound thread spends 95%+ of its wall-clock time blocked on a socket, releasing the GIL while it waits. So you can run hundreds of them on 8 cores — they're not fighting for CPU, they're fighting for the network. The right count comes straight from Little's Law:

# Little's Law: workers ≈ target_throughput × per_request_latency
#
#   want 500 downstream calls/sec, each takes 200ms (0.2s):
#       workers ≈ 500 × 0.2 = 100 in flight
#
# So max_workers ≈ 100 for I/O-bound — even on an 8-core host.
# The threads are asleep on recv(); cores are almost idle.
target_qps = 500
latency_s = 0.2
max_workers = round(target_qps * latency_s)   # 100

CPU-bound: threads don't help — use processes

Under CPython's GIL only one thread executes Python bytecode at a time. Throwing 16 threads at a pure-Python CPU task doesn't run it in parallel — they take turns, plus you pay context-switch and GIL-handoff overhead. The result is often slower than a single thread.

import time
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def burn(n):                     # pure-Python CPU work — holds the GIL
    total = 0
    for i in range(n):
        total += i * i
    return total

work = [10_000_000] * 8

t = time.perf_counter()
with ThreadPoolExecutor(max_workers=8) as pool:   # GIL-serialized
    list(pool.map(burn, work))
print("threads:", round(time.perf_counter() - t, 2), "s")

t = time.perf_counter()
with ProcessPoolExecutor(max_workers=8) as pool:  # real parallelism
    list(pool.map(burn, work))
print("processes:", round(time.perf_counter() - t, 2), "s")
# On 8 cores: threads ≈ single-threaded time; processes ≈ ~8× faster.

Worker-Count Heuristics

Measuring the Sweet Spot

Don't guess — sweep max_workers and time it. Throughput climbs, plateaus (you've saturated the real bottleneck), then degrades as overhead and contention take over. The knee is your number.

import time
from concurrent.futures import ThreadPoolExecutor

def io_task(_):
    time.sleep(0.05)      # simulate a 50ms downstream call (blocked, GIL freed)

def measure(n_workers, n_tasks=2000):
    t = time.perf_counter()
    with ThreadPoolExecutor(max_workers=n_workers) as pool:
        list(pool.map(io_task, range(n_tasks)))
    elapsed = time.perf_counter() - t
    return n_tasks / elapsed          # throughput, tasks/sec

for w in (1, 2, 4, 8, 16, 32, 64, 128, 256, 512):
    print(f"{w:>4} workers -> {measure(w):8.1f} tasks/s")
# Throughput rises steeply, then flattens once workers ≫ (tasks × latency).
# Past the knee you gain nothing but memory and scheduler overhead.

The Danger of Too Many Threads

"Just bump it to 1000" is not free. Every thread costs real resources, and past the knee you buy nothing but risk:

• Memory per thread stack — each OS thread reserves a stack (commonly ~8 MB virtual on Linux). 1000 threads is a lot of address space and non-trivial committed memory.
• Context-switch overhead — the scheduler thrashes shuffling far more runnable threads than cores.
• Blowing past downstream limits — 1000 threads slamming a DB with max_connections=100 means 900 pile up or get refused. You've moved the queue, not removed it.

import threading

print(threading.stack_size())     # default thread stack (bytes); often 0 = OS default (~8MB)

# Shrink stacks BEFORE creating threads if you truly need many of them:
threading.stack_size(512 * 1024)  # 512 KB — enough for shallow I/O work
# But if you need thousands of concurrent I/O tasks, threads are the wrong tool.
# That's exactly what asyncio is for — tens of thousands of tasks, one thread.

Pool Starvation — the Deadlock Nobody Sees Coming

If a task running in a pool submits another task to the same pool and then blocks waiting on that child's result, you can deadlock. All workers get stuck holding parents that are waiting on children that can never be scheduled — the pool is starved by itself.

from concurrent.futures import ThreadPoolExecutor

pool = ThreadPoolExecutor(max_workers=2)   # small to make it deterministic

def child(x):
    return x * 2

def parent(x):
    fut = pool.submit(child, x)   # ❌ submits to the SAME pool...
    return fut.result()           # ...then blocks a worker waiting for it

# With 2 workers, 2 parents occupy both slots and block on results.
# Their children never get a worker -> deadlock, hangs forever.
futures = [pool.submit(parent, i) for i in range(4)]
# for f in futures: print(f.result())   # <- would hang

from concurrent.futures import ThreadPoolExecutor

parent_pool = ThreadPoolExecutor(max_workers=2)
child_pool  = ThreadPoolExecutor(max_workers=4)   # ✅ different pool for nested work

def child(x):
    return x * 2

def parent(x):
    fut = child_pool.submit(child, x)   # child runs in its own pool — no starvation
    return fut.result()

results = [f.result() for f in [parent_pool.submit(parent, i) for i in range(4)]]
print(results)   # [0, 2, 4, 6]

# Better still for fan-out/fan-in: don't nest at all — flatten to one map,
# or use asyncio where awaiting doesn't consume a worker.

Bounding the Work Queue

ThreadPoolExecutor's internal work queue is unbounded. If producers submit faster than workers drain, tasks pile up in memory indefinitely — a slow-motion OOM under backlog. The pool size caps concurrency, not how much waiting work you accept.

import threading
from concurrent.futures import ThreadPoolExecutor

# A semaphore admits at most (workers + queue_slack) outstanding tasks.
# submit() blocks once full -> producer feels backpressure instead of buffering forever.
class BoundedExecutor:
    def __init__(self, max_workers, max_queued):
        self._pool = ThreadPoolExecutor(max_workers=max_workers)
        self._slots = threading.Semaphore(max_workers + max_queued)

    def submit(self, fn, *args, **kwargs):
        self._slots.acquire()                 # blocks when the bound is hit
        fut = self._pool.submit(fn, *args, **kwargs)
        fut.add_done_callback(lambda _: self._slots.release())
        return fut

    def shutdown(self, **kw):
        self._pool.shutdown(**kw)

# Now a burst of 1,000,000 submits can't balloon memory — it blocks the producer.

One Global Executor vs Per-Purpose Executors

A single shared pool is simple but couples unrelated workloads: a burst of slow DB tasks starves your fast cache lookups because they share the same workers. Isolate workloads that must not block each other into separate pools with independent sizing.

# ❌ One pool for everything — slow tasks starve fast ones
shared = ThreadPoolExecutor(max_workers=20)

# ✅ Per-purpose pools sized to their own bottleneck
db_pool    = ThreadPoolExecutor(max_workers=20, thread_name_prefix="db")    # = DB pool size
http_pool  = ThreadPoolExecutor(max_workers=100, thread_name_prefix="http") # I/O-heavy fan-out
cpu_pool   = ProcessPoolExecutor(max_workers=os.cpu_count())                # CPU work escapes the GIL

# Bonus: thread_name_prefix makes stacks/py-spy dumps readable per workload.

Graceful Shutdown

On shutdown you usually want in-flight work to finish but pending work to be dropped fast — especially during a deploy or SIGTERM. Since Python 3.9, cancel_futures=True discards tasks that haven't started yet.

pool = ThreadPoolExecutor(max_workers=50)
# ... submit work ...

# Wait for running tasks, but cancel everything still queued:
pool.shutdown(wait=True, cancel_futures=True)   # Python 3.9+

# As a context manager, __exit__ calls shutdown(wait=True) for you — running
# AND queued tasks all complete before the block exits (no cancel):
with ThreadPoolExecutor(max_workers=50) as pool:
    pool.map(io_task, range(1000))
# <- blocks here until all 1000 finish

The Whole Picture

Tasks flow into a bounded pool of N workers; N is your concurrency, but the real limit is whichever downstream resource saturates first.