⚙️ Pattern 2: ProcessPoolExecutor

The Pitch

concurrent.futures.ProcessPoolExecutor is the modern, high-level way to run work across cores. Raw Process makes you plumb queues, join by hand, and invent your own error handling. The executor does all of it: it keeps a fixed pool of worker processes alive, feeds them tasks, hands back a Future for each, and re-raises any worker exception when you read the result.

The Unifier: One Word Swaps Threads ↔ Processes

The whole point of concurrent.futures is a single API over two very different execution models. Threads and processes share the exact same submit / map / Future surface — you switch bottleneck strategies by changing one class name.

from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor

def work(x):
    return x * x

if __name__ == "__main__":
    # I/O-bound? threads. CPU-bound? processes. Same code below the swap:
    Executor = ProcessPoolExecutor        # <-- change this ONE word to ThreadPoolExecutor
    with Executor() as pool:
        print(list(pool.map(work, range(6))))   # [0, 1, 4, 9, 16, 25]

submit() and Futures

submit(fn, *args) schedules one call and immediately returns a Future — a handle to a result that doesn't exist yet. Call .result() to block until it's ready (and to receive the value, copied back from the worker).

from concurrent.futures import ProcessPoolExecutor, as_completed

def slow_square(x):
    total = 0
    for i in range(x * 5_000_000):   # variable CPU work per task
        total += i
    return x, x * x

if __name__ == "__main__":
    with ProcessPoolExecutor(max_workers=4) as pool:
        futures = [pool.submit(slow_square, x) for x in range(6)]
        # as_completed yields each Future the moment ITS worker finishes,
        # not in submission order — good for streaming results.
        for fut in as_completed(futures):
            arg, result = fut.result()   # blocks only if this one isn't done yet
            print(f"{arg} -> {result}")

Useful Future methods: .result(timeout=), .exception() (returns the raised exception instead of raising it), .done(), and .cancel() (only works while still queued, never once running).

map() vs submit + as_completed

Two ways to fan out; pick by whether you care about order or latency.

from concurrent.futures import ProcessPoolExecutor

def work(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        # map: simplest, results come back IN ORDER
        ordered = list(pool.map(work, range(6)))     # [0, 1, 4, 9, 16, 25]

        # map with several arg columns via zip / positional lists:
        def add(a, b): return a + b
        # NOTE: map passes one item per iterable, column-wise:
        sums = list(pool.map(add, [1, 2, 3], [10, 20, 30]))   # [11, 22, 33]
        print(ordered, sums)

max_workers and Why More ≠ Faster

max_workers defaults to os.cpu_count(). For CPU-bound work, that's the right ceiling: each worker saturates one core, so with N cores you get ~N× speedup and no more. Adding workers beyond core count doesn't add cores — it just adds processes contending for the same cores, plus extra interpreter memory and context-switching overhead.

import os
from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    print("cores:", os.cpu_count())
    # Default is fine for CPU work; set it explicitly to leave a core for the OS/UI:
    with ProcessPoolExecutor(max_workers=os.cpu_count() - 1) as pool:
        ...

Exceptions: Silent Until You Read

This is the Python gotcha that bites everyone. A worker exception does not crash the pool and does not print. It is captured, pickled, and stashed in that task's Future. You never see it until you read the result — at which point it re-raises in your process, with the original traceback attached as its cause.

from concurrent.futures import ProcessPoolExecutor, as_completed

def risky(x):
    if x == 3:
        raise ValueError(f"cannot process {x}")   # blows up inside the worker
    return x * 10

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        futures = {pool.submit(risky, x): x for x in range(6)}
        for fut in as_completed(futures):
            x = futures[fut]
            try:
                print(x, "->", fut.result())   # re-raises here for x == 3
            except ValueError as e:
                print(f"task {x} failed: {e}")   # handle per-task, keep going

Always Use It as a Context Manager

A pool holds real OS processes. Leak it and you leave orphaned interpreters running. The with block calls shutdown(wait=True) on exit, which stops accepting new tasks and blocks until every in-flight task finishes and every worker is torn down.

from concurrent.futures import ProcessPoolExecutor

def work(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:        # spins up workers
        results = list(pool.map(work, range(1000)))
    # <-- shutdown(wait=True) ran here: all tasks done, all workers gone
    print(sum(results))

    # Equivalent without the sugar:
    pool = ProcessPoolExecutor()
    try:
        results = list(pool.map(work, range(1000)))
    finally:
        pool.shutdown(wait=True)   # never skip this

Key Takeaways