🛑 Pattern 9: Shutdown & Errors

The Big Surprise: A Raw Process Exception Does Not Reach You

Coming from most languages, you expect an unhandled exception in a child to blow up your program. It doesn't. A child Process runs a separate interpreter: when its target raises, the traceback prints to the child's stderr and the child exits with a non-zero exitcode. The parent sails on, oblivious — unless you check exitcode yourself.

import multiprocessing as mp

def worker():
    raise ValueError("boom")          # prints a traceback in the CHILD, then the child dies

if __name__ == "__main__":
    p = mp.Process(target=worker)
    p.start()
    p.join()
    # The parent never saw the exception. You must inspect the exit code.
    print("exitcode:", p.exitcode)    # -> 1  (0 == clean, >0 == raised, <0 == killed by signal N)
    if p.exitcode != 0:
        raise RuntimeError(f"child failed with exitcode {p.exitcode}")

Pools & Executors Re-Raise For You — When You Read the Result

This is the practical reason to prefer a pool over raw Process for anything that returns a value. When a worker in a Pool or ProcessPoolExecutor raises, the library pickles the exception, ships it back, and re-raises it in the parent — but only at the moment you actually ask for that task's result.

from concurrent.futures import ProcessPoolExecutor

def maybe_fail(x):
    if x == 3:
        raise ValueError(f"cannot handle {x}")
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        fut = pool.submit(maybe_fail, 3)
        try:
            fut.result()               # <- the ValueError is re-raised HERE, in the parent
        except ValueError as e:
            print("parent caught:", e) # traceback shows both parent and child frames

        # With map(), the exception surfaces when you ITERATE onto the failing item:
        results = pool.map(maybe_fail, [1, 2, 3, 4])
        try:
            for r in results:          # 1, 4, then boom on the 3rd
                print(r)
        except ValueError as e:
            print("map raised:", e)

When the OS Kills a Worker: No Exception Can Come Back

A Python exception is a cooperative thing — the worker has to be alive to raise and pickle it. A segfault in a C extension or the OOM killer reaping your process gives the worker no chance to say goodbye. The two pool libraries handle this catastrophe differently:

The Worker Lifecycle

Every one of these endings routes back through the same place: join() (or the pool's shutdown) is where the process is reaped and where its outcome becomes visible.

Stopping Work: The Blunt Way vs. The Graceful Way

Sometimes you need to stop a worker that isn't done. There are two philosophies, and you should reach for them in this order — graceful first, blunt only as a last resort.

The Blunt Instruments: terminate() and kill()

• p.terminate() — sends SIGTERM. The worker stops immediately, mid-instruction. No finally blocks, no cleanup, no chance to flush.
• p.kill() — sends SIGKILL (3.7+). Even more brutal; the process cannot ignore or catch it.

The Graceful Way: An Event the Worker Polls

Instead of killing the worker, ask it to stop and let it finish its current unit of work and clean up. A shared multiprocessing.Event is the standard signal.

import multiprocessing as mp
import time

def worker(stop_event, results):
    while not stop_event.is_set():            # check the flag each loop
        results.put(do_one_unit_of_work())    # a whole, safe unit
    cleanup()                                 # runs because we EXIT normally

def do_one_unit_of_work():
    time.sleep(0.2)
    return "chunk"

def cleanup():
    print("worker cleaned up and exited gracefully")

if __name__ == "__main__":
    stop_event = mp.Event()
    results = mp.Queue()
    p = mp.Process(target=worker, args=(stop_event, results))
    p.start()

    time.sleep(1)              # let it run a while
    stop_event.set()           # ask it to stop — no corruption, cleanup runs
    p.join(timeout=5)          # give it a bounded chance to exit
    if p.is_alive():           # it ignored us / is stuck
        p.terminate()          # NOW escalate to the blunt instrument
        p.join()

A sentinel value on a queue is the equivalent idiom for pipeline workers: the parent puts a special marker (often None) and each worker exits when it pulls that marker off the queue (Pattern 3).

Shutting Down a Pool or Executor

Pools have their own vocabulary for the graceful-vs-abrupt distinction.

from concurrent.futures import ProcessPoolExecutor

def task(x):
    return x * x

if __name__ == "__main__":
    # PREFER the context manager: it calls shutdown(wait=True) for you on exit,
    # even if the block raises. This is the clean, correct default.
    with ProcessPoolExecutor() as pool:
        futures = [pool.submit(task, i) for i in range(1000)]
    # <-- all workers reaped here; no zombies, no leaks

    # Manual equivalent when you need to bail out early and drop pending work:
    pool = ProcessPoolExecutor()
    futures = [pool.submit(task, i) for i in range(1000)]
    pool.shutdown(wait=True, cancel_futures=True)   # 3.9+: skip not-yet-started tasks

Ctrl-C: Why It Hits Everyone, and How to Tame It

The standard recipe: ignore SIGINT in the workers so only the parent handles the interrupt. You install the ignore inside a pool initializer, which runs once per worker at startup.

import multiprocessing as mp
import signal, time

def init_worker():
    # Workers ignore Ctrl-C; only the PARENT will react to SIGINT.
    signal.signal(signal.SIGINT, signal.SIG_IGN)

def task(x):
    time.sleep(1)
    return x * x

if __name__ == "__main__":
    pool = mp.Pool(initializer=init_worker)      # every worker ignores SIGINT
    try:
        result = pool.map_async(task, range(20))
        print(result.get(timeout=60))            # parent blocks here, still sees Ctrl-C
    except KeyboardInterrupt:
        print("\ninterrupted — terminating workers")
        pool.terminate()                         # parent decides to stop everyone
    else:
        pool.close()
    finally:
        pool.join()

Zombies, Orphans, and Daemon Children

• Zombie: a finished child not yet join()ed. Harmless individually, but leaks table entries at scale. Fix: always join() / use with.
• Orphan: a still-running child whose parent died. It keeps consuming CPU/RAM with no one to reap it. Fix: reap on exit; consider atexit to terminate children if the parent unwinds, or make workers watch for the parent's death.
• Daemon child (p.daemon = True before start()): the parent abruptly kills it when the parent exits — abrupt, so no cleanup, same corruption caveats as terminate(). And a daemon cannot have children of its own (multiprocessing forbids it). Use daemons for fire-and-forget background helpers whose loss you don't care about, never for work whose results or side effects matter.

import multiprocessing as mp
import atexit

def heartbeat():
    while True:
        emit_heartbeat()          # a background helper we don't need to join

def emit_heartbeat():
    pass

if __name__ == "__main__":
    p = mp.Process(target=heartbeat, daemon=True)  # dies with the parent
    p.start()
    atexit.register(p.terminate)                   # belt-and-braces cleanup on exit
    # ... main program runs; when it exits, the daemon is killed automatically