🗺️ Pattern 5: Pool & map

What Pool Is

multiprocessing.Pool keeps a fixed set of worker processes alive and feeds them tasks. It predates concurrent.futures (Pattern 2), and it's still the tool of choice when you want fine control over how work is distributed: lazy streaming, completion-order results, or batching many tiny tasks. For everyday "run N functions, get N results" you can reach for ProcessPoolExecutor — but Pool gives you imap and chunksize, which the executor's map exposes only partially.

The Methods

pool.map(fn, iterable) — simplest, ordered, blocking

Splits the iterable into chunks, dispatches them, blocks until every result is ready, and returns a list in the same order as the input. It materializes the whole input and the whole output list in memory.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.map(square, range(10))   # blocks here
    print(results)   # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81] — input order

pool.imap(fn, iterable) — lazy, ordered, streaming

Returns a lazy iterator. Results are yielded in input order as they become ready, and the input is consumed lazily too — so you can feed a huge or even unbounded source without materializing it. Use when the input doesn't fit comfortably in memory, or you want to start processing results before the whole job finishes.

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        # nothing computed until you iterate; results stream in input order
        for result in pool.imap(square, range(1_000_000)):
            if result > 100:
                break   # we never had to build a million-element list

pool.imap_unordered(fn, iterable) — completion order

Like imap, but yields each result the moment it's done, ignoring input order. Fastest-finishing tasks come back first. Ideal when order doesn't matter and task durations vary widely — you get a steady stream instead of stalling on one slow item at the head of the queue.

import time
from multiprocessing import Pool

def work(x):
    time.sleep(x)   # varying durations
    return x

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        for result in pool.imap_unordered(work, [3, 1, 2, 1]):
            print("done:", result)   # 1, 1, 2, 3 — completion order, not input order

pool.starmap(fn, iterable_of_tuples) — multi-arg functions

map passes exactly one argument to fn. When your function takes several arguments, starmap unpacks each tuple as positional args — fn(a, b) from (a, b).

from multiprocessing import Pool

def add(a, b):
    return a + b

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        results = pool.starmap(add, [(1, 2), (3, 4), (5, 6)])
    print(results)   # [3, 7, 11]

pool.apply_async(fn, args, callback=, error_callback=) — one async call

Fires a single call and returns an AsyncResult immediately (non-blocking). Call .get(timeout=) to retrieve the value — it re-raises in the parent if the worker threw. Optional callback/error_callback run in the parent when the task succeeds/fails.

from multiprocessing import Pool

def slow_double(x):
    return x * 2

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        handle = pool.apply_async(slow_double, args=(21,))
        # ... do other work in the parent ...
        print(handle.get(timeout=5))   # 42 — raises TimeoutError if not ready

chunksize: The Performance Knob

Every dispatch of work to a worker crosses the process boundary — that means pickling and an IPC round-trip (Pattern 8). With many tiny tasks, that per-task overhead dwarfs the actual computation. chunksize tells the pool to hand each worker a batch of items per dispatch instead of one at a time — collapsing thousands of round-trips into a handful.

from multiprocessing import Pool

def tiny(x):
    return x + 1        # trivial work — overhead-dominated

if __name__ == "__main__":
    data = range(1_000_000)
    with Pool(processes=8) as pool:
        # chunksize=1 (default for imap): ~1M IPC hops → slow
        # chunksize=10_000: ~100 hops → often 10-50x faster here
        results = pool.map(tiny, data, chunksize=10_000)
    print(len(results))

Streaming vs Materializing (Memory)

from multiprocessing import Pool

def process_record(record_id):
    return {"id": record_id, "ok": True}   # imagine a big dict

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        written = 0
        # stream: handle each result and let it be garbage-collected
        for out in pool.imap_unordered(process_record, range(10_000_000), chunksize=1_000):
            written += 1   # e.g. append to a file here instead of a list
        print("processed", written)

Pool Lifecycle

A pool holds live child processes and OS resources, so it must be shut down. The clean way is the context manager, which handles teardown for you:

from multiprocessing import Pool

def square(x):
    return x * x

if __name__ == "__main__":
    with Pool(processes=4) as pool:        # workers spawned on entry
        print(pool.map(square, range(5)))
    # on exit: pool.terminate() is called, workers are stopped

Manual control uses three verbs; know the difference between the graceful pair and the abrupt one (Pattern 9 goes deep on shutdown):

    pool = Pool(processes=4)
    pool.map(square, range(5))
    pool.close()   # no more tasks accepted
    pool.join()    # wait for everything to drain

Pool vs ProcessPoolExecutor

Both keep a set of workers and fan out tasks. They differ in their result abstraction and how much distribution control they expose.

Key Takeaways

• map — ordered, blocking, materializes everything. The simple default.
• imap / imap_unordered — lazy streaming; input-order vs completion-order. Use for huge inputs and flat memory.
• starmap — for multi-argument functions.
• apply_async — one call, AsyncResult.get(timeout=), optional callbacks.
• chunksize — batch tiny tasks to kill IPC overhead; not too big or the tail straggles.
• Lifecycle — prefer with Pool(...); else close() → join(), terminate() only to abort.