🥒 Pattern 8: Pickling & the Serialization Tax

The Core Fact: Everything Crosses as Bytes

Because each process has its own private memory, a worker can't just read a variable the parent holds. So multiprocessing serializes every argument with pickle, ships the bytes across the OS boundary, and rebuilds (unpickles) an independent copy on the other side. The result travels back the same way. This happens on every task submission and every return.

What Can't Be Pickled

Pickle stores a reference to a top-level, importable name plus the object's state. Anything that can't be named that way, or that wraps a live OS resource, fails to travel.

from concurrent.futures import ProcessPoolExecutor

if __name__ == "__main__":
    # ❌ A lambda has no importable name — the parent can't pickle it to send it.
    with ProcessPoolExecutor() as pool:
        pool.map(lambda x: x * 2, range(4))
    #   _pickle.PicklingError: Can't pickle <function <lambda>>:
    #   attribute lookup <lambda> on __main__ failed

The same failure hits a function defined inside another function, or a partial that closed over a socket. The rule of thumb: if you can't import it by a stable dotted path, pickle can't either.

The Fixes

from concurrent.futures import ProcessPoolExecutor
from functools import partial

def scale(factor, x):        # top-level, importable — picklable
    return x * factor

if __name__ == "__main__":
    # (a) + (b): partial binds `factor`, replacing the un-picklable lambda.
    double = partial(scale, 2)
    with ProcessPoolExecutor() as pool:
        print(list(pool.map(double, range(4))))   # [0, 2, 4, 6]

For connections, the fix is the initializer pattern: don't pickle a live DB handle (you can't). Pickle the DSN, and let each worker build its own connection once and stash it in a global for reuse.

from concurrent.futures import ProcessPoolExecutor

_CONN = None    # per-worker connection, rebuilt in each worker

def init_worker(dsn):
    global _CONN
    _CONN = connect(dsn)        # open ONCE per worker, not per task

def run_query(query):
    return _CONN.execute(query) # reuse the per-worker connection

def connect(dsn):
    return dsn                  # stand-in for a real driver

if __name__ == "__main__":
    dsn = "postgresql://localhost/app"        # a picklable STRING, not a handle
    with ProcessPoolExecutor(initializer=init_worker, initargs=(dsn,)) as pool:
        print(list(pool.map(run_query, ["SELECT 1", "SELECT 2"])))

See Pattern 6 for passing sync primitives through initargs the same way (they can't be pickled as plain arguments either).

The Serialization Tax: It's Also About Speed

Rules to cut the tax:

• Pass references, not payloads. Send indices, file paths, row ranges, or small params — never the big blob itself.
• Let workers read their own slice. Hand each worker a path + offset and have it load its chunk from disk, so the big data never goes through a pipe.
• Share, don't copy, big numeric arrays. Put them in shared_memory (Pattern 4) and pass the name; workers attach to the same buffer with zero copy.

from concurrent.futures import ProcessPoolExecutor

# ❌ Copies the whole big list into every task's pickle.
# def work(big_list, i): ...

def work(path_and_index):        # ✅ pass a tiny (path, index) tuple instead
    path, i = path_and_index
    with open(path) as f:
        line = f.readlines()[i]  # worker reads only what it needs
    return len(line)

if __name__ == "__main__":
    jobs = [("data.txt", i) for i in range(100)]   # small, cheap to pickle
    with ProcessPoolExecutor() as pool:
        print(sum(pool.map(work, jobs)))

spawn Raises the Stakes

Under the spawn start method (Pattern 7) — the default on macOS and Windows — pickling isn't optional plumbing you can dodge with fork's inherited memory. spawn pickles the target function's arguments and effectively requires your module to be importable so the child can rebuild the target by name. So the picklability rules here aren't a corner case: on the two most common developer platforms they govern whether your program starts at all.

Checking Picklability & the cloudpickle Escape Hatch

To test quickly whether something can travel, try to dump it in a REPL — if pickle.dumps raises, so will passing it to a worker.

import pickle

pickle.dumps({"a": 1})           # ok → bytes
pickle.dumps(lambda x: x)        # raises PicklingError — this would fail in a pool

There's also cloudpickle, a third-party serializer that can pickle lambdas and locally-defined functions by capturing their bytecode. It's why joblib, Dask, and Ray let you pass a lambda straight to a worker where the stdlib pool refuses. You don't get it for free with multiprocessing — those libraries swap the serializer in for you. More in Common Libraries.

Decision Table: Can It Travel?