🧠 Pattern 4: Shared Memory

The Problem: Copying Isn't Free

Pattern 3 gave you message passing — safe, simple, and it copies. Send a 2 GB NumPy array to four workers and you've pickled and copied 8 GB. Increment a shared counter through a queue and every tick is a round-trip. For big buffers or hot shared state, the copy costs more than the computation.

Shared memory fixes this: the OS maps the same physical bytes into multiple processes. There's one copy of the data, and every process reads and writes it directly. The catch is that you've now re-introduced the thing processes were protecting you from — shared mutable state — so you're back to needing locks (Pattern 6).

Value and Array: Shared Scalars and Buffers

mp.Value and mp.Array are ctypes-backed primitives that live in shared memory. You declare a C type code up front: Value('i', 0) is a shared 32-bit int, Array('d', [1.0, 2.0, 3.0]) is a shared array of doubles. You read and write through the .value attribute (for Value) or by indexing (for Array). Because they're plain C types, they're fast and involve no pickling.

import multiprocessing as mp

def increment(counter, times):
    for _ in range(times):
        with counter.get_lock():      # MANDATORY — += is not atomic
            counter.value += 1

if __name__ == "__main__":
    counter = mp.Value('i', 0)        # shared 32-bit int, starts at 0

    workers = [
        mp.Process(target=increment, args=(counter, 100_000))
        for _ in range(4)
    ]
    for w in workers:
        w.start()
    for w in workers:
        w.join()

    print(counter.value)              # 400000 — exact, because of the lock

Drop the with counter.get_lock(): and you'll get some number less than 400000, and a different one on every run. See Pattern 6 for why the lock is non-negotiable and how cross-process locks differ from threading ones.

shared_memory.SharedMemory: Zero-Copy Buffers (3.8+)

multiprocessing.shared_memory.SharedMemory is a raw, named block of shared memory — just bytes, addressable by a string name. It's the way to share a large NumPy array with zero copy: you allocate a block, wrap it with a NumPy view via the buffer= argument, and then pass only the block's name (a small string) to the workers. Each worker attaches to the same block by that name and sees the same bytes — no pickling of the array itself.

import numpy as np
from multiprocessing import Process
from multiprocessing.shared_memory import SharedMemory

def worker(name, shape, dtype):
    shm = SharedMemory(name=name)                 # ATTACH by name (no copy)
    arr = np.ndarray(shape, dtype=dtype, buffer=shm.buf)
    arr[:] = arr * 2                              # mutate the shared bytes in place
    shm.close()                                   # detach THIS process's mapping

if __name__ == "__main__":
    data = np.arange(10, dtype=np.int64)
    shm = SharedMemory(create=True, size=data.nbytes)   # allocate the block

    # wrap the block as a NumPy array and fill it
    shared = np.ndarray(data.shape, dtype=data.dtype, buffer=shm.buf)
    shared[:] = data[:]

    p = Process(target=worker, args=(shm.name, data.shape, data.dtype))
    p.start()
    p.join()

    print(shared)          # [0 2 4 6 8 10 12 14 16 18] — worker's writes are visible
    shm.close()            # detach the parent's mapping
    shm.unlink()           # EXACTLY ONE process frees the block (see gotcha)

Note what crossed the process boundary: just shm.name, data.shape, and data.dtype — a few dozen bytes. The array itself never got pickled.

The Gotcha: SharedMemory Won't Free Itself

Manager: Shared Objects via a Server Process

A multiprocessing.Manager spins up a separate server process that holds real Python objects. Children get proxies — objects that look like a dict, list, or Namespace but forward every operation to the server over IPC. This is the convenient option: you can share arbitrary, nested Python objects, not just C scalars and raw bytes.

import multiprocessing as mp

def worker(shared_dict, key):
    shared_dict[key] = key * key      # each op is an IPC round-trip to the server

if __name__ == "__main__":
    with mp.Manager() as manager:
        shared_dict = manager.dict()  # a PROXY backed by the server process

        workers = [
            mp.Process(target=worker, args=(shared_dict, n))
            for n in range(5)
        ]
        for w in workers:
            w.start()
        for w in workers:
            w.join()

        print(dict(shared_dict))      # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

        # ❌ silently lost — mutates a local copy, server never updated
        shared_dict[k].append(x)

        # ✅ reassign so the proxy ships the new value back to the server
        shared_dict[k] = shared_dict[k] + [x]

Which One? A Decision Table

The Default You Should Reach For First