🧰 Common Libraries Compared

Same Goal, Different Layers & Scales

Every library below gets you "more than one thing at once," but they sit at very different altitudes. At the bottom is one primitive — the stdlib multiprocessing module — and almost everything else is either a friendlier wrapper over it or a bigger system that competes with it. Two axes separate them:

• Layer — how much boilerplate the library hides. Low-level (multiprocessing) gives you raw control and maximum verbosity; high-level (concurrent.futures, joblib) gives you one clean call.
• Scale — how far it stretches. Most stop at the cores of one machine; Dask, Ray, and mpi4py cross the machine boundary to a cluster.

The stdlib Foundation: multiprocessing

multiprocessing is the primitive everything else stands on. It ships in the standard library (no install) and hands you the raw building blocks: Process to launch a worker, Pool for a batch of them, Queue/Pipe to pass messages, shared_memory for zero-copy buffers, and the full set of sync primitives (Lock, Semaphore, Event). It gives you maximum control at the cost of being the most verbose, and it moves data with the built-in pickle — so arguments and results must be picklable.

import multiprocessing as mp

def square(x):
    return x * x

if __name__ == "__main__":                 # required on spawn platforms
    with mp.Pool(processes=4) as pool:      # classic worker pool
        results = pool.map(square, range(10))
    print(results)                          # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

The Recommended Default: concurrent.futures

Also stdlib, but a layer up. concurrent.futures exposes one clean Executor abstraction with two backends — ProcessPoolExecutor (real parallelism, separate processes) and ThreadPoolExecutor (shared memory, for I/O). You submit() work and get a Future back, or map() over an iterable. Results come out of .result(), and — critically — exceptions raised in a worker are re-raised there, so errors surface cleanly instead of vanishing into a child process.

from concurrent.futures import ProcessPoolExecutor

def square(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:       # defaults to os.cpu_count() workers
        futures = [pool.submit(square, x) for x in range(10)]
        results = [f.result() for f in futures]   # re-raises worker exceptions here
    print(results)

The Scientific Favorite: joblib (and loky under it)

joblib is a third-party library (pip install joblib) beloved in the NumPy/scikit-learn world — it's what powers n_jobs=-1 across sklearn. Its signature idiom is a one-liner list comprehension:

from joblib import Parallel, delayed

def square(x):
    return x * x

# n_jobs=-1 means "use every core"
results = Parallel(n_jobs=-1)(delayed(square)(x) for x in range(10))
print(results)

delayed(fn)(x) just packages the call for later; Parallel dispatches the batch. By default it runs on the loky backend — a robust reusable process pool — but you can switch to threading or the classic multiprocessing backend with one argument. Three features earn it its fans:

• cloudpickle serialization — unlike stdlib pickle, it can serialize lambdas and closures, so the picky "can't pickle a local function" errors from Pattern 8: Pickling mostly disappear.
• Automatic memory-mapping of large NumPy arrays — big read-only arrays are mmap'd into workers instead of pickled and copied, avoiding the serialization tax on numeric data.
• Transparent disk caching via joblib.Memory — memoize expensive function results to disk across runs.

from joblib import Parallel, delayed

# closures & lambdas work here (cloudpickle) — they would fail with stdlib pickle
factor = 3
results = Parallel(n_jobs=-1, backend="loky")(
    delayed(lambda x: x * factor)(x) for x in range(10)
)

loky, briefly

loky is the standalone executor joblib is built on. It's a drop-in, concurrent.futures-style pool (get_reusable_executor()) that is more robust to worker crashes and reuses workers across batches instead of respawning them. You rarely import it directly — you get its benefits for free through joblib — but it's why joblib's pool tolerates segfaulting native code better than a plain ProcessPoolExecutor does.

The Odd One Out: subprocess

subprocess is stdlib and often confused with multiprocessing — but it does something completely different. It does not parallelize Python. It launches an external program: a compiled binary, a shell command, ffmpeg, git, or another script. The "work" isn't a Python function you're fanning out across cores — it's a separate executable you're driving from Python.

import subprocess

# run an EXTERNAL program and capture its output
result = subprocess.run(
    ["ffmpeg", "-i", "in.mov", "out.mp4"],
    capture_output=True,
    text=True,
    check=True,           # raise CalledProcessError on non-zero exit
)
print(result.stdout)

Crossing the Machine Boundary: dask & ray

When one machine's cores or RAM aren't enough, you climb to a distributed framework. Both are heavy third-party dependencies and both scale from a laptop to a cluster — but they have different shapes.

dask — parallel arrays, DataFrames, and lazy task graphs

dask mirrors familiar APIs: dask.array looks like NumPy and dask.dataframe looks like pandas, but the data is chunked and operations build a lazy task graph that only executes on .compute(). That enables out-of-core work — datasets larger than RAM, processed chunk by chunk — plus a general dask.delayed / futures API for arbitrary functions.

import dask

@dask.delayed
def square(x):
    return x * x

# builds a lazy graph; nothing runs until .compute()
graph = [square(x) for x in range(10)]
results = dask.compute(*graph)     # executes across threads / processes / a cluster
print(results)

ray — distributed tasks and stateful actors

ray targets distributed compute for Python and ML. Decorate a function with @ray.remote to turn it into a task, or decorate a class to get a stateful actor running on some node in the cluster. Its object store keeps large shared objects in memory and hands out references, so data isn't re-pickled for every task that touches it.

import ray

ray.init()                          # connects to a local or remote cluster

@ray.remote
def square(x):
    return x * x

futures = [square.remote(x) for x in range(10)]   # returns ObjectRefs
results = ray.get(futures)                         # gather actual values
print(results)

The HPC Option: mpi4py

mpi4py is Python bindings for MPI (Message Passing Interface), the decades-old standard for high-performance computing across supercomputer nodes. The model is explicit: every process has a rank, and you send/receive messages over a communicator. You launch it with a job scheduler (mpirun/srun), not from inside a single Python program.

# run with:  mpirun -n 4 python squares.py
from mpi4py import MPI

comm = MPI.COMM_WORLD
rank = comm.Get_rank()          # which process am I?
size = comm.Get_size()          # how many are there?

result = rank * rank            # each rank squares its own id
gathered = comm.gather(result, root=0)   # rank 0 collects everything
if rank == 0:
    print(gathered)             # [0, 1, 4, 9]

The Primitive Underneath: multiprocessing.shared_memory

Worth calling out on its own: multiprocessing.shared_memory is a stdlib primitive that gives multiple processes a single, zero-copy block of RAM — no pickling, no copying. It's the mechanism several of the libraries above lean on (joblib's array memory-mapping, Ray's object store solve the same "don't copy the big buffer" problem). You'll usually meet it through a higher-level library, but when you're hand-rolling multiprocessing and a large NumPy array is the bottleneck, it's the escape hatch. Deep dive in Pattern 4: Shared Memory.

Same Task, Many Ways

The clearest way to feel the ergonomic differences: one identical embarrassingly-parallel task — square(x) over a list — expressed in four libraries. The work is the same every time; only the wrapper changes.

(a) concurrent.futures.ProcessPoolExecutor — stdlib default

from concurrent.futures import ProcessPoolExecutor

def square(x):
    return x * x

if __name__ == "__main__":
    with ProcessPoolExecutor() as pool:
        results = list(pool.map(square, range(10)))

(b) joblib.Parallel + delayed — the numeric one-liner

from joblib import Parallel, delayed

def square(x):
    return x * x

results = Parallel(n_jobs=-1)(delayed(square)(x) for x in range(10))

(c) dask.delayed — same call, but a lazy graph that can span a cluster

import dask

@dask.delayed
def square(x):
    return x * x

results = dask.compute(*[square(x) for x in range(10)])

(d) ray — .remote tasks, gathered via the object store

import ray
ray.init()

@ray.remote
def square(x):
    return x * x

results = ray.get([square.remote(x) for x in range(10)])

The Big Comparison

Which One? — The Decision Flowchart

The Takeaway

Installation Reality

For the broader "should I even be using processes?" question — the serialization tax, when threads or async beat processes — see Processes vs Threads vs Async, and start from the top with Processes 101.