Processes 101: What They Are & Why They're Different

What Is a Process?

A process is an operating-system-level program with its own private memory space, its own Python interpreter, and — crucially — its own GIL. When you start a Python program, the OS gives you exactly one process. multiprocessing lets you spawn more, and each one is a fully independent Python running in parallel.

Key Insight: Threads share memory; processes don't. That single difference flips everything. Threads communicate for free (just read a variable) but fight over one GIL. Processes each get their own GIL — real parallelism — but can't see each other's variables, so every byte of shared data must be copied across a boundary. Multiprocessing is the art of managing that copy.

Threads vs Processes in One Picture

flowchart TB subgraph PROC["Multiprocessing — isolated memory, a GIL each"] direction LR subgraph A["Process A"] MA["own memory
own GIL"] end subgraph B["Process B"] MB["own memory
own GIL"] end A -.->|"pickle + copy"| B end

Why Use Processes?

One reason, and it's a big one: true CPU parallelism. Threads and asyncio both run Python bytecode on a single core at a time because of the GIL. If your bottleneck is the CPU — crunching numbers, parsing, compressing, running a model — the only way in the standard library to use all your cores is to run multiple processes, each with its own GIL.

Scenario: a pure-Python function that burns 1 second of CPU, run 4 times, on a 4-core machine.

Sequential: 1s + 1s + 1s + 1s = 4 seconds.

4 threads: still ≈ 4 seconds — the GIL serializes them; threads do not help CPU work.

4 processes: all four cores run at once ≈ 1 second. This is the whole point.

from concurrent.futures import ProcessPoolExecutor
import time

def burn(_):
    # pure-Python CPU work — the GIL cannot be released here
    total = 0
    for i in range(30_000_000):
        total += i
    return total

if __name__ == "__main__":          # REQUIRED — see Pattern 7
    start = time.perf_counter()
    with ProcessPoolExecutor(max_workers=4) as pool:
        results = list(pool.map(burn, range(4)))
    print(f"done in {time.perf_counter() - start:.1f}s")   # ≈ 1s on 4 cores, not 4s

The Catch: Isolation Has a Price

Processes don't share memory, so everything you send to a child and every result it returns is serialized with pickle, copied across an OS boundary, and rebuilt on the other side. That has three consequences you'll feel constantly:

Startup is expensive. A new process means a new interpreter — tens of milliseconds each. Processes are worth it only when each task does real work.
Data isn't free. Passing a 500 MB array to a worker copies 500 MB. For tiny tasks the copy can cost more than the computation. (Pattern 8)
Not everything can travel. Lambdas, open sockets, database connections, and file handles can't be pickled — so they can't cross into a child. (Pattern 8)

The mental model: a child process is a coworker in another building. You can't just point at a variable on your desk — you have to photocopy it and mail it over. Cheap communication is the whole reason threads exist; expensive communication is the price processes pay for real parallelism.

The Rule You Can't Skip: `if name == "main"`

On macOS and Windows, Python starts child processes with the spawn method: a brand-new interpreter that re-imports your module to rebuild the worker. If your process- launching code runs at module top level, each child re-runs it on import — spawning children that spawn children, forever.

import multiprocessing as mp

def worker():
    print("working")

# ❌ WITHOUT the guard, on spawn platforms: infinite process explosion / RuntimeError
if __name__ == "__main__":          # ✅ this block only runs in the ORIGINAL process
    p = mp.Process(target=worker)
    p.start()
    p.join()

Remember this now: nearly every "my multiprocessing code spawns endless processes" or "RuntimeError: An attempt has been made to start a new process before…" bug is a missing __main__ guard. There's a whole guide on start methods (Pattern 7) — but internalize the guard before anything else.

The Building Blocks You'll Meet

Process — create, start(), join() a single process. The low-level primitive. (Pattern 1)
ProcessPoolExecutor — the modern, high-level way to run many tasks and get results back as Futures. Reach for this first. (Pattern 2)
Queue / Pipe — pass messages between processes, since you can't share variables. (Pattern 3)
Value / Array / shared_memory / Manager — the escape hatches for actually sharing state without copying. (Pattern 4)
Pool — the classic worker pool with map/imap/starmap. (Pattern 5)

Start high-level: in real code, reach for ProcessPoolExecutor (or Pool) first. Raw Process + Queue is worth understanding, but the pool handles spawning, result collection, and cleanup for you.

Processes vs Threads vs Async: The One-Line Rule

Your bottleneck is…	Use	Why
CPU (pure-Python number crunching)	processes	Only way past the GIL — a GIL per process
A handful of blocking I/O calls (sync libs)	threads	Shared memory, cheap, GIL frees on I/O
Thousands of concurrent I/O tasks	async	Scales huge for almost no memory

Rule of thumb: if adding cores would make your task faster, you want processes. If your task is mostly waiting, you don't — use threads or async instead. The full comparison lives in the decision guide.

The One-Paragraph Recap

A process is an independent Python with its own memory and its own GIL, so multiple processes genuinely run in parallel across cores — the one thing threads and async can't do. The price is isolation: nothing is shared, so arguments and results are pickled and copied, startup is costly, and some objects can't travel at all. You create workers with Process or (better) a ProcessPoolExecutor, move data with queues, pipes, or shared memory, and always guard the launch with if __name__ == "__main__". Use processes when the CPU is your bottleneck; reach for threads or async when you're just waiting.