🧠 Memory Optimization Under Load

The Problem: RSS × Concurrency = OOM

Under load, memory is never "how much does one request use" — it's per-request RSS multiplied by in-flight concurrency. Little's Law (intro) sizes that concurrency for you. The single most effective memory technique is to make per-request peak memory bounded and small, independent of input size. That almost always means: stream.

Generators: Lazy Iteration Instead of Giant Lists

A list comprehension materializes every element at once. A generator expression yields them one at a time and holds only the current one. Same syntax, one character of difference, a 1000× memory difference.

import sys

# ❌ Eager: builds the whole list in RAM before you touch a single element
squares_list = [x * x for x in range(10_000_000)]
print(sys.getsizeof(squares_list))   # ~85 MB just for the list object

# ✅ Lazy: holds one value at a time, computes on demand
squares_gen = (x * x for x in range(10_000_000))
print(sys.getsizeof(squares_gen))    # ~200 bytes — the generator object, nothing more

total = sum(squares_gen)             # streams through, peak memory stays flat

The same idea powers yield — a function that produces a sequence without ever holding it:

# Process a multi-GB log file line by line — RSS stays at one-line size
def parse_errors(path):
    with open(path) as f:
        for line in f:                    # the file object is itself a lazy iterator
            if "ERROR" in line:
                yield line.rstrip()

# Never materializes the file. Works on a 100 GB log on a 512 MB container.
error_count = sum(1 for _ in parse_errors("app.log"))

Streaming DB Cursors — Don't Fetch a Million Rows at Once

The classic memory bomb: cursor.fetchall() on a huge result set pulls the entire table into the driver's client buffer and your Python list. Use a server-side cursor so the DB streams rows in windows.

# psycopg (Postgres): a NAMED cursor is server-side and streams
import psycopg

with psycopg.connect(dsn) as conn:
    with conn.cursor(name="stream") as cur:   # named => server-side => streamed
        cur.itersize = 5_000                  # fetch 5k rows per network round-trip
        cur.execute("SELECT id, payload FROM events")
        for row in cur:                       # peak memory = one window, not the table
            process(row)

# SQLAlchemy: yield_per streams the ORM result in batches
from sqlalchemy import select

with Session(engine) as session:
    stmt = select(Event).execution_options(yield_per=1_000)
    for event in session.scalars(stmt):      # 1,000 rows in flight at a time
        process(event)                        # NOT a 10M-row list in RAM

Chunked HTTP Downloads — Never .content a Big Response

response.content / response.json() buffers the entire body. For large payloads, stream the socket and process (or write) in chunks.

import requests

# ✅ stream=True + iter_content: peak memory = one chunk, not the whole file
with requests.get(url, stream=True, timeout=30) as r:
    r.raise_for_status()
    with open("big.parquet", "wb") as f:
        for chunk in r.iter_content(chunk_size=1 << 20):   # 1 MiB windows
            f.write(chunk)

# Async equivalent with httpx — stream and consume as bytes arrive
import httpx

async def download(url):
    async with httpx.AsyncClient(timeout=30) as client:
        async with client.stream("GET", url) as r:
            r.raise_for_status()
            async for chunk in r.aiter_bytes(1 << 20):
                await sink.write(chunk)      # backpressure lives here, see below

__slots__: Shrink Millions of Instances

Every ordinary Python object carries a per-instance __dict__ to hold attributes — flexible, but ~100+ bytes of overhead each. When you have millions of small objects (rows, points, cache entries), __slots__ drops the dict and stores attributes in a fixed C array.

from pympler import asizeof   # pip install pympler — measures the FULL object graph

class PointDict:                # normal: has __dict__
    def __init__(self, x, y):
        self.x, self.y = x, y

class PointSlots:               # slotted: no __dict__
    __slots__ = ("x", "y")
    def __init__(self, x, y):
        self.x, self.y = x, y

print(asizeof.asizeof(PointDict(1, 2)))    # ~152 bytes
print(asizeof.asizeof(PointSlots(1, 2)))   # ~ 56 bytes

# 10M instances:  ~1.5 GB  vs  ~0.56 GB  — the slotted version fits, the other OOMs

Interning: Stop Storing Duplicate Small Objects

If you parse a million records with a status field of "active" / "pending", a naive parser creates a million distinct string objects with the same content. Interning collapses them to one shared object.

import sys

# ❌ each parsed value is its own object even when identical
statuses = [record["status"] for record in stream]   # 1M separate "active" strings

# ✅ intern: identical strings share one object in memory
statuses = [sys.intern(record["status"]) for record in stream]

# Same trick for any small hashable value set — a manual cache:
_cache = {}
def dedupe(value):
    return _cache.setdefault(value, value)   # first wins; the rest reuse it

Object Pools: Reuse Expensive Buffers

Allocating and freeing large buffers on every request churns the allocator and the GC. When the buffer is fixed-size and reusable (a parse scratch area, an I/O buffer), pool it: hand it out, clear it, hand it back.

import queue

# A bounded pool of reusable 1 MiB bytearrays — allocate N once, reuse forever
class BufferPool:
    def __init__(self, count=8, size=1 << 20):
        self._free = queue.Queue()
        for _ in range(count):
            self._free.put(bytearray(size))

    def acquire(self):
        return self._free.get()        # blocks if all buffers are checked out (backpressure!)

    def release(self, buf):
        buf[:] = b"\x00" * len(buf)    # or just reuse the bytes in place
        self._free.put(buf)

pool = BufferPool()

def handle(sock):
    buf = pool.acquire()
    try:
        n = sock.recv_into(buf)        # write straight into the reused buffer, zero new alloc
        process(memoryview(buf)[:n])
    finally:
        pool.release(buf)

Numeric Data: array, memoryview, NumPy — Not Lists of Ints

A Python int is a full heap object (~28 bytes) and a list stores 8-byte pointers to them. Ten million ints in a list ≈ 360 MB. The same values in a NumPy int64 array ≈ 80 MB — a flat C buffer, no per-element objects.

import sys, array
import numpy as np

n = 10_000_000

py_list = list(range(n))                 # list of int objects
arr     = array.array("q", range(n))     # stdlib: packed 64-bit ints, no numpy dep
np_arr  = np.arange(n, dtype=np.int64)   # numpy: packed + vectorized ops

print(sys.getsizeof(py_list))            # ~ 90 MB list + ~ 270 MB of int objects
print(np_arr.nbytes)                     # 80,000,000 bytes — exactly 8 bytes × 10M

# memoryview: slice/process a big buffer WITHOUT copying it
data = bytearray(open("frame.raw", "rb").read())
view = memoryview(data)
header = view[:64]          # a window into the same memory — zero-copy
body   = view[64:]          # no new allocation, no duplication
process(body)

The Garbage Collector Under Load

CPython frees most objects immediately via reference counting. The separate generational cyclic GC exists only to reclaim reference cycles (a.b = b; b.a = a). Under load it can pause your process at bad moments — and in preforked servers it can silently blow up memory via copy-on-write.

import gc

# 1) Preforked servers (gunicorn/uvicorn workers): freeze long-lived startup
#    objects so the child's GC never touches them → copy-on-write pages stay SHARED.
def post_fork_setup():
    gc.collect()      # clean up first
    gc.freeze()       # move everything alive now into a permanent, unscanned set

# 2) Tune thresholds if gen-0 collections fire too often on allocation-heavy work
gc.set_threshold(50_000, 100, 100)     # raise gen-0 trigger (default is 700)

# 3) Manual control for a short, latency-critical batch: disable, then re-enable
gc.disable()
try:
    crunch_numbers()        # no GC pauses during the hot path
finally:
    gc.enable()
    gc.collect()            # pay the cost once, deliberately, afterwards

Batch Processing: Bound the Peak

When you must transform a huge dataset, process it in fixed-size batches so peak memory is a constant you choose, not a function of input size. This is the memory face of batching and pairs with backpressure.

from itertools import islice

def chunked(iterable, size):
    it = iter(iterable)
    while batch := list(islice(it, size)):   # pull exactly `size` items, then stop
        yield batch

# Peak memory = one batch (1,000 rows), regardless of total row count
for batch in chunked(stream_rows(), size=1_000):
    enriched = enrich(batch)      # bulk-transform 1k at a time
    bulk_insert(enriched)         # bulk-write 1k at a time
    # `batch` and `enriched` go out of scope here → reclaimed before the next pull

Key Takeaways