🛡️ Resilience Patterns Under Load

The Core Problem: One Slow Dependency Takes Everything Down

Timeouts Everywhere — the #1 Fix

A missing timeout is the single most common cause of cascading hangs. The default for most clients is "wait forever," which is exactly what you must never do under load. Every network call gets a per-request timeout — no exceptions.

In Threading — requests / httpx

import requests, httpx

# requests: ALWAYS pass a timeout. Without it, requests waits indefinitely.
# Prefer a (connect, read) tuple so a slow body can't hang forever.
requests.get(url, timeout=(3.05, 10))   # 3.05s to connect, 10s to read

# httpx: granular, and you can set a default on the client:
client = httpx.Client(timeout=httpx.Timeout(10.0, connect=3.0))
client.get(url)

In Asyncio — asyncio.timeout() (3.11+)

import asyncio, httpx

async def fetch(client, url):
    # Wraps ANY awaitable in a deadline — cleaner than wait_for, cancels on exit.
    async with asyncio.timeout(5):          # Python 3.11+
        return await client.get(url)

# Pre-3.11 equivalent:
async def fetch_old(client, url):
    return await asyncio.wait_for(client.get(url), timeout=5)

In a Thread Pool — future.result(timeout=)

from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout

with ThreadPoolExecutor(max_workers=10) as pool:
    fut = pool.submit(requests.get, url, timeout=10)
    try:
        resp = fut.result(timeout=12)   # caller-side deadline (slightly > the request timeout)
    except FutureTimeout:
        resp = None   # NOTE: the worker thread keeps running — the underlying call MUST
                      # also have its own timeout, or you leak a stuck thread.

Retries with Exponential Backoff + Jitter

Transient failures (a dropped connection, a brief 503) deserve a retry. But naive retries are dangerous: a fixed delay makes every client retry in lockstep, producing synchronized traffic spikes that keep a recovering service down. This is a retry storm.

import time, random

def call_with_backoff(fn, *, max_attempts=5, base=0.2, cap=10.0):
    """Exponential backoff with FULL jitter: sleep = random(0, min(cap, base * 2**attempt))."""
    for attempt in range(max_attempts):
        try:
            return fn()
        except (ConnectionError, TimeoutError) as exc:   # only RETRYABLE errors
            if attempt == max_attempts - 1:
                raise
            expo = min(cap, base * (2 ** attempt))        # 0.2, 0.4, 0.8, 1.6, ...
            sleep = random.uniform(0, expo)               # FULL jitter spreads the herd
            time.sleep(sleep)

# Decorrelated jitter (AWS): each sleep grows off the PREVIOUS one, bounded by cap.
def decorrelated_sleep(prev, base=0.2, cap=10.0):
    return min(cap, random.uniform(base, prev * 3))

In Asyncio — non-blocking backoff

import asyncio, random

async def call_with_backoff_async(coro_factory, *, max_attempts=5, base=0.2, cap=10.0):
    for attempt in range(max_attempts):
        try:
            return await coro_factory()          # a fresh awaitable each attempt
        except (ConnectionError, TimeoutError, asyncio.TimeoutError):
            if attempt == max_attempts - 1:
                raise
            expo = min(cap, base * (2 ** attempt))
            await asyncio.sleep(random.uniform(0, expo))   # await, don't block the loop

Circuit Breaker — Stop Hammering What's Already Down

Retries help with transient failures. But when a dependency is genuinely down, retrying just wastes resources and slows recovery. A circuit breaker tracks the failure rate and, once it crosses a threshold, trips open — failing fast without even attempting the call — then periodically probes to see if the dependency has recovered.

import time, threading

class CircuitOpenError(Exception):
    pass

class CircuitBreaker:
    """Thread-safe closed -> open -> half-open breaker."""
    def __init__(self, fail_threshold=5, cooldown=30.0):
        self._fail_threshold = fail_threshold
        self._cooldown = cooldown
        self._failures = 0
        self._state = "closed"          # closed | open | half_open
        self._opened_at = 0.0
        self._lock = threading.Lock()

    def call(self, fn, *args, **kwargs):
        with self._lock:
            if self._state == "open":
                if time.monotonic() - self._opened_at >= self._cooldown:
                    self._state = "half_open"      # time to probe
                else:
                    raise CircuitOpenError("circuit open — failing fast")
        try:
            result = fn(*args, **kwargs)
        except Exception:
            self._on_failure()
            raise
        else:
            self._on_success()
            return result

    def _on_success(self):
        with self._lock:                # a good call closes the circuit and resets
            self._failures = 0
            self._state = "closed"

    def _on_failure(self):
        with self._lock:
            self._failures += 1
            # a failed probe in half_open, or crossing the threshold, trips it open
            if self._state == "half_open" or self._failures >= self._fail_threshold:
                self._state = "open"
                self._opened_at = time.monotonic()

# Usage:
breaker = CircuitBreaker(fail_threshold=5, cooldown=30)
try:
    resp = breaker.call(requests.get, url, timeout=5)
except CircuitOpenError:
    resp = get_from_cache_or_default()   # degrade gracefully instead of hanging

Load Shedding & Deadline Propagation

Timeouts, retries, and breakers protect you from downstream failure. Load shedding protects you from your own overload: when you're already saturated, accepting more work makes everything slower and helps no one. Reject early — return 503 immediately when the queue is full — so the requests you do accept get served fast.

import threading

class Overloaded(Exception):
    pass

# Admission control: bound in-flight work; reject (503) the moment we're full.
class LoadShedder:
    def __init__(self, max_in_flight):
        self._sem = threading.BoundedSemaphore(max_in_flight)

    def __enter__(self):
        if not self._sem.acquire(blocking=False):   # DON'T queue — reject now
            raise Overloaded("shedding load: at capacity")
        return self

    def __exit__(self, *exc):
        self._sem.release()

shedder = LoadShedder(max_in_flight=200)

def handle(request):
    try:
        with shedder:
            return process(request)
    except Overloaded:
        return Response(status=503, headers={"Retry-After": "1"})   # fail fast, tell client to back off

import time

def with_deadline(deadline_monotonic, fn, *args, **kwargs):
    remaining = deadline_monotonic - time.monotonic()
    if remaining <= 0:
        raise Overloaded("deadline exceeded before call — shedding")
    # Never let a downstream call outlive the caller's deadline:
    return fn(*args, timeout=remaining, **kwargs)

Failure Mode → Pattern

Combine Them — They Only Work Together

No single pattern is sufficient. A timeout without a breaker still lets you hammer a dead service every request. A breaker without timeouts never trips because calls hang forever. Retries without jitter cause storms. Layer them, innermost to outermost:

# Order of composition (inside -> out):
#   1. timeout      — bound each individual attempt
#   2. retry+jitter — recover transient failures (idempotent only)
#   3. breaker      — after repeated failure, stop trying entirely
#   4. shed/deadline— at the edge, reject when overloaded

breaker = CircuitBreaker(fail_threshold=5, cooldown=30)

def resilient_get(url, deadline):
    def attempt():
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise Overloaded("deadline exceeded")
        return breaker.call(requests.get, url, timeout=min(remaining, 5))  # timeout + breaker

    return call_with_backoff(attempt, max_attempts=3)                      # + retry/jitter