🎫 Rate Limiting Under Load

Rate Limit ≠ Concurrency Limit

These two get conflated constantly, and mixing them up is how you either starve a fast downstream or flood a slow one. They bound different things.

• Rate limit — how many requests per unit of time (e.g. 100 req/sec). Bounds the flow.
• Concurrency limit — how many requests are in flight at once (e.g. 20 simultaneous). Bounds the depth. See concurrency caps.

The Problem: A Slow-Motion Self-DDoS

The Two Classic Algorithms

Almost every limiter in the wild is one of these two. The difference is entirely about burst tolerance.

Token Bucket — allows bursts

A bucket holds up to B tokens and refills at rate R tokens/sec. Each request spends one token. If the bucket is full (idle period), you can spend all B at once — a burst. Once drained, you're throttled to the steady refill rate R.

Leaky Bucket — smooths to a constant drain

Requests pour into a queue (the bucket) and leak out at a fixed rate. Output is perfectly smooth — no bursts ever reach the downstream. If the bucket overflows, requests are dropped. Think of it as a shock absorber that converts spiky input into a flat, constant output stream.

In Threading — a thread-safe token bucket

The canonical implementation: refill lazily on each acquire using a monotonic clock (never time.time() — it can jump backwards on NTP corrections), and do all mutation under a Lock. Callers can either block until a token frees up, or fail fast and reject.

import time
import threading

class TokenBucket:
    def __init__(self, rate: float, capacity: float):
        self.rate = rate                # tokens added per second
        self.capacity = capacity        # max tokens (== max burst)
        self._tokens = capacity         # start full
        self._last = time.monotonic()   # monotonic: immune to clock jumps
        self._lock = threading.Lock()

    def _refill(self) -> None:
        now = time.monotonic()
        elapsed = now - self._last
        # add tokens for the time that passed, clamp to capacity
        self._tokens = min(self.capacity, self._tokens + elapsed * self.rate)
        self._last = now

    def try_acquire(self, n: int = 1) -> bool:
        """Non-blocking: spend n tokens if available, else reject (→ 429)."""
        with self._lock:
            self._refill()
            if self._tokens >= n:
                self._tokens -= n
                return True
            return False

    def acquire(self, n: int = 1) -> None:
        """Blocking: sleep until n tokens are available, then spend them."""
        while True:
            with self._lock:
                self._refill()
                if self._tokens >= n:
                    self._tokens -= n
                    return
                # how long until we'll have enough?
                deficit = n - self._tokens
                wait = deficit / self.rate
            time.sleep(wait)   # sleep OUTSIDE the lock so other threads can refill/check

# Usage from a thread pool: shared limiter, 100 req/sec, burst of 100
from concurrent.futures import ThreadPoolExecutor

limiter = TokenBucket(rate=100, capacity=100)

def call_api(url):
    limiter.acquire()            # blocks here until the bucket allows it
    return session.get(url, timeout=5).status_code

with ThreadPoolExecutor(max_workers=20) as pool:
    results = list(pool.map(call_api, urls))   # paced to ~100/sec, absorbs bursts

In Asyncio — an async token-bucket limiter

Same math, but await asyncio.sleep() instead of blocking — the event loop stays free to run thousands of other coroutines while one waits for a token. Wrap it around a gather to pace fan-out.

import asyncio
import time

class AsyncTokenBucket:
    def __init__(self, rate: float, capacity: float):
        self.rate = rate
        self.capacity = capacity
        self._tokens = capacity
        self._last = time.monotonic()
        self._lock = asyncio.Lock()     # protects the refill math across coroutines

    async def acquire(self, n: int = 1) -> None:
        while True:
            async with self._lock:
                now = time.monotonic()
                self._tokens = min(self.capacity,
                                   self._tokens + (now - self._last) * self.rate)
                self._last = now
                if self._tokens >= n:
                    self._tokens -= n
                    return
                wait = (n - self._tokens) / self.rate
            await asyncio.sleep(wait)   # yields to the loop — nothing else blocks

import httpx

async def main(urls):
    limiter = AsyncTokenBucket(rate=100, capacity=100)
    async with httpx.AsyncClient(timeout=5) as client:
        async def fetch(url):
            await limiter.acquire()          # pace every call through the bucket
            r = await client.get(url)
            return r.status_code
        # 5,000 coroutines, but they leave at ≤100/sec — no 429 storm
        return await asyncio.gather(*(fetch(u) for u in urls))

asyncio.run(main(urls))

In Multiprocessing — per-process buckets don't coordinate

Here's the trap: if each of your 4 worker processes owns its own TokenBucket(rate=100), your service actually sends 400 req/sec. Buckets are process-local state; they have no idea the others exist. You need a shared, centralized limiter.

# A crude shared bucket via multiprocessing.Manager (single machine only).
# The Manager hosts the state in one process; workers RPC into it under a lock.
import time
from multiprocessing import Manager, Process

def make_shared_bucket(mgr, rate, capacity):
    state = mgr.dict(tokens=capacity, last=time.monotonic())
    lock = mgr.Lock()
    return state, lock, rate, capacity

def acquire(state, lock, rate, capacity, n=1):
    while True:
        with lock:
            now = time.monotonic()
            state["tokens"] = min(capacity,
                                  state["tokens"] + (now - state["last"]) * rate)
            state["last"] = now
            if state["tokens"] >= n:
                state["tokens"] -= n
                return
            wait = (n - state["tokens"]) / rate
        time.sleep(wait)

Client-Side vs Server-Side Limiting

Distributed Rate Limiting — one bucket for the whole fleet

With N stateless pods behind a load balancer, the rate limit must live outside any single process. Redis is the standard home for it: one shared bucket, mutated atomically. Do the refill + spend in a single Lua script so the check-and-decrement can't race across pods.

import redis, time

r = redis.Redis()

# Atomic token bucket in Redis. KEYS[1]=bucket key.
# ARGV: rate, capacity, now(seconds), requested tokens.
# Returns 1 if allowed, 0 if throttled.
TOKEN_BUCKET_LUA = """
local key      = KEYS[1]
local rate     = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local now      = tonumber(ARGV[3])
local want     = tonumber(ARGV[4])

local data   = redis.call('HMGET', key, 'tokens', 'last')
local tokens = tonumber(data[1]) or capacity
local last   = tonumber(data[2]) or now

tokens = math.min(capacity, tokens + (now - last) * rate)  -- lazy refill
local allowed = 0
if tokens >= want then
    tokens = tokens - want
    allowed = 1
end
redis.call('HMSET', key, 'tokens', tokens, 'last', now)
redis.call('EXPIRE', key, math.ceil(capacity / rate) + 1)   -- self-cleaning
return allowed
"""

_take = r.register_script(TOKEN_BUCKET_LUA)

def allow(key: str, rate=100, capacity=100, want=1) -> bool:
    # every pod calls this against the SAME key → one true fleet-wide limit
    return bool(_take(keys=[key], args=[rate, capacity, time.time(), want]))

# Per-user (or per-API-key) limiting is just a key prefix:
def allow_user(user_id: str) -> bool:
    return allow(f"ratelimit:{user_id}", rate=10, capacity=20)

if not allow_user(request.user_id):
    raise HTTP429(retry_after=1)   # tell the caller to back off

Key Takeaways