Deadlines & Cancellation — Absolute Time That Propagates

Deadline, not Timeout

Timeout = a duration, measured fresh at each hop. Deadline = one absolute instant, shared by all hops. gRPC speaks deadlines. That difference is the whole point of this page.

Say the frontend allows 5s for a request. It calls service A, which calls B, which calls C. With per-hop timeouts, each hop starts its own 5s clock — so the total budget silently balloons to 15s+, and C may keep grinding on work the user abandoned 10 seconds ago.

With a deadline, the frontend stamps "finish by T = now + 5s" and sends it down. Every hop subtracts the time already spent. When T passes, everyone stops at once.

Propagation Across Hops

sequenceDiagram participant U as Frontend (budget 5s) participant A as Service A participant B as Service B U->>A: call, deadline = T (4.9s left) A->>B: call, deadline = T (4.2s left) Note over B: T passes while B works B-->>A: DEADLINE_EXCEEDED A-->>U: DEADLINE_EXCEEDED Note over U,B: all three abandon the work together

gRPC carries the remaining time in the grpc-timeout request header on each call. A well-behaved service passes the incoming context's deadline down to its own outbound calls, so the budget shrinks monotonically along the chain instead of resetting.

The rule: derive every downstream deadline from the one you were given. Never start a fresh clock mid-chain unless you deliberately want a sub-budget (e.g. a fast-fail cache lookup capped well under the parent).

Setting a Deadline (Client)

In most languages you pass a timeout at the call site and the library converts it to an absolute deadline on the wire:

import grpc

try:
    # 'timeout' seconds -> becomes an absolute deadline internally
    resp = stub.GetUser(req, timeout=0.5)
except grpc.RpcError as e:
    if e.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
        # the 500ms budget elapsed — do NOT blindly retry with the same budget
        log.warning("user service too slow")
        raise

Missing deadline = the silent killer. A call with no deadline can hang until the connection dies. Under load, hung calls pile up, exhaust the thread pool / connection limits, and turn a slow dependency into a full outage. Set a deadline on every RPC.

Honoring Deadlines & Cancellation (Server)

The deadline is only useful if the server actually stops. Long handlers should check the context and bail early — otherwise you burn CPU producing a response no one will read.

def ExportRows(self, request, context):
    for row in db.iter_rows(request.query):
        # client gave up, or the deadline passed? stop now.
        if not context.is_active():
            context.cancel()
            return
        yield to_proto(row)

Cancellation propagates too. When a client cancels (or its deadline fires), the server's context becomes inactive. Pass that same context into your downstream calls and the cancellation flows through the whole tree — nobody keeps working on an abandoned request.

Why Deadlines Beat Timeouts + Retries

Concern	Per-hop timeout + retry	Propagated deadline
Total latency bound	Multiplies down the chain	Fixed end-to-end
Wasted work	Deep services keep computing	Everyone stops together
Retry storms	Each layer retries → amplification	No budget left → no retry
User experience	Unbounded tail latency	Predictable p99 ceiling

Retries still have a place — but scope them to the remaining deadline. A retry that would blow the budget shouldn't fire. Combine deadlines with UNAVAILABLE-only retries and jittered backoff (see Status Codes) so a struggling dependency doesn't get hammered by its own callers.

Key Takeaways

Every RPC gets a deadline. No exceptions. A missing deadline is an unbounded hang waiting to happen.
Propagate, don't reset. Derive downstream deadlines from the incoming context.
Check is_active() in long-running handlers and streams so cancellation actually frees resources.
Deadline exceeded surfaces as DEADLINE_EXCEEDED — treat it as "over budget," not "retry immediately."