deskthebuildSYSTEM DESIGN
← All notes · 5 min read

How would you enforce rate limits across a distributed system?

Suppose a hypothetical public API runs across several regions and hundreds of instances. Each instance enforces 100 requests per minute locally, so a client can exceed the intended global quota by spreading traffic across instances. A centralized counter fixes accuracy but becomes a latency and availability dependency for every request.

Distributed rate limiting is a resource-allocation problem. The design must define the protected resource, identity, time semantics, acceptable overshoot, and behavior when coordination is unavailable.

Why the common answer is incomplete

Most candidates answer: “Store a counter in Redis with an expiration.”

An atomic counter can enforce a simple fixed window, but window boundaries allow bursts, one global store adds network latency, and multi-region coordination may be unavailable during a partition. A counter also does not define fairness among tenants or distinguish cheap requests from operations that consume much more CPU, tokens, or downstream capacity.

A production limiter combines a clear quota model with bounded local enforcement and an explicit consistency trade-off.

A production approach

1. Define the quota and protected resource

Specify whether the limit applies per user, API key, tenant, IP, endpoint, model, or combination. Define the measured unit: requests, bytes, concurrent jobs, tokens, or weighted cost. State burst allowance and refill rate. Authentication-derived identities are stronger than user-supplied headers, while IP limits require proxy trust and careful treatment of shared networks.

2. Choose an algorithm from traffic semantics

A token bucket allows bounded bursts while enforcing an average refill rate. A leaky bucket smooths output. A sliding window provides intuitive recent-history limits at greater state cost. A fixed window is simple but permits boundary bursts. Use concurrency limits for long-running operations whose service time, not arrival count, consumes the scarce resource.

3. Enforce close to the work

Reject excess traffic at the edge or API gateway before it consumes application and database capacity. Apply a second limiter near especially scarce dependencies when request cost becomes known later. Return a stable status and retry guidance. Do not perform expensive authentication, body parsing, or model calls before a limit that could have rejected the request earlier.

4. Allocate local budgets from a global quota

Instead of synchronously incrementing one global counter for every request, lease a bounded number of tokens to regions or instances. Local checks are fast and continue during brief coordination failure. Refill before exhaustion and return unused capacity when practical. The lease size bounds possible overshoot: larger allocations reduce coordination while weakening global precision.

flowchart LR C[Client request] --> E[Edge identity and policy] E --> L{Local token available?} L -->|Yes| A[Admit request] L -->|No| R[Request quota lease] R --> G{Global budget available?} G -->|Yes| B[Grant bounded tokens] B --> L G -->|No| X[Return rate limit] A --> D[Protected dependency] D --> M[Usage reconciliation] M --> G

5. Preserve fairness and priority

Reserve capacity by tenant tier or workload class so one noisy client cannot consume all shared headroom. Add per-tenant limits beneath a system-wide protection limit. Interactive and background work may have different queues and budgets. Priority should come from authenticated product policy, not a client-provided value.

6. Handle variable request cost

Estimate cost before admission when possible, then reconcile with actual tokens, bytes, or compute. Reserve an upper bound for a long operation and release unused capacity. For streaming responses, enforce concurrent stream and generated-token limits. A request count alone lets one extremely expensive call consume the capacity intended for many small ones.

7. Define partition and dependency behavior

Choose fail-open, fail-closed, or bounded local continuation per endpoint. A public read may use cached local leases during a quota-store outage. An expensive write or abuse-sensitive action may fail closed. Cap emergency local allowance and log its use. An unavailable limiter should not create unlimited access to the resource it protects.

8. Version policies and test boundary behavior

Store limit, burst, scope, algorithm, and rollout version. Canary changes and watch rejection by tenant and endpoint. Test concurrent requests, clock skew, window boundaries, instance churn, regional partition, lease expiry, and client retries. Verify that response headers and backoff guidance do not synchronize every client into the same retry wave.

Engineering trade-offs

Strong global accuracy requires coordination and adds latency or reduces availability. Local token leases improve performance and resilience while allowing bounded overshoot. Smaller leases improve precision but increase coordinator traffic. Edge enforcement protects capacity early, while service-level enforcement has more accurate cost and identity context.

Fail-open favors availability but risks overload and abuse. Fail-closed protects scarce resources but can turn a limiter outage into a product outage. The choice should follow the cost of excess traffic for each operation, not one global preference.

Common mistakes

  • Applying the full global quota independently on every instance.
  • Choosing a fixed window without considering boundary bursts.
  • Trusting client-supplied tenant or priority headers.
  • Limiting request count while ignoring tokens, concurrency, or payload size.
  • Making every request synchronously depend on one remote counter.
  • Failing open without a bounded emergency allowance.
  • Returning retry guidance that causes synchronized client retries.

Failure handling and observability

Trace identity class, policy version, local tokens, lease ID, estimated cost, actual cost, admission decision, and rejection reason without logging secret API keys. Monitor allowed and rejected traffic, lease utilization, overshoot, coordinator latency, emergency allowance, fairness by tenant, retry volume, and protected-resource saturation.

If the quota coordinator fails, use the endpoint-specific policy and bounded local lease until expiry. If reconciliation detects overshoot, reduce future allocations rather than terminating accepted work unpredictably. If a tenant repeatedly exhausts capacity, expose stable quota metrics and upgrade or backoff guidance. During an attack, preserve management and recovery traffic through separate authenticated limits.

Interview insight

The interviewer is not testing whether you know an atomic Redis command. They are testing whether you understand global coordination, bounded inconsistency, fairness, and overload protection. A staff-level answer states how much overshoot is acceptable and ensures the limiter’s own failure cannot expose unlimited capacity.