Imagine a hypothetical AI product whose provider begins returning rate-limit responses during a traffic spike. Application instances retry independently, queues grow, users refresh requests, and background jobs compete with interactive traffic. The original capacity constraint turns into a system-wide incident.
A provider rate-limit incident is cascading through our AI product. What would you change?
Rate limits are not unusual provider behavior. They are a resource boundary. The outage occurs because the application reacts to that boundary without coordinated admission control or a defined degradation policy.
Why the common answer is incomplete
Most candidates answer: “Add exponential backoff and retry.”
Backoff is necessary for some transient failures, but uncoordinated retries can multiply demand when the provider has the least available capacity. A retry also consumes the user’s latency budget and may repeat billable or side-effecting work. If every server waits and retries independently, the fleet has no shared view of remaining capacity or request priority.
A production system must prevent excess work from entering the constrained path, reserve capacity for important traffic, and make every retry bounded and observable.
A production approach
1. Model the actual limiting dimensions
Providers may limit requests, input tokens, output tokens, concurrent streams, or a combination over different time windows. Record resolved model, tenant, region, token estimate, attempt number, and returned limit headers. Capacity planning based only on requests per second fails when a small number of long prompts consumes most token or concurrency capacity.
2. Put admission control before the provider call
Use a shared limiter or scheduler to decide whether work can begin. Reserve estimated capacity before sending the request, then reconcile against actual usage. Reject or queue work that cannot fit within its deadline. Admission control turns a provider error into an application decision and prevents thousands of requests from discovering the same limit independently.
3. Separate traffic by priority and deadline
Interactive requests, paid-tier workflows, evaluation jobs, and background enrichment should not share one undifferentiated queue. Define priority based on product commitments rather than which request arrived first. Pause delay-tolerant work during pressure and protect a bounded share for interactive traffic. Include per-tenant fairness so one customer cannot consume the recovery capacity.
4. Retry only when the error can recover
Honor provider retry guidance when available, use exponential backoff with jitter, and cap attempts by both count and total deadline. Do not retry invalid input, authentication failures, policy rejections, or requests that already exceeded the user’s latency budget. Carry an idempotency key across attempts so repeated execution cannot duplicate an external write or create multiple workflow records.
5. Control concurrency independently from throughput
Even when average token throughput is below the documented quota, too many long-lived streams can exhaust connections, memory, or provider concurrency. Apply a semaphore per model and workload class. Release it on cancellation and timeout, not only successful completion. Tune concurrency from observed service time and provider behavior rather than setting it equal to the number of application workers.
6. Define degradation paths before the incident
For low-risk requests, reduce optional context, choose a validated smaller model, or return a result without background enrichment. Queue asynchronous work with a visible completion state. For high-risk tasks, preserve the stronger model and reject quickly when it is unavailable rather than producing an unchecked substitute. The user should know whether the request is delayed, reduced in capability, or not accepted.
7. Treat multi-provider fallback as a separate product path
Another provider may improve resilience, but its models differ in prompt behavior, tool schemas, safety rules, context limits, and regional data handling. Maintain provider-specific adapters and evaluation results. Route only workflows that passed those checks. A fallback that returns syntactically valid but semantically different tool calls can extend an incident into data corruption.
8. Test recovery under coordinated load
Load tests should include burst traffic, long prompts, slow streams, provider timeouts, partial responses, user cancellations, and changing quota headers. Verify queue delay, fairness, retry amplification, and recovery time after capacity returns. A system can look stable at a fixed rate while oscillating badly when delayed retries arrive in synchronized waves.
Engineering trade-offs
Rejecting early protects system health but creates visible errors. Queueing improves acceptance while increasing time-to-result and requiring durable job state. Reserving capacity for priority traffic may reduce overall utilization. A secondary provider improves availability but doubles portions of the evaluation, security, and operational surface.
Token estimation is imperfect, especially when tool calls or output length vary. Reserving too much wastes capacity; reserving too little produces limit errors. Reconciliation and adaptive estimates improve utilization but add state and feedback loops that must themselves remain stable during an incident.
Common mistakes
- Retrying every provider error with the same policy.
- Running retries inside each application instance without shared admission control.
- Letting background evaluation consume the same recovery capacity as user traffic.
- Ignoring token and concurrency limits because request rate looks normal.
- Queueing work without a deadline, cancellation path, or maximum backlog.
- Switching providers without testing prompts and tool contracts.
- Returning HTTP success while silently dropping optional but correctness-critical context.
Failure handling and observability
Track admitted, queued, rejected, attempted, retried, cancelled, and completed requests separately. Monitor estimated and actual tokens, concurrency, queue age, retry-after values, provider error classes, per-tenant fairness, and cost per successful task. Alert on retry amplification and on backlog age before queue length alone; a modest queue can still violate every interactive deadline.
When capacity falls, shed work in the documented order and expose a stable error code with retry guidance. Stop accepting new background jobs before their queue consumes storage. If a provider response is ambiguous after a timeout, reconcile using the application request identifier before retrying a side effect. During recovery, ramp concurrency gradually so the backlog does not immediately recreate the limit.
Interview insight
The interviewer is not testing whether you know exponential backoff. They are testing whether you understand admission control, backpressure, fairness, idempotency, and graceful degradation. A staff-level design keeps a provider limit from becoming an uncontrolled retry storm across the rest of the product.