deskthebuildLLM SYSTEMS
← All notes · 6 min read

How would you route requests between small and frontier models safely?

Consider a hypothetical product that sends every request to its most capable model. Quality is acceptable, but cost and capacity limit growth. The team wants simple work to use a smaller model while reserving the frontier model for ambiguous, high-risk, or reasoning-heavy tasks.

This sounds like a classifier problem, but routing changes the system’s failure modes. A routing mistake can save a few cents while producing an incorrect answer, or send easy traffic to the expensive path and erase the intended savings.

Why the common answer is incomplete

Most candidates answer: “Use another LLM to classify whether the request is easy or hard.”

That creates a routing cost and latency tax on every request, and “difficulty” is not a stable label. A short request can be high risk, while a long request may be deterministic. The router also cannot know from the input alone whether retrieval will find strong evidence or whether a tool call will fail.

The decision should be based on product risk, measurable task requirements, and the ability to escalate when the cheaper path is uncertain.

A production approach

1. Define routing objectives by workflow

List each product workflow and its acceptable quality, latency, and cost boundaries. Extracting a date from a controlled form is different from generating a customer-facing policy explanation. Mark workflows that always require the stronger model because failures are expensive, regulated, irreversible, or difficult to detect. Routing should optimize within an explicit service objective, not apply one global “easy versus hard” rule.

2. Build a single-model baseline first

Measure the current model on a representative evaluation set before adding a router. Record task success, groundedness, refusal quality, latency, token use, and cost by workflow and input segment. The baseline establishes what the routed system must preserve. Without it, the team may report cost savings while silently accepting regressions that were never measured.

3. Start with deterministic routing signals

Use authenticated product context, workflow type, requested operation, input size, language support, and risk tier before adding learned classification. Deterministic rules are explainable and cheap. For example, a read-only summarization of a short internal note may start small, while an external write or a request involving sensitive policy always uses the stronger path. Do not infer authorization or business risk from model-generated text.

4. Use a cascade when correctness can be checked

Let the smaller model attempt suitable tasks, then escalate when validation fails or confidence is insufficient. Useful gates include schema validation, citation coverage, retrieval strength, policy checks, and domain-specific invariants. Self-reported model confidence is weak evidence by itself; combine it with signals the application can verify. The cascade must cap attempts so escalation does not become an uncontrolled retry loop.

flowchart LR A[Authenticated request] --> B{Risk policy} B -->|High risk| F[Frontier model] B -->|Eligible| S[Small model] S --> V{Deterministic checks pass?} V -->|Yes| R[Return result] V -->|No| F F --> G{Final checks pass?} G -->|Yes| R G -->|No| H[Safe fallback or human review]

5. Version the routing policy as product code

Store the policy version with every trace, along with selected model, route reason, validation results, and escalation path. Keep model identifiers, prompt versions, tool sets, and decoding parameters explicit. A routing decision cannot be reproduced if “small model” is an alias that changes silently or if the prompt differs between evaluation and production.

6. Design provider and capacity fallbacks separately

Quality routing and outage routing solve different problems. A provider fallback may have different context limits, tool semantics, safety behavior, or regional availability. Test fallback prompts and tool schemas rather than assuming model compatibility. During capacity pressure, preserve high-risk traffic, queue delay-tolerant work, and reject excess low-priority requests with a clear retry signal instead of routing everything to an untested provider.

7. Evaluate route errors, not only final averages

Track false-cheap routes where the small model should have escalated and false-expensive routes where the frontier model added no value. Segment results by language, document length, tenant, retrieval quality, and workflow. Compare the routed result with the baseline and with human or deterministic checks where available. A good aggregate score can hide a router that consistently underserves one customer segment.

8. Roll out with shadow decisions and canaries

First compute the proposed route without changing the serving model. This reveals expected traffic distribution and capacity needs. Then enable routing for low-risk workflows and a small traffic percentage. Set automatic rollback conditions for task success, escalation rate, latency, and cost per successful task. Expand only when the observed savings survive retries, escalations, and operational overhead.

Engineering trade-offs

Deterministic rules are predictable but may leave savings on the table. A learned router can capture subtle patterns but needs labels, monitoring, and retraining. Cascades protect quality but add latency on escalated requests and can cost more than calling the stronger model once. Multiple providers improve resilience while increasing evaluation work and semantic differences across tool calling, safety, and context handling.

Model routing also creates organizational complexity. Support teams need to know which path produced an answer. Evaluation datasets must cover every route. Capacity planning must account for correlated escalation during difficult traffic, when the frontier model may receive a larger share precisely as the system is under stress.

Common mistakes

  • Routing solely from prompt length or a model’s self-reported confidence.
  • Sending high-risk actions to a cheaper model because the language looks simple.
  • Measuring cost per request instead of cost per successful task.
  • Ignoring the latency and cost of the routing classifier.
  • Treating provider models as interchangeable without fallback evaluations.
  • Allowing unlimited small-model retries before escalation.
  • Changing model aliases without recording the resolved version in traces.

Failure handling and observability

Log the workflow, risk tier, policy version, routing signals, selected model, validation failures, escalation reason, and final outcome. Monitor route share, false-cheap and false-expensive rates, escalation frequency, cost per successful task, provider errors, and capacity by model. Alert when route distribution changes without a product release; this may indicate traffic drift, a broken classifier, or degraded retrieval.

When the router is unavailable, use a documented safe default by workflow. For high-risk paths, that may mean the stronger model or human review. For low-risk background work, it may mean queueing until capacity returns. If validators disagree or required evidence is missing, do not guess based on cost pressure. Escalate once, then return a bounded failure if the stronger path also fails.

Interview insight

The interviewer is not testing whether you can name a cheap model and an expensive model. They are testing whether you can turn quality, risk, and cost into an observable policy with safe escalation. A mature routing system saves money only after it proves that the product outcome remains within its required boundary.