Treat eight seconds and the target of “under one second” as hypothetical constraints. Before changing the architecture, clarify whether the target means time to first useful token, time to a complete answer, or time to finish every background action. Those are different service-level objectives and require different designs.
Our AI app takes 8 seconds to respond. How would you reduce latency?
Why the common answer is incomplete
Most candidates answer: “Use a faster model and stream the response.”
Both can improve perceived latency, but neither explains where the eight seconds are spent. Streaming does not reduce completion time; it only exposes partial output earlier. A faster model cannot help if authentication, retrieval, sequential tool calls, or overloaded queues dominate the request. Optimizing the wrong stage can increase cost and complexity without moving the user-visible metric.
Start with a latency budget for the complete request path.
A production approach
1. Trace every stage of the request
Create a distributed trace from edge receipt through authentication, orchestration, retrieval, reranking, prompt construction, provider connection, time to first token, generation, tool calls, and response flush. Record queue time separately from execution time. Compare median and tail latency because a healthy average can hide a p95 dominated by cold connections, large contexts, or a single slow dependency.
2. Define the user-visible milestone
For a conversational product, the first useful token may matter more than the final punctuation. For an agent that must execute a transaction, displaying speculative text before the operation completes may be misleading. Define milestones such as acknowledgement, first grounded answer, final answer, and completed side effect. Assign a budget to each so the team does not celebrate fast streaming while the actual task still takes eight seconds.
3. Remove unnecessary sequential work
Draw the dependency graph. Independent retrieval queries, authorization checks, and metadata lookups may run concurrently. Work that does not affect the initial answer can continue after the response or move to a durable background job. Preserve ordering where correctness requires it: do not execute a tool before validating its arguments, and do not parallelize steps that mutate the same state without a consistency strategy.
4. Reduce prompt and retrieval latency together
Large contexts increase both retrieval work and model prefill time. Limit candidate documents early, filter by tenant and metadata before vector ranking, deduplicate overlapping chunks, and rerank only a bounded set. Keep the final context large enough to answer the question. A smaller prompt that excludes the decisive document is fast but useless, so evaluate retrieval recall and answer quality alongside latency.
5. Route requests by workload
Small, well-defined requests may use a lower-latency model or a deterministic path. Complex analysis can use a stronger model with a different service-level objective. Routing features might include input length, required tools, risk level, language, or retrieval confidence. Maintain a fallback when the fast path has low confidence; otherwise latency improves by converting difficult requests into confidently wrong answers.
6. Cache stable intermediate results
Cache embeddings, document parsing, frequently requested retrieval results, and stable response fragments when authorization and freshness rules permit. Use request coalescing so a burst of identical cache misses performs one expensive computation rather than many. Cache keys must include tenant, document version, prompt version, and other correctness inputs. Track hit rate and stale-result incidents, not merely cache size.
7. Protect dependencies with deadlines
Pass a deadline through every downstream call. A retrieval service should not consume the entire user budget and leave no time for generation. Use short, dependency-specific timeouts, bounded retries, circuit breakers, and fallback behavior. Retries belong within the remaining deadline and should target only transient failures. Hedged requests can reduce tail latency for safe, idempotent reads, but they increase load and cost.
8. Stream an honest partial experience
Once server work can produce useful output, stream it through the entire path: provider to worker, worker to CDN, and CDN to browser. Flush meaningful chunks and surface tool progress without inventing certainty. If citations or safety validation finish later, label the intermediate state and prevent users from acting on an unverified answer. Streaming is a product contract, not just a transport option.
Engineering trade-offs
Parallelism reduces wall-clock time but increases concurrency, provider pressure, and coordination complexity. Caching lowers repeated latency but introduces staleness and authorization risks. Smaller models may respond faster but can lower task success. Aggressive timeouts protect the overall budget but turn slow successes into failures. Multi-region deployment reduces network distance while complicating data locality and consistency.
An “under one second” complete response may be impossible for long generation or dependent tools. The responsible answer is to separate first-response and completion targets, then redesign the experience around what the system can guarantee.
Common mistakes
- Measuring only end-to-end average latency without stage timing or percentiles.
- Treating streaming as a reduction in total compute time.
- Parallelizing dependent mutations and creating race conditions.
- Caching across tenants or document versions.
- Adding retries that exceed the original deadline.
- Choosing the fastest model on a small benchmark that excludes real prompt sizes.
- Optimizing generation while queue time or retrieval remains dominant.
Failure handling and observability
Publish service-level indicators for time to acknowledgement, time to first useful token, time to final answer, and successful task completion. Segment them by model, input size, feature, region, cache state, and dependency. Alert on p95 and p99 regressions, queue growth, timeout rate, provider connection time, token throughput, and streaming disconnects.
When a dependency misses its deadline, return a deliberate fallback: a cached answer with freshness metadata, a narrower result, or a message that the full task is continuing asynchronously. Use idempotency keys for background completion and make cancellation observable so abandoned browser requests do not keep consuming model and tool capacity unnoticed.
Interview insight
The interviewer is not testing whether you know a fast model. They are testing whether you can decompose latency, distinguish perception from completion, manage deadlines, and preserve correctness while changing concurrency. A strong answer turns “eight seconds” into a trace and a budget before proposing technology.