deskthebuildOBSERVABILITY
← All notes · 5 min read

Average latency looks healthy, but p99 is timing out. How would you find the cause?

Imagine a hypothetical API with a 120-millisecond average response time and a two-second p99. Most dashboards look healthy, but a meaningful group of users sees timeouts. Requests fan out to several services, share connection pools, and occasionally trigger cache misses or garbage collection.

Tail latency is where queueing, rare paths, resource contention, and dependency variance become visible. The average compresses those different populations into a reassuring number.

Why the common answer is incomplete

Most candidates answer: “Add caching and scale the service.”

Either change can help, but neither identifies which requests are slow or why. More instances may increase connection pressure on a shared database. A cache can improve the common path while making cold misses even more distinct. Without tracing latency by stage and request attributes, the team may optimize the median and leave the tail unchanged.

A production investigation isolates the slow cohort, finds where time is spent, and distinguishes service time from waiting time.

A production approach

1. Use distributions, not averaged percentiles

Record latency histograms at service boundaries and compute percentiles over meaningful windows. Do not average p99 values from separate instances or time buckets; that does not produce the fleet p99. Track request count alongside each percentile. Compare p50, p90, p95, p99, and maximum to understand whether the distribution has a long tail or several distinct modes.

2. Segment the slow cohort

Break latency down by endpoint, tenant tier, region, payload size, cache state, response code, deployment version, and dependency route. Add high-cardinality attributes to traces rather than metric labels when necessary. Ask whether slow requests are large, cold, cross-region, retried, or concentrated on one shard. A global p99 can mix unrelated workloads that need different fixes.

3. Trace queueing and execution separately

Measure time waiting for an application worker, connection pool, thread pool, rate limiter, queue, and downstream response. A database query that executes in 20 milliseconds after waiting 800 milliseconds for a connection is not a query-plan problem. Propagate deadlines and trace context across calls so spans show both work and contention.

4. Examine fan-out amplification

A request that waits for ten parallel dependencies is limited by the slowest one. Even moderately variable dependencies create a worse end-to-end tail as fan-out grows. Record child-call counts and completion distributions. Reduce unnecessary fan-out, batch requests, or return partial optional data. Do not hide a required dependency timeout by dropping correctness-critical results.

5. Look for coordinated resource pauses

Correlate slow traces with CPU throttling, garbage collection, event-loop delay, page faults, disk pauses, container restarts, and autoscaling events. Check whether connection reuse, DNS, TLS handshakes, or load-balancer queues appear only on a small subset. A periodic pause across many endpoints suggests shared runtime or infrastructure rather than one business query.

6. Validate timeout and retry budgets

Set an end-to-end deadline and allocate smaller budgets to dependencies. Cancel work when the caller no longer needs it. A retry should fit inside the remaining deadline and target a failure likely to recover. Otherwise retries increase load and turn a slow request into two slow requests. Track original attempts and retries as one user operation.

7. Use hedging only for safe, controlled reads

For idempotent reads with rare unpredictable slowness, a delayed duplicate to an independent replica can reduce tail latency. Start the hedge only after a percentile-based delay and cancel the loser. Cap hedge rate and ensure it does not hit the same saturated resource. Hedging writes or sending immediate duplicates doubles load and can create duplicate effects.

8. Reproduce with realistic load

Test production-like payloads, cache states, concurrency, and downstream latency. Avoid coordinated omission, where the load generator waits for a slow response and therefore sends fewer requests during the exact period of saturation. Verify the fix at the user-operation level and by segment. A lower p99 at half the real arrival rate is not evidence of production capacity.

Engineering trade-offs

Detailed traces improve diagnosis but increase telemetry cost and can capture sensitive attributes. Tail-based sampling retains slow and failed traces while requiring buffering and careful sampling policy. Lower timeouts reduce resource occupancy but may fail work that would have completed successfully. Larger pools reduce waiting until they overload the shared dependency.

Hedged reads trade extra capacity for lower tail latency. Caching reduces common work but introduces invalidation and cold paths. Reducing fan-out may simplify latency while returning less data. The appropriate change depends on which stage owns the tail and whether the data is required.

Common mistakes

  • Averaging p99 values across hosts or time windows.
  • Investigating only slow execution and ignoring pool or queue wait.
  • Adding application instances without checking shared dependency saturation.
  • Retrying after the end-to-end deadline is already exhausted.
  • Hedging requests to the same overloaded shard.
  • Load-testing with a client that reduces arrival rate during slow responses.
  • Declaring success from global percentiles while one tenant or region still times out.

Failure handling and observability

Keep end-to-end request IDs and trace queue wait, execution, dependency attempts, cancellations, and retries. Monitor latency distributions, in-flight work, pool utilization, queue age, runtime pauses, dependency tails, and timeout causes. Use tail-based sampling for slow requests plus a representative baseline of normal traffic.

When a dependency degrades, stop optional fan-out, cap concurrency, and fail within the caller’s deadline. Cancel abandoned work so it does not continue consuming capacity after a timeout. If the system is saturated, shed requests before queues become unbounded. During recovery, watch tail latency and queue depth together; the average can improve while old queued requests still violate deadlines.

Interview insight

The interviewer is not testing whether you know the definition of p99. They are testing whether you can reason about distributions, queueing, fan-out, and coordinated failure. A staff-level answer isolates the slow population, traces waiting separately from work, and verifies improvements under the arrival pattern that creates the tail.