deskthebuildCACHING
← All notes · 5 min read

One hot Redis key is slowing down the entire service. How would you fix it?

Imagine a hypothetical service that caches a popular configuration object in Redis. A major event causes most requests to read the same key. Network throughput and one Redis shard become saturated even though overall cache memory is mostly empty. When the key expires, thousands of application instances miss together and query the database.

The problem is not insufficient cache capacity. It is concentrated traffic and synchronized refresh on one piece of data.

Why the common answer is incomplete

Most candidates answer: “Add more Redis nodes.”

Adding shards distributes different keys, but a single key normally maps to one shard. More nodes may leave the hot key exactly where it was. Read replicas can spread some traffic, yet stale reads, replica lag, and connection behavior must be considered. The application also continues to depend on one remote lookup for data that may change rarely.

A production response reduces demand on the hot key, controls refresh, and makes fallback behavior explicit.

A production approach

1. Prove that the key is hot

Measure command rate, bytes, latency, connection distribution, CPU, network usage, and per-key sampling where safe. Correlate the key with endpoints and tenants. Distinguish one hot read key from a large value, expensive command, hot hash field, or synchronized miss. Avoid enabling heavy key inspection on an already saturated production node without understanding its cost.

2. Add a bounded local cache

For small, widely shared, slowly changing data, cache the value inside each application instance for a short period. Local reads remove network hops and distribute load across process memory. Include version or expiration and cap memory. This works best when modest staleness is acceptable; security revocations and rapidly changing inventory need stronger invalidation.

3. Use request coalescing

When a local or remote cache miss occurs, allow one request per process or key to load the value while others await the same promise. Across processes, use a short lease or refresh marker carefully so one worker rebuilds the entry. Always bound the wait and provide a fallback. Coalescing prevents a cold key from turning into a database stampede.

4. Refresh before hard expiry

Use stale-while-revalidate or refresh-ahead for data where a slightly old value is safer than a mass miss. One worker refreshes near expiration while others continue serving the previous value within a bounded stale window. Add jitter to expiration times so related keys do not refresh simultaneously. Preserve a hard maximum age for correctness.

5. Reduce the payload and read frequency

Store only fields needed on the request path and avoid transferring a large serialized object for every request. Push stable configuration into deployment artifacts or a versioned local snapshot when updates are rare. If callers read the same key several times in one request, memoize it in request context. The cheapest remote cache call is the one the application no longer makes.

6. Replicate or shard reads only when necessary

Read replicas can distribute a hot read workload if the client balances correctly and the allowed staleness matches replication behavior. For an immutable large object, create replicated keys with deterministic client selection, but define update and invalidation across all copies. Manual key sharding adds operational complexity and is unsafe for counters or mutable state without a merge strategy.

7. Protect the source database

Rate-limit cache rebuilds and set a separate, small connection budget for fallback reads. If Redis fails, do not allow every application request to query the database. Serve a bounded stale local value, return a degraded response, or reject optional traffic. The cache should protect the database during normal operation and during its own partial outage.

8. Revisit whether Redis owns the right data

A cache is appropriate when low latency matters more than strict durability and the value can be reconstructed. A globally popular configuration may belong in a push-based configuration system or application memory. A high-contention counter may require partitioned increments and periodic aggregation. Choose the storage pattern from update semantics rather than forcing every fast lookup through Redis.

Engineering trade-offs

Local caching reduces Redis load and latency but increases staleness and per-process memory. Pub/sub invalidation speeds updates while messages can be lost, so expiration remains necessary. Replicas add read capacity and failover options while increasing lag and operational cost. Stale-while-revalidate improves availability but is unsuitable when old data creates a security or financial error.

Replicated hot keys spread traffic but make writes and invalidation harder. Precomputing small views simplifies requests while moving complexity to the update pipeline. The correct option depends on value size, change frequency, fan-out, and the cost of stale data.

Common mistakes

  • Adding cache shards without checking how the hot key is partitioned.
  • Giving every application instance permission to rebuild the value simultaneously.
  • Using the same fixed TTL for a large set of related keys.
  • Falling back to the database without a concurrency limit.
  • Caching authorization or revocation data longer than its safety boundary.
  • Replicating a mutable key without defining update consistency.
  • Treating Redis memory utilization as the only capacity signal.

Failure handling and observability

Track cache commands, network bytes, latency, hit rate, local-hit rate, refresh attempts, coalesced waiters, stale serves, source queries, and value size. Monitor per-shard skew and top-key samples with low-overhead tooling. Alert on refresh failure and database fallback concurrency before the cache fully expires.

If Redis is slow, open a circuit for optional reads and serve a recent local snapshot within its maximum age. If refresh fails, keep the old value only when product policy permits and expose degraded state. If the source database is also under pressure, reject or simplify requests rather than bypassing every cache layer. Recovery should ramp refresh concurrency gradually.

Interview insight

The interviewer is not testing whether you know how to add a Redis cluster node. They are testing whether you understand skew, cache stampedes, locality, staleness, and fallback protection. A staff-level answer reduces concentrated demand before adding infrastructure and prevents a cache failure from becoming a database failure.