Imagine a hypothetical event pipeline where consumer lag grows rapidly during peak traffic and takes hours to recover. The team adds more consumer instances, but some remain idle while a few partitions continue falling behind. Downstream API errors and database latency rise at the same time.
Kafka consumer lag keeps growing during traffic spikes. How would you debug it?
Lag is a symptom: records arrive faster than the effective consumer path can complete them. The constraint may be partitions, skew, processing time, downstream capacity, rebalances, or retry behavior.
Why the common answer is incomplete
Most candidates answer: “Add more consumers.”
A consumer group cannot process one partition with several active consumers at the same time. If the topic has too few partitions or one key dominates a partition, additional instances wait idle. More consumers can also overload the database or external service that caused processing to slow. Scaling the caller without protecting the dependency may increase failure and retries.
A production investigation measures arrival rate, service rate, partition distribution, and every stage of processing before changing capacity.
A production approach
1. Distinguish lag from processing latency
Track record lag by partition, oldest unprocessed event age, end-to-end event latency, and consumer processing time. A high record count can be acceptable for tiny events processed quickly, while a small backlog containing hour-old events violates the product objective. Use broker offsets and event timestamps carefully; producer clock skew should not be the only latency source.
2. Compare arrival and completion rates
Measure records and bytes produced per second against successfully completed records per second. Estimate required concurrency from processing time and peak arrival rate, then validate with real workload distribution. If completion remains below arrival, lag must grow. Recovery also needs spare capacity above normal traffic; matching the peak rate exactly never drains the backlog.
3. Inspect partition skew
Break lag, throughput, and processing time down by partition and message key. One large tenant, customer, or entity can create a hot partition while group-level averages look reasonable. Check whether the key is required for ordering. If order is needed only per account rather than globally, a better key or more granular ownership may distribute load without breaking semantics.
4. Decompose processing time
Trace deserialization, validation, business logic, database transactions, cache calls, external APIs, and acknowledgement. Separate active CPU from waiting on connections or network. A consumer that spends most of its time waiting for a database pool will not improve by adding application threads. Optimize or isolate the slow dependency and bound its concurrency.
5. Control retries and poison records
Classify transient, permanent, and data-quality failures. Apply bounded retries with backoff for recoverable errors and move unrecoverable records to a quarantined path with context for review. Retrying one bad event indefinitely can block an ordered partition. Preserve the original event ID and ordering implications when replaying repaired records.
6. Tune polling, batching, and commits together
Batching can improve throughput by amortizing network and transaction cost, while increasing per-record latency and the amount replayed after a crash. Ensure maximum processing time stays within consumer liveness settings, or the broker may revoke partitions during a long batch. Commit progress only after durable business effects, and make handlers idempotent because redelivery remains possible.
7. Reduce rebalance disruption
Monitor group membership, assignment changes, heartbeat failures, deployment churn, and processing pauses. Use graceful shutdown so consumers stop polling, finish or checkpoint bounded work, and release partitions cleanly. Cooperative assignment can reduce movement where supported. Frequent autoscaling may cost more capacity through rebalances than it adds through new workers.
8. Add partitions only with a migration plan
If processing is parallelizable and partition count is the proven ceiling, add partitions and ensure consumers can scale to use them. Understand that changing partition count alters key-to-partition mapping and can affect ordering for new events. Existing hot historical partitions remain hot until their backlog drains. Plan producer changes, state stores, and replay behavior before treating repartitioning as an incident switch.
Engineering trade-offs
More partitions increase parallelism while adding broker metadata, file handles, consumer coordination, and ordering complexity. Larger batches improve throughput but increase latency and replay size. Per-record transactions simplify failure isolation but can overload the database. Bulk writes improve efficiency while requiring partial-failure handling.
Backpressure protects downstream services but intentionally increases broker lag. That can be the correct behavior if the broker is the durable buffer and retention is sufficient. The key is keeping backlog age within the agreed recovery window and ensuring storage will not run out before capacity returns.
Common mistakes
- Looking only at group-wide lag instead of partition skew and event age.
- Adding consumers beyond the partition count.
- Scaling consumers when the downstream database is already saturated.
- Retrying a permanent validation failure in the ordered processing path.
- Increasing batch size without updating liveness and timeout settings.
- Committing offsets before durable business effects complete.
- Repartitioning without considering per-key ordering and state.
Failure handling and observability
Monitor production and completion rates, bytes, record lag, oldest-event age, processing-time histograms, partition skew, retry classes, dead-letter age, commit failures, rebalances, and dependency saturation. Trace event ID, topic, partition, offset, attempt, handler version, and downstream transaction without logging sensitive payloads.
When a dependency slows, cap consumer concurrency and let the durable broker absorb pressure within retention limits. Pause non-critical topics or tenants before they consume all recovery capacity. If retention risk approaches, increase safe downstream capacity, archive raw events, or shed eligible low-value work through an explicit policy. After recovery, drain backlog gradually so replay traffic does not recreate the incident.
Interview insight
The interviewer is not testing whether you know how to increase a consumer replica count. They are testing whether you understand queueing, partition ownership, ordering, backpressure, and downstream bottlenecks. A staff-level answer explains why lag grows, where parallelism stops helping, and how the system recovers without corrupting effects.