Suppose a hypothetical service operates in two regions. During a network incident, automation promotes the secondary region while the original primary is still accepting a small amount of traffic. Both regions process writes. When connectivity returns, records conflict and several side effects have happened twice.
A multi-region failover caused inconsistent data. How would you redesign recovery?
Multi-region availability is not achieved by copying data and switching DNS. The design must define who can write, how promotion is fenced, which data can be lost, and how divergent effects are reconciled.
Why the common answer is incomplete
Most candidates answer: “Use active-active replication so either region can serve traffic.”
Active-active can reduce failover time, but it moves conflict resolution into every data model and side effect. Network partitions make it impossible to provide unrestricted writes in both regions while also guaranteeing immediate consistency. If the product has not defined merge semantics, active-active turns an outage into silent business inconsistency.
A production design begins with recovery objectives and chooses ownership and consistency per workload.
A production approach
1. Define RTO, RPO, and failure scope
Set a recovery time objective for how long the service may be unavailable and a recovery point objective for how much committed data may be lost. Define whether the design covers one availability zone, a whole region, provider control-plane failure, or network partition. Different components can have different objectives; the login page and financial ledger do not need the same topology.
2. Classify data by write semantics
Separate strongly ordered business state, commutative counters, append-only events, caches, and reconstructable indexes. A shopping cart may merge items, while a payment authorization requires a single authoritative transition. Choose single-writer, quorum, or conflict-resolution behavior per class. Do not apply one replication mode to every table because the infrastructure supports it.
3. Establish explicit write ownership
For single-writer data, route writes to one region and replicate to standby. Use a durable epoch or fencing token so only the current owner can commit. Promotion must advance the epoch in a coordination system that both regions respect. Health checks alone cannot distinguish a failed primary from a network partition where the primary is still alive.
4. Make replication state part of promotion
Record the last durable replication position and measure lag. Before a planned failover, drain writes and confirm the standby reached the required point. During an emergency, calculate the possible data-loss window and make it visible to the incident commander. Do not promote a replica whose state is unknown merely because it responds to health checks.
5. Separate traffic failover from data promotion
Read-only traffic, static assets, and cached responses may move before write ownership. Write traffic should move only after the new owner is fenced and ready. DNS has caching and propagation delay; use layered traffic controls and make both regions reject writes with invalid epochs. The database boundary, not the load balancer, is the final protection against split brain.
6. Make external effects idempotent
Payments, emails, webhooks, and jobs can be retried during failover. Give every logical action a globally stable idempotency key and persist its outcome. Replicate or reconcile action state according to its recovery requirement. A database failover is incomplete if a replayed event sends a second refund from another region.
7. Reconcile after connectivity returns
Preserve logs and versioned events from both sides. Compare data, identify writes accepted under each epoch, and apply domain-specific repair. Some conflicts can merge; others require compensating actions or human review. Never reconnect two divergent primaries and assume replication will choose the correct business result automatically.
8. Practice failover and failback
Run game days that include replication delay, stale DNS, partial control-plane failure, lost credentials, and an old primary returning. Measure actual recovery time and data loss. Test failback as a new migration rather than reversing traffic casually. Update runbooks from observed operator steps and automate only decisions whose safety conditions are measurable.
Engineering trade-offs
Single-writer designs simplify consistency while adding cross-region write latency or temporary unavailability during promotion. Multi-writer designs improve local availability but require conflict-free operations or explicit resolution. Synchronous replication reduces data loss while increasing latency and coupling availability to network health.
Keeping a warm standby costs more but shortens recovery. A cold recovery is cheaper and may meet objectives for a non-critical product. The right architecture comes from business recovery requirements, not from a general desire to appear globally distributed.
Common mistakes
- Using DNS failover as the only split-brain protection.
- Promoting a replica without checking replication position.
- Allowing both regions to write without domain conflict semantics.
- Treating caches and financial state as if they need the same consistency.
- Replaying side effects without global idempotency keys.
- Testing regional loss but not the old primary returning.
- Planning failover while leaving failback undefined.
Failure handling and observability
Monitor replication position and lag, ownership epoch, rejected stale-epoch writes, traffic distribution, regional error and latency, external-effect deduplication, and recovery checkpoints. Keep audit logs for promotions, operator decisions, data-loss estimates, and reconciliation. Alert when more than one region believes it owns a single-writer resource even if no user error is visible yet.
During uncertain ownership, reject writes rather than accept conflicting truth. Preserve read-only access when safe. If promotion exceeds its deadline, communicate reduced capability and follow the manual recovery path. After recovery, keep the old region fenced until reconciliation and a reviewed failback complete. Availability pressure should not bypass the data-ownership invariant.
Interview insight
The interviewer is not testing whether you can draw two regions behind a global load balancer. They are testing whether you understand failure domains, recovery objectives, replication, fencing, and split brain. A staff-level answer decides who owns writes during a partition and proves that promotion cannot create two authorities.