deskthebuildVECTOR SEARCH
← All notes · 5 min read

How would you migrate embedding models without taking retrieval offline?

Imagine a hypothetical RAG service with millions of indexed chunks. A new embedding model appears to improve retrieval on the team’s evaluation set, but it produces vectors with a different dimension and similarity distribution. Replacing the model in application code would make new queries incompatible with the existing index.

Embedding migrations are data migrations. The model version, preprocessing, vector dimension, distance function, and indexed corpus together define a retrieval space.

Why the common answer is incomplete

Most candidates answer: “Re-embed every document and switch when the job finishes.”

That describes the central computation but not the production transition. Documents continue changing during a long backfill. Failed chunks need repair. The new index may be complete but still produce worse results for a particular query slice. A one-time switch without dual operation creates a difficult rollback if the new space behaves badly under real traffic.

A safe migration keeps old and new retrieval paths independently usable until the new path is complete, evaluated, and reversible.

A production approach

1. Version the complete embedding contract

Assign an immutable version to the embedding model, input preprocessing, chunk text construction, dimension, normalization, and distance metric. Store that version with every vector. Do not use a mutable provider alias as the only identifier. Two vectors are comparable only when they were produced under a compatible contract, even if their array lengths happen to match.

2. Create a separate target index

Build a new collection or namespace with its own schema and lifecycle. Avoid mixing old and new vectors in one field or overwriting the current vectors in place. Independent indexes allow different dimensions, tuning, and rollback. They also make completeness measurable: every eligible source chunk should have a target-version record or an explicit failure state.

3. Start dual writes before the backfill

Once the target pipeline is ready, send every create, update, permission change, and deletion to both index versions. Use the same source version and deterministic chunk identity in each path. Dual writing prevents the backfill from chasing a moving corpus forever. If one write fails, record the divergence and retry it; do not acknowledge both versions as synchronized when only one succeeded.

flowchart LR S[Source change] --> O[Versioned ingestion] O --> A[Current embedding index] O --> B[Target embedding index] C[Historical chunks] --> D[Checkpointed backfill] D --> B Q[Search request] --> R{Migration policy} R -->|Serve| A R -->|Shadow| B A --> E[Compare retrieval] B --> E

4. Backfill from a durable snapshot and checkpoint

Enumerate source chunks from a known snapshot or version boundary, process them in bounded batches, and persist progress. Use content hashes to skip unchanged work and rate limits to protect the embedding provider and index. If a document changes after the snapshot, the dual-write event should supersede the backfilled version. Workers must reject stale writes so late batches cannot overwrite newer content.

5. Verify completeness and integrity

Compare eligible source counts, chunk counts, deletion states, permissions, and versions between source, old index, and target index. Sample vector dimensions and reject non-finite values. Reconcile missing and orphaned chunks. Completeness is not simply “the queue is empty”; it is evidence that every current source object has the expected target representation.

6. Shadow retrieval on production queries

Continue serving the current index while querying the target for a sampled, policy-allowed set of requests. Compare candidate recall, rank overlap, citation quality, latency, and no-result behavior. Do not log sensitive query text without authorization. Shadow evaluation reveals traffic patterns that offline datasets miss, but it should remain read-only and must not double downstream tool execution.

7. Canary the read path

Route a small eligible cohort to the target index with sticky assignment. Keep the generator, prompt, and other components stable so retrieval differences remain attributable. Monitor outcomes by query type, language, tenant, and document class. Increase exposure only after target completeness remains healthy and the new path meets quality and latency gates.

8. Preserve rollback until the migration settles

Keep dual writes and the old read path for a defined observation window. A rollback should change the routing policy, not require another corpus rebuild. Once the target is stable, stop old writes, verify no clients remain, archive required audit metadata, and delete the old index through a reviewed retention process. Record which source versions were covered when the old path was retired.

Engineering trade-offs

Dual indexes temporarily double vector storage and write work. Dual writes complicate failure handling, but skipping them makes a long backfill inconsistent. Shadow reads increase query cost and can affect index capacity. A long observation window improves rollback confidence while prolonging operational expense.

Teams can sometimes recompute embeddings lazily on access, reducing the initial batch, but cold content may remain on the old version indefinitely and query behavior becomes harder to reason about. A full backfill provides a clear completion boundary. The right choice depends on corpus size, update rate, provider limits, and how quickly old infrastructure must be retired.

Common mistakes

  • Overwriting current vectors in place and losing the rollback path.
  • Treating the model name as the whole embedding contract.
  • Starting a backfill before dual writes capture ongoing changes.
  • Letting late backfill records overwrite newer document versions.
  • Declaring completion because workers have no queued jobs.
  • Switching every tenant at once after only offline evaluation.
  • Deleting the old index before in-flight jobs and clients stop referencing it.

Failure handling and observability

Track source version, embedding version, batch checkpoint, provider attempts, vector validation, index-write result, dual-write divergence, and reconciliation status. Monitor backfill throughput, estimated completion, failed and stale writes, target coverage, shadow disagreement, latency, and quality by query slice. Alert when the target falls behind current changes even if historical backfill is progressing.

If the target provider fails, pause or slow the backfill while preserving change events. If dual writes diverge, keep serving the old path and repair the target from durable source state. If the canary regresses, route reads back immediately while continuing to collect evidence. Never “fix” a mixed vector space by changing the query model without rebuilding the corresponding stored vectors.

Interview insight

The interviewer is not testing whether you can call an embedding API in a loop. They are testing whether you understand online data migration, version compatibility, backfills, dual writes, canaries, and rollback. A staff-level answer treats retrieval state as production data that must move without losing freshness or reversibility.