Suppose a hypothetical team has a new model and prompt that look better in manual testing. The update improves several showcase examples, costs less, and responds faster. The team wants to replace the current production configuration immediately.
How would you roll out a new model or prompt without breaking production?
The risk is not that the new version always fails. It is that it behaves differently on rare inputs, tool calls, languages, long contexts, or safety boundaries that the manual examples did not cover.
Why the common answer is incomplete
Most candidates answer: “Run an A/B test and compare user engagement.”
An A/B test is useful after the system is safe enough to expose, but engagement can reward confident, verbose, or agreeable answers even when they are less correct. Random traffic splitting may also send high-risk workflows to an unvalidated path. If model and prompt change together, the team cannot tell which change caused a regression.
A production rollout needs offline evidence, workflow-specific gates, versioned artifacts, limited exposure, and a fast rollback path.
A production approach
1. Define acceptance criteria before comparing outputs
Write down the product outcomes the current configuration must preserve: task completion, groundedness, schema validity, tool success, safety behavior, latency, and cost. Assign thresholds by workflow and risk level. A customer-support draft can tolerate different failure modes from a tool that modifies infrastructure. Do not choose metrics after seeing which ones make the new version look favorable.
2. Build a representative evaluation set
Sample anonymized production patterns across common requests, hard cases, long inputs, languages, customer tiers, retrieval quality, and known incidents. Add adversarial and policy cases. Keep a separate holdout set so repeated prompt tuning does not overfit the visible examples. Record dataset version and provenance, and remove sensitive data according to retention and access policies.
3. Evaluate components and the complete workflow
Test prompt behavior, model behavior, retrieval, tool selection, and output validation independently where possible, then run end-to-end tasks. A model may score well on isolated answers but choose tools differently or produce arguments that fail downstream validation. Compare final task outcomes, not only semantic similarity to a reference response.
4. Version every behavior-changing artifact
Store the resolved model version, prompt template, system policy, tool schemas, retrieval configuration, decoding parameters, and validator versions together as a deployable configuration. Attach that version to traces and user-visible outcomes. “The latest prompt” is not reproducible, and provider aliases may change underneath an apparently unchanged release.
5. Shadow the new path before serving it
For a sampled set of eligible requests, run the new configuration without showing its answer or executing side effects. Compare outputs, validator results, latency, and cost with the production path. Redact or avoid duplicating sensitive requests when policy forbids shadow processing. Shadow traffic reveals capacity and compatibility issues, but it cannot measure user behavior and may double spend temporarily.
6. Canary by workflow and risk
Start with internal users, low-risk workflows, or a small eligible traffic percentage. Keep high-impact tools and sensitive tenants on the established version until the new path has evidence for those segments. Use sticky assignment where multi-turn consistency matters; switching models midway through an agent workflow can invalidate its plan or memory assumptions.
7. Set automated rollback and manual kill switches
Define rollback triggers for task failures, invalid tool calls, groundedness, refusal changes, latency, cost, and support escalations. Make the previous configuration immediately deployable without rebuilding code. A kill switch should disable a model, prompt, tool, or route independently. Test rollback before rollout, including in-flight jobs whose checkpoints reference the new version.
8. Expand slowly and continue post-release evaluation
Increase exposure only after each segment has enough representative outcomes. Review qualitative samples alongside metrics, because automated judges can share biases with the system under test. Continue evaluating after full rollout to detect traffic drift, provider changes, and delayed customer impact. Retain the ability to attribute every incident to a specific configuration version.
Engineering trade-offs
Offline evaluations are repeatable and safe but never capture every production behavior. Shadow traffic provides realistic inputs while increasing cost and sometimes privacy exposure. Canaries reduce blast radius but require parallel configurations, sticky assignment, and operational discipline. Changing one variable at a time improves attribution but can slow a migration that requires a coupled model and prompt update.
Automated judges scale review, yet their scores depend on rubric quality and may favor responses similar to their own style. Human review is richer but expensive and inconsistent. Production teams usually combine deterministic validators, task-specific metrics, model-based grading, and sampled human inspection rather than treating any one signal as ground truth.
Common mistakes
- Promoting a configuration because a few hand-picked demos look better.
- Using engagement as the only quality metric.
- Changing the model, prompt, retrieval, and tools in one untraceable release.
- Sending high-risk workflows into the first canary cohort.
- Relying on a provider alias without recording the resolved model version.
- Creating a rollback button that cannot handle in-flight jobs or new schemas.
- Stopping evaluation once the rollout reaches 100 percent.
Failure handling and observability
Log configuration version, eligibility decision, cohort, model resolution, prompt hash, tool versions, validator results, latency, cost, and final task outcome. Compare metrics by workflow and input segment, not only globally. Monitor cohort crossover in multi-turn sessions, shadow-path errors, disagreement rates, rollback triggers, and changes in user correction or escalation behavior.
If the canary regresses, stop new assignment first, preserve traces, and return affected workflows to the last known-good configuration. Decide explicitly how to handle in-flight tasks: complete them on their pinned version when safe, or restart from a durable checkpoint when compatibility changed. If evaluation data is insufficient, hold the rollout rather than widening exposure to manufacture confidence.
Interview insight
The interviewer is not testing whether you know how to run an A/B test. They are testing whether you can manage behavioral change in a probabilistic dependency. A staff-level rollout defines quality before deployment, limits blast radius, preserves attribution, and makes rollback a tested production capability.