deskthebuildAI AGENTS
← All notes · 5 min read

A tool schema change broke every running agent. How would you version tool contracts?

Imagine a hypothetical agent platform where a team renames a required field in a ticket-creation tool. New deployments use the updated schema, but long-running workflows still contain plans and checkpoints based on the old contract. Tool validation fails, the model retries with guessed fields, and active jobs become stuck.

Tools are APIs consumed by both deterministic orchestration and probabilistic model output. Changing a schema therefore requires the same compatibility discipline as changing a public service contract, plus additional evaluation of how models interpret it.

Why the common answer is incomplete

Most candidates answer: “Update the prompt so the model uses the new field.”

That may fix newly started requests, but it does not update in-flight checkpoints, cached tool definitions, old prompt versions, or clients deployed on a different schedule. A model may also continue producing the familiar old shape even after seeing the new schema. Prompt text cannot create backward compatibility at the service boundary.

A production platform needs immutable contract versions, adapters, capability negotiation, and a staged deprecation process.

A production approach

1. Treat the tool schema as a versioned API

Assign an immutable version to the tool name, input schema, output schema, error model, authorization requirements, and side-effect semantics. Store that resolved version with every workflow and trace. Avoid silently changing the meaning of an existing field. If behavior changes materially, publish a new contract even when the JSON shape appears compatible.

2. Prefer additive compatible changes

Add optional fields with clear defaults before making them required. Preserve existing enum values and output fields during the migration window. A field becoming stricter can break models and deterministic callers even if application code compiles. Document which combinations are valid and enforce them in one validator shared by execution and tests.

3. Pin running workflows to resolved versions

When a workflow starts, record the exact tool versions it can use. On resume, reload those contracts or run through a compatibility adapter. Do not let a deployment replace definitions underneath an existing plan. New workflows can adopt the new version after evaluation, while old jobs complete on their original contract within an explicit support window.

4. Put adapters at the deterministic boundary

Translate old arguments to the new internal service shape in application code when semantics remain compatible. Normalize defaults and map output errors back to the old contract. Do not ask the model to perform the migration on every call. Reject transformations that would guess a required business value; those need clarification or a new workflow decision.

5. Validate before executing side effects

Parse model-proposed arguments against the pinned schema, then apply authorization and business rules. Return structured validation errors that identify correctable fields without exposing internals. Cap repair attempts and never relax a required field merely to make the model succeed. Validation separates a malformed proposal from a tool-service outage.

6. Test contract behavior with recorded task patterns

Maintain deterministic schema tests, adapter tests, and end-to-end agent evaluations using representative calls. Include optional fields, unknown fields, enum changes, old checkpoints, and tool errors. Compare tool-selection and argument-validity rates before rollout. A schema that is technically valid may still confuse the model because descriptions or field boundaries became ambiguous.

7. Roll out producers and consumers in a safe order

Deploy the service and adapter support before exposing the new contract to agents. Shadow or canary the new definition on eligible workflows. Monitor old and new usage separately. Only after new producers are stable should the team stop creating old-version workflows. Continue serving old contracts until their maximum workflow lifetime and retry window have passed.

8. Deprecate with evidence, not a calendar alone

Inventory active workflows, cached prompts, external clients, and stored checkpoints that reference the old version. Alert on new old-version usage after the cutoff. Remove execution support only when usage reaches zero and rollback no longer depends on it. Preserve enough metadata to interpret historical traces after the implementation is retired.

Engineering trade-offs

Supporting multiple versions increases code, test, and operational surface. For short-lived, read-only tools, restarting workflows on the new version may be simpler. For long-running or side-effecting agents, compatibility is usually worth the cost. Additive schemas reduce breakage while accumulating fields and ambiguous states if deprecated shapes are never removed.

Adapters preserve old clients but can conceal semantic differences. A renamed field is easy to map; a new authorization rule or business concept may require a new workflow. Versioning should not be used to pretend incompatible behavior is compatible.

Common mistakes

  • Mutating an existing tool schema in place.
  • Versioning inputs while leaving output and error semantics implicit.
  • Letting resumed workflows load whatever tool definition is currently deployed.
  • Asking the model to translate old arguments instead of using deterministic adapters.
  • Deploying the new producer before the service can accept its calls.
  • Retrying schema validation failures as if they were transient outages.
  • Removing an old version without inventorying stored checkpoints.

Failure handling and observability

Trace workflow version, tool contract version, proposed argument shape, validation category, adapter path, authorization result, service response, and repair attempts. Monitor calls and failures by version, old-version creation after cutoff, adapter conflicts, invalid arguments, unknown fields, and workflows stuck during resume. Avoid logging sensitive argument values; store hashes or approved identifiers where possible.

If a new version regresses, stop assigning it to new workflows and keep old execution support intact. If a running job references an unavailable version, pause it with an operator-visible compatibility error rather than handing the model a different contract. If an adapter cannot map a required value, request clarification or restart from a safe checkpoint. Never guess through an incompatible side-effecting call.

Interview insight

The interviewer is not testing whether you can edit a JSON schema. They are testing whether you understand compatibility across distributed deployments and durable workflows. A staff-level answer pins contracts, validates deterministically, rolls out consumers safely, and retires versions only after evidence shows nothing still depends on them.