Suppose a hypothetical research agent runs for forty minutes, calls several tools, waits for human approval, and produces a final report. A deployment restarts its worker after thirty minutes. The in-memory plan disappears, completed tools are called again, and the user has no reliable way to learn whether the task is still running.
How would you make a long-running AI agent survive crashes and deployments?
Long-running agents are workflows. Model reasoning may be probabilistic, but execution state, side effects, retries, cancellation, and recovery must be durable and deterministic.
Why the common answer is incomplete
Most candidates answer: “Save the conversation after every step.”
A transcript can help reconstruct context, but it does not prove which side effects completed, which approval was granted, or whether another worker currently owns the task. Replaying the entire conversation can make the model choose a different next action. It may also repeat a tool whose response was lost after successful execution.
A production design stores an explicit workflow state machine and treats model calls as individual activities within it.
A production approach
1. Represent the workflow as durable state
Create a workflow record with tenant, authenticated initiator, goal, status, current step, configuration versions, deadlines, and cancellation state. Store step inputs and normalized outputs separately from model-visible prose. The database or workflow engine becomes the source of truth; a worker process is only a temporary executor.
2. Append events before advancing state
Record decisions such as step_scheduled, tool_started, tool_completed, approval_requested, and workflow_completed with stable identifiers. Advance the state machine transactionally with the event when possible. On recovery, rebuild the current state from the durable snapshot and recent events rather than asking the model to infer what probably happened.
3. Make every activity independently retryable
Assign a stable activity and idempotency key to model calls, retrieval, and tools. Store attempt count, deadline, and outcome. Read-only activities may retry automatically; side effects require downstream idempotency and reconciliation of ambiguous timeouts. A recovered worker must reuse the activity identity instead of creating a new logical action.
4. Use leases to prevent concurrent execution
Workers claim a step with a time-bounded lease and heartbeat. Only the lease owner can commit its result. If the worker disappears, another worker may resume after expiry. Include a fencing token or version check so a delayed original worker cannot write after ownership moved. A distributed lock without fencing can still allow two workers to complete the same step.
5. Checkpoint compact, reproducible context
Store the selected evidence, tool results, decisions, and unresolved constraints required for the next step, with pointers to original data. Pin model, prompt, tool, and policy versions. Do not store hidden model reasoning as workflow truth. After recovery, assemble a bounded context from durable facts and let the model propose the next action under the same validated contract.
6. Wait asynchronously
When a workflow awaits a user, webhook, timer, or external job, persist the wait condition and release the worker. Resume from an authenticated event matched to the workflow and expected state. Keep timeouts and escalation behavior in the state machine. Holding an application process or open request for hours is not durable execution.
7. Support cancellation and compensation
Cancellation sets durable intent and prevents new steps from starting. In-flight activities must observe cancellation where safe. Completed side effects may require compensating actions rather than rollback; cancelling an order workflow cannot pretend an already sent email or issued payment never occurred. Define compensation per tool and require authorization for consequential reversals.
8. Version workflows across deployments
New code must know how to resume states created by older versions or migrate them explicitly. Use compatible state schemas and versioned handlers. Canary new workflow versions on newly created jobs while allowing existing jobs to finish on pinned logic. Test deployment, crash, replay, timeout, and resume at every state transition—not only the happy path.
Engineering trade-offs
Durable workflow engines provide timers, retries, history, and recovery, but add infrastructure and determinism constraints. A relational state machine can be sufficient for a smaller product if transitions, leases, and idempotency are implemented carefully. Frequent checkpoints improve recovery while increasing storage and write load.
Pinning old workflow versions improves reproducibility but prolongs support. Migrating in-flight workflows reduces version count while adding conversion risk. Long history is useful for audit and debugging but may contain sensitive tool data that needs redaction and retention controls.
Common mistakes
- Treating an in-memory agent loop as a durable workflow.
- Replaying a transcript and assuming the same actions will result.
- Retrying side effects with new identifiers after an ambiguous timeout.
- Using a lock without lease expiry and fencing.
- Keeping a worker occupied while waiting for human approval.
- Assuming cancellation can undo completed external effects.
- Deploying a new state schema that cannot resume old checkpoints.
Failure handling and observability
Trace workflow and activity IDs, state version, lease owner, fencing token, attempts, deadlines, idempotency keys, approvals, model and tool versions, and every state transition. Monitor workflow age, time in each state, stuck leases, retry rates, ambiguous activities, duplicate suppression, approval wait, cancellation latency, and recovery success after deployments.
If a worker dies, expire its lease and resume from the last committed checkpoint. If an activity outcome is unknown, reconcile before retrying. If the current deployment cannot interpret a stored version, quarantine the workflow and alert rather than guessing a transition. Preserve an operator path to inspect, resume, cancel, or compensate without editing database rows manually.
Interview insight
The interviewer is not testing whether you can keep a long chat history. They are testing whether you understand durable execution, state machines, leases, idempotency, and recovery. A staff-level agent design allows probabilistic decisions inside steps while keeping the workflow lifecycle deterministic and auditable.