The short answer
When an AI agent crashes mid-task, the damage depends entirely on whether you designed for recovery. If the agent was running as an ephemeral process with no execution journal, you lose the reasoning chain, the intermediate tool results, and any plan it was following. You also risk duplicating side effects on retry — sending the same message twice, charging the same card twice, or re-running a shell command that already modified your system. Durable execution solves this by persisting meaningful boundaries: which model response was used for a decision, which tool call completed, which approval was granted. After a crash, the agent resumes from the last recorded boundary instead of redoing work blindly.
The problem is real and getting worse. Production agents are long-running, stateful, and compositional — each step can fail independently, and failures compound. If you run five steps at 99% reliability each, your overall success rate is 95%. At ten steps, 90%. Real agents involve dozens of operations. Without recovery primitives, every crash is a manual fire.
Why agents break differently from traditional software
Traditional software is deterministic. Same input, same output. When something fails, you retry the whole operation, and side effects are usually predictable. AI agents break three of these assumptions at once.
First, agents are probabilistic. The same prompt can produce different responses across calls. A retry after a crash might not produce the same plan or the same tool selection. You cannot simply re-run and expect identical behavior.
Second, agents are compositional. A single user request might trigger a planning phase, a web search, content extraction, a structured-data call, and a synthesis step. Each component can fail independently — the search API rate-limits, the scraper hits a timeout, the model hallucinates a field that does not exist. The failure modes stack, and they stack fast.
Third, agents are stateful in a way that a simple API call is not. The context window accumulates reasoning, intermediate results, and the plan the agent was following. Lose that state mid-execution and you lose the thread. The agent cannot pick up where it left off because it does not know where it was.
The compounding failure math
Reliability in multi-step systems is not additive. It is multiplicative, and the math is brutal.
If each step in your agent workflow has a 99% success rate — which is optimistic for anything involving external APIs and LLM calls — a five-step workflow succeeds 95% of the time. Ten steps drops you to 90%. Twenty steps, which is common for real research or operations agents, puts you at 82%. That means roughly one in five runs fails somewhere, and without durable boundaries, each failure is a full restart.
This is why naive retry logic does not work for agents. A simple retry helps with transient failures — a flaky API, a network blip. But it does not preserve state across restarts. If your serverless function times out after step seven of twelve, retrying from step one means re-running six expensive, already-completed operations, paying for them again, and risking side-effect duplication on any step that touched an external system.
The alternative — manual checkpointing — works but shifts the burden to every developer. You end up writing save-state logic after each operation, handling partial failures, and implementing custom recovery paths. At scale, you are effectively building a durable execution engine from scratch, badly.
What durable execution actually means for agents
Durable execution is a programming model that guarantees code completion despite failures. The concept existed before AI agents — Temporal popularized it for general distributed systems, and Azure Durable Functions and Inngest brought it to serverless. But AI agents made it urgent, because agents combine long runtimes, external side effects, and probabilistic behavior in ways that traditional retry logic cannot handle.
For agent infrastructure, durable execution means five concrete things. First, automatic state persistence: the runtime records meaningful execution boundaries without requiring the developer to manually checkpoint. Second, automatic retries with backoff: transient failures get retried without losing the execution context. Third, workflow resumption: after a crash, the agent picks up from the last durable boundary, not from scratch. Fourth, human-in-the-loop primitives: workflows can suspend for hours or days awaiting approval, then resume with full state intact. Fifth, idempotent tool boundaries: the system knows which operations completed and does not repeat them on recovery.
The critical distinction is between session memory and durable execution. Saving chat history helps an agent remember what was said. It does not prove which command ran, which email was sent, which approval was granted, or whether a retry would duplicate a side effect. Durable agents need an execution journal, not a conversation log.
How to build recovery boundaries into your agent
Whether you use a workflow engine like Temporal or Restate, a serverless platform like Inngest or Cloudflare Workflows, or a framework-level checkpoint like LangGraph's persistence layer, the design principles are the same.
Record tool calls as durable steps. Every external operation — an LLM call, a shell command, an API request, a file write — should be wrapped as a step whose input and result are persisted once. On recovery, completed steps are skipped and stored results are injected. This prevents both wasted compute and side-effect duplication.
Separate deterministic orchestration from nondeterministic agent steps. This is the Temporal model: workflow code must be deterministic, while LLM calls, API requests, file writes, random values, and wall-clock reads are pushed into activities recorded as durable external results. If you mix nondeterministic operations into your workflow logic, replay will produce different results and your journal becomes unreliable.
Version your prompts and tools. A model or prompt can change between the original run and a recovery attempt. If you update a prompt mid-workflow, recovery behavior diverges from the original execution. Pin prompt versions and tool definitions to the workflow instance.
Design human-in-the-loop as suspend and resume, not as polling. When an agent needs approval before executing a side effect, the workflow should durably suspend — persisting its full state — and resume when the approval arrives, whether that takes minutes or days. This maps directly to the suspend and resume primitives that durable execution platforms expose.
Observability: you cannot recover what you cannot see
Recovery design is inseparable from observability. If you cannot reconstruct what an agent did before it crashed, you cannot verify that recovery picked up at the right point.
The OpenTelemetry GenAI semantic conventions provide a standardized vocabulary for this. They define gen_ai.* span and metric attributes that capture model calls, token usage, tool invocations, and retrieval operations in a consistent format. This matters because agent observability is not just about tracing HTTP requests — it is about reconstructing the reasoning chain, the tool-call sequence, and the decision points that led to a specific action.
In practice, every durable boundary should produce a trace span. When an agent crashes, you should be able to open the trace, see exactly which steps completed, which tool calls returned, and where execution stopped. Without this, recovery is blind — you are trusting that the journal is correct without being able to verify it.
For agents that run on scheduled jobs or background workers, observability also means knowing when they fail silently. A cron-driven agent that crashes at 3 AM with no alert is worse than no agent at all, because it creates the illusion of work being done. Health checks should verify not just that the process is alive, but that the last execution produced expected output.
What I check before trusting an agent in production
Before putting an agent into a production workflow, I verify five things. These are not theoretical — each one comes from a failure I have seen or caused.
One: does the agent survive a process kill mid-task? If I kill the process after step three of seven, does it resume from step three or restart from step one? If the answer involves manual intervention, the agent is not production-ready.
Two: are tool calls idempotent? If the agent calls an external API and the response is ambiguous — timeout, partial response, unclear status — does the retry path safely detect that the operation already happened, or does it blindly re-execute?
Three: is there a verification step after the agent claims completion? Agents can report success while the actual work is incomplete or wrong. The system needs an independent check, not just the agent's self-assessment.
Four: can I reconstruct what happened from logs alone? If the agent made a decision, I should be able to trace which model call produced the decision, what the input was, and what tools were available at that moment.
Five: what happens when the model changes? If the provider updates the model or deprecates a version, does the agent's behavior drift? Pinned versions and prompt versioning are the defense.
Limitations
Durable execution is not free. It adds latency — each durable boundary is a persistence operation. For interactive, user-facing agents where response time matters, you need to be selective about which steps get checkpointed. Checkpointing every LLM call in a real-time chat agent will feel sluggish.
It also adds infrastructure complexity. Running Temporal means running the Temporal server. Using Inngest means a platform dependency. LangGraph checkpoints need a backing store (Postgres, Redis, or a custom adapter). These are not insurmountable, but they are real operational costs that should be weighed against the cost of failures.
Finally, durable execution does not solve reasoning quality. If the agent makes bad decisions, persists them durably, and resumes from a bad plan, you get reliably bad execution. Durability makes agents consistent, not correct. You still need evals, prompt testing, and human review for high-stakes workflows.
The principle
The teams shipping reliable AI agents in production are not the ones with the smartest models. They are the ones who treat agent execution like distributed systems engineering — because that is what it is. Agents are long-running, stateful, probabilistic workflows that touch external systems and accumulate side effects. The reliability patterns for that class of system have existed for years. The new part is applying them to AI.
If your agent cannot answer the question 'where was I when I crashed, and what did I already do,' it is running on hope. Fix that before adding another tool.