How do you debug an AI agent that is failing in production?
Debug a production AI agent by tracing one failed run from input to final side effect, then finding the first step that differs from a healthy run. Isolate whether the fault came from the model, a tool, workflow logic, state, or infrastructure. Fix that layer, replay representative cases, and verify the real business outcome rather than trusting the agent's success message.
The expensive mistake is starting with the model. A bad customer reply might come from a weak instruction, but it could also come from stale state, the wrong CRM record, a tool timeout, a retry that duplicated an action, or a verifier that accepted partial work. Changing the prompt before locating the failing layer can hide the symptom without fixing the system.
The operating rule is simple: preserve the run, compare it with a healthy one, and move from evidence to cause. You are not trying to explain why an AI behaved strangely in the abstract. You are trying to identify the first observable point where a business workflow stopped matching its contract.
What evidence should you collect before changing anything?
Start with one concrete incident. Record the original input, the expected outcome, the actual outcome, the model and tool calls, retries, approvals, state reads and writes, and the final response from the destination system. If a message, booking, payment, or CRM update was involved, preserve the destination receipt too. A polished agent response is not a receipt.
OpenTelemetry's GenAI conventions provide a useful mental model: one agent run is a trace, and each model call, tool invocation, or agent step is a span inside that trace. A trace lets you see sequence and dependency. Token use, latency, finish reasons, tool results, and error status become part of one run instead of scattered logs that an operator must reconstruct later.
Do not capture sensitive content by accident. OpenTelemetry's official guidance notes that GenAI telemetry records metadata by default, while prompts and tool content require explicit content capture. Treat that setting as a privacy decision. If you need payloads to reproduce an incident, redact secrets and personal data, restrict access, and define retention before turning detailed capture on.
Write the expected outcome in business terms. 'The agent should answer correctly' is too vague. 'The agent should use the current account record, draft one response, request approval before sending, and store the approved message ID' can be tested. Debugging gets faster when the workflow has an observable contract.
How do you find the first step that went wrong?
Read the trace in execution order. Compare the failed run with a healthy run using the same workflow and similar inputs. Look for the first divergence, not the loudest downstream error. If a publishing step received a malformed brief, the publisher is where the problem became visible, but the research handoff may be where it began.
A useful comparison has five checkpoints: input selection, state retrieval, model decision, tool execution, and completion verification. At each checkpoint, ask whether the system used the correct data, made an allowed decision, called the intended tool with the intended target, received the expected response, and proved completion against the destination.
Microsoft's agent tracing guidance describes spans such as agent invocation, workflow invocation, planning, tool execution, and memory operations. Those labels are useful because they turn a vague complaint like 'the agent went off track' into a narrower question: did it retrieve the wrong memory, make a bad plan, call the wrong tool, or mishandle the tool result?
Check retries early. A timeout does not always mean the first attempt failed. The destination may have completed the action while the agent lost the response. An automatic retry can then create a duplicate. If the workflow changes external state, use an idempotency key or reconcile the destination before repeating the action. The guide to [durable execution](https://gallmur.com/en/notes/agent-crash-recovery-durable-execution/) explains why recovery must resume safely instead of starting over blindly.
Which layer is actually failing: model, tool, workflow, state, or infrastructure?
Model failures include unsupported claims, instruction conflicts, poor tool selection, and inconsistent judgment on equivalent cases. Test the same preserved input several times before blaming the entire workflow. If the output changes while the inputs, state, and tools remain fixed, model variability is a real suspect. If every run receives the wrong record, it is not.
Tool failures include authentication errors, schema mismatches, stale selectors, incomplete API responses, and successful requests aimed at the wrong target. Inspect the exact arguments and the exact returned payload. A status code can prove that a server accepted a request. It cannot prove that the right customer record changed.
Workflow failures live between steps: a missing field is treated as optional, an upstream warning becomes a downstream fact, an approval is requested too late, or a retry repeats a side effect. These failures often survive prompt edits because the orchestration contract itself is weak. The broader [production failure-mode guide](https://gallmur.com/en/notes/ai-agent-failure-modes-production/) helps name these patterns; debugging is the procedure that locates the specific one in a real run.
State failures happen when the agent reads stale memory, mixes records, loses progress after restart, or treats conversation context as the system of record. Separate conversational memory, execution state, and business records. The article on [AI agent state management](https://gallmur.com/en/notes/ai-agent-state-management-persistence/) explains that boundary in detail.
Infrastructure failures include overloaded services, routing defects, network timeouts, queue delays, and partial deployments. Anthropic's postmortem of three infrastructure bugs is a useful warning: overlapping defects produced inconsistent degradation, and existing evaluations did not detect the problem quickly. One routing issue initially affected about 0.8% of Sonnet 4 requests and reached 16% during the worst hour. Small aggregate error rates can still create severe failures for a subset of users.
How do you prove that the fix works?
Replaying the original incident is necessary, but it is not enough. Build a small evaluation set with the failed case, nearby variants, a healthy control, missing data, a tool timeout, a duplicate request, and a case that should stop for human review. A fix that passes only the original example may be a patch shaped around one trace.
Link each evaluation result back to its trace. The score tells you which case failed. The trace tells you where it failed. Microsoft recommends correlating evaluation run identifiers with traces so operators can move from a weak aggregate score to the exact model call, tool invocation, or state operation that produced it.
Verify the destination after every state-changing test. Read the record, fetch the page, inspect the booking, or confirm the message receipt. For deterministic requirements, prefer deterministic checks: schema validation, exact-field comparison, duplicate detection, permission tests, and reconciliation against the system of record. Use model judgment where meaning is genuinely ambiguous, not where software can establish the fact.
Then run the evaluation continuously on representative production behavior. Anthropic's postmortem says its existing evaluations missed the degradation and lists continuous evaluations on production systems, more sensitive tests, better debugging tools, and user feedback signals among the improvements. That is the practical standard: a fix is not complete until the system can detect the same class of failure next time.
What should a production debugging checklist include?
For each incident, preserve the failed run and destination evidence. Define the expected business outcome. Compare the trace with a healthy run. Find the first divergent span. Classify the failing layer. Check retries and state before repeating any action. Apply the smallest causal fix. Replay the incident and neighboring cases. Verify the destination. Add a regression evaluation and an alert tied to the observable failure.
Also assign ownership. Someone must decide whether the incident is a model-quality issue, a workflow defect, a tool integration problem, or an infrastructure event. Without that decision, teams bounce the failure between prompt edits, application code, and vendor support while the customer keeps seeing the same result.
If your current system cannot reconstruct a single failed run, that is the first problem to fix. You do not need a giant observability program on day one. You need a stable run identifier, ordered steps, tool arguments and responses with sensitive data protected, durable state transitions, and proof from the destination. That is enough to stop guessing.
What are the limits of trace-based debugging?
A trace shows what the system recorded. It does not guarantee that every important event was instrumented, that the log is truthful, or that the expected outcome was well defined. Missing spans can make a clean trace look convincing. Detailed payload capture can also create privacy and security risk if access, redaction, and retention are weak.
Evaluation sets are samples, not proof that an open-ended agent will handle every future input. Models, tools, data, and external services change. High-impact workflows still need bounded permissions, approval at the point of consequence, a manual operating path, and incident review.
The goal is not to make an agent impossible to break. It is to make failure attributable, bounded, reproducible, and visible before it becomes a repeated business outcome. If your team is debugging from screenshots and the agent's own explanation, the system is not observable enough yet.