It is 3:07 a.m. and your phone buzzes: agent success rate < 80%. You open the laptop, pull up the logs for the support-agent service, and scroll through a wall of HTTP 429s from the LLM provider, a few timeouts from the CRM tool, and a stack trace that ends inside a retry wrapper. Most pods look healthy. CPU is flat.
Two questions matter, and you can answer neither. Which customer requests are actually affected? And if you restart the deployment, will the agent re-send the refund confirmation emails it already sent to half of them?
Many SRE teams inherited agent services with runbooks written for stateless APIs, and those runbooks stop working the first time an agent fails at night.
Why agent incidents are different
A classic web request is short, deterministic, and isolated, and replaying it in staging usually reproduces the bug. Agent runs break each of those assumptions.
They are long-running. A single run might plan, call an LLM, query three tools, wait for a rate limit to clear, call the LLM again, and send an email. Your request/response dashboards measure latency at the edge, so a run that is stuck in step four looks identical to one that is merely slow.
They are non-deterministic. Ask the model the same question twice and you may get a different tool call. You cannot reproduce the failure by re-sending the prompt, because the model will take a different path this time.
They are distributed. One run touches an LLM provider, an internal API, a vector store, an email service, and sometimes another agent. The failure might start in any of them, and each one has its own logs, clocks, and retry behavior.
Logs show symptoms, not sequence. You can see that 429s spiked at 3:01. You cannot easily see that run order-4711 called the CRM successfully, got a 429 on its second LLM call, retried four times in-process, and then died when the pod was evicted.
State lives in memory. Most agent frameworks keep the conversation, the plan, and the list of completed tool calls inside the process. When the process dies, that knowledge dies with it. This is the root cause behind the restart dilemma: the system has no durable record of which side effects already happened, so a restart has to choose between losing work and repeating it.
Why the usual workarounds fall short
The first instinct is to restart the pods. For a stateless API that is safe. For an agent with in-memory state, a restart either drops in-flight runs silently or, if a queue redelivers them, re-executes them from the top. That second case is how customers get two refund emails.
The second instinct is to add more logging. More log lines still give you fragments.
The third is to wrap every call in retries. In-process retries multiply load during a rate-limit event and vanish with the process. They don’t tell you whether a run is retrying, stuck, or gone.
The approach: treat each agent run as a durable execution
The fix comes from workflow engines, not AI tooling. Model each agent run as a durable workflow. The orchestration logic (the agent loop) is deterministic code, and every side effect, including each LLM call and each tool call, runs as a separate step whose input and output are persisted before the run moves on.
That one design choice gives an on-call engineer five things.
- An execution history per run. The persisted, event-sourced history becomes your primary debugging artifact. For any run ID you can see the exact input and output of every LLM call and tool call, in order, with timestamps. You read what happened instead of trying to reproduce it.
- Queryable status. Every run has an explicit state such as running, suspended, completed, or failed. “Stuck vs. slow vs. lost” becomes a query instead of a guess.
- Correlated traces. With OpenTelemetry and W3C trace context propagated across services, one trace can follow a run through the agent, the LLM gateway, and every tool it called.
- Meaningful metrics. Instead of only p99 latency, you track step failure rate, retry counts per step, time spent in each step, and the number of runs that haven’t advanced in N minutes.
- Safe remediation. Because completed steps are recorded, recovery means resuming from the point of failure. Steps that already succeeded, like “send email,” are not executed again on replay.
The rule that makes replay safe is simple: the orchestrator must be deterministic, and all I/O must live in steps. If a developer calls the LLM or reads the clock directly inside the orchestration code, replay can diverge from history. That rule belongs in your code review checklist, not just your runbook.
How to implement it with Dapr Workflow
Dapr, the CNCF graduated runtime, provides one open-source way to get these properties. Dapr Workflow stores an event-sourced history for each workflow instance in a state store and replays it after a crash, skipping activities that already completed. Side effects go in activities, and retry policies are declared per activity, so backoff survives a crashed process.
For agents specifically, Dapr Agents is an open-source Python framework built on Dapr Workflow. Vendor integration packages, for example from Diagrid, run agents written with frameworks such as LangGraph, the OpenAI Agents SDK, or Pydantic AI on Dapr Workflow, so existing agents can gain durable execution without a rewrite.
From an SRE’s point of view, the useful part is the management surface. Dapr Workflow supports operations to:
- Query the status of a workflow instance by ID.
- Pause and resume an instance.
- Terminate an instance that should not continue.
- Raise an external event into a waiting instance, for example an approval or a manual override.
Check the Dapr documentation for the exact commands in your version, and rehearse them in staging.
On observability, Dapr sidecars emit Prometheus metrics and propagate W3C trace context across service invocation and pub/sub, so traces connect across calls that go through Dapr service invocation and pub/sub, without hand-wired correlation IDs. YAML resiliency policies (timeouts, retries, circuit breakers) help contain a provider rate-limit storm.
If you would rather not run the workflow backend yourself, managed options exist. One example is Catalyst from Diagrid, a platform built on Dapr that visualizes every activity in a run, provides OpenTelemetry tracing, and supports opt-in signing of workflow history for teams that need verifiable records. As with any vendor, validate those claims against your own failure scenarios before you depend on them during an outage.
The 3 a.m. runbook
Here is the opening incident, replayed with durable agent runs.
- Scope the blast radius. Query workflow status for the affected agent. Count runs that are running, suspended, and failed since the alert window started. Export the failed and stalled run IDs; those map directly to affected customer requests.
- Open one failed run’s history. Read the step sequence. Confirm where it stopped: in this case, the second LLM call, after exhausting retries on 429s. Note which side-effecting steps (the email send) already completed.
- Follow the trace. Jump from the run ID to its trace to confirm the root cause sits at the provider, not in your tool or network.
- Check the step metrics. Look at retry counts and time-in-step for the LLM activity. If retries are amplifying load, tighten the resiliency policy or route to a fallback model instead of adding capacity.
- Stop the bleeding. Pause new or in-flight runs if continuing would make things worse. Do not restart pods to “clear” state.
- Remediate per run. Once the provider recovers, resume paused runs. For runs that failed outright, check what your engine version supports for re-running from a failed step, or start a new instance whose activities skip side effects that already completed. Terminate runs that are no longer valid, and raise an event for any that need a human decision.
- Verify. Watch the stuck-run count and success rate return to baseline.
- Hand off. Share affected run IDs and final states so support knows which customers to contact.
Turning history into better postmortems
The same recorded history makes postmortems faster. Instead of rebuilding a timeline from logs and memory, attach the exact LLM and tool call sequence of representative runs. Patterns jump out, such as a tool that times out only for large accounts or a retry policy too aggressive for the provider’s limits.
Before your next on-call rotation, work through this checklist:
- ☐ Every agent run has a stable, searchable run ID tied to the customer request.
- ☐ All LLM and tool calls execute as persisted steps, not inline in the agent loop.
- ☐ Orchestration code is reviewed for determinism.
- ☐ Dashboards show step failure rate, retries, time-in-step, and stuck runs.
- ☐ Traces propagate W3C context across agents and tools.
- ☐ The runbook uses status, pause, resume, terminate, and raise-event instead of pod restarts.
- ☐ Remediation commands have been rehearsed in staging.
Conclusion
Agents are not going to become deterministic, and LLM providers will keep rate-limiting at inconvenient hours. What you can change is whether each run leaves behind a durable, readable record and a safe way to resume it. Whether you build on open-source Dapr Workflow or adopt a managed durable workflow platform, the goal is the same: when the pager goes off at 3:07 a.m., you should know exactly which runs broke, why, and how to finish them without sending anyone a second email.
