AI agents on long-horizon tasks rarely fail cleanly. What you see instead is drift, where the agent slides off track and the slide looks like normal activity if nobody is watching closely. An agent can exit with RETURN_CODE=0 and a non-empty output file after spending 40 turns running ls in the same directory, retrying an auth endpoint 20 times against a flat 401, or talking to itself in prose while producing nothing on disk.
We have watched agents report success after writing an empty findings file because they decided they were done. Others loop forever on a parameter they are not allowed to modify. Picture a testing agent hitting the same broken endpoint 47 times. It burns tokens, produces nothing, and from outside the process with no monitor attached it is hard to tell apart from a run that is genuinely making progress.
EVA already catches fabricated findings after an agent finishes. The watchdog is aimed at the middle of execution, where drift is still eating into the budget. Reviewing the final artifact still matters, but by then the spend is already locked in. You want to catch the behaviour, not wait for the receipt.
Architecture
The watchdog runs as a daemon thread next to each agentic loop. Two checks, one cheap and one expensive. Five correction tiers that escalate from a nudge into a hard kill. A file-based signal protocol the main loop drains at every turn boundary.
Both check paths converge on a single correction output, and the tier is decided by how many corrections have already fired in this run, not by which check tripped. The Nth correction lands at tier min(N, 5). Putting the correction in a file rather than a direct callback keeps the watchdog and the agent decoupled. They share a directory on disk, not memory, so if the watchdog crashes the agent can keep running.
How it decides an agent is stuck
On each 30-second tick the watchdog reads the last 10 assistant messages and runs four cheap heuristic checks.
- Same bash command repeated three times or more
- Same error returned three times or more
- No file writes after the agent has been running for 15 turns
- Recent tool calls skewed heavily toward read/explore operations
Each one is a few lines of Python. Here is the repeated-command check, which sets the shape for the other three.
function check_repeated_commands(messages, window):
counts = {}
for each tool_call in the last `window` messages:
if tool is bash:
key = normalize(tool_call.command)
counts[key] += 1
for key, count in counts:
if count >= 3:
return triggered, "repeated command: " + key
return not_triggered, ""
If every heuristic stays quiet but the agent still has not produced its primary output file, a second check takes over. A mid-tier LLM call reviews the same 10-message window and answers a simpler question, whether the agent is progressing or drifting. Each call costs around a cent and runs at most once every 90 seconds, so we only spend it when the cheap rules cannot tell and the agent has nothing to show yet.
The drift prompt feeds the model four pieces of context. They are the agent's task description, the file it should produce, a snapshot of what the filesystem actually contains right now, and a compact view of recent activity. It answers with a triple (stuck, reason, done_but_continuing). That last flag matters, because some agents finish the job and then keep typing, and a correction aimed at a looping agent would be wrong for one that has already succeeded but simply does not know to stop.
Different agent categories get different minimum-turns thresholds before any check fires. A login agent that should terminate in a handful of turns starts with a lower floor, while authorization testing, which by design hits the same endpoint once per role, starts with a higher one. Most categories sit at the default of 15. Authorization is 25.
The five tiers
When a check fires, the watchdog writes a correction. Correction number N maps to tier min(N, 5), with a hard cap of 5 corrections per agent run, and the ladder runs from a light touch to a kill.
tier 1 → Nudge
tier 2 → Redirect
tier 3 → Force-wrap
tier 4 → Hard stop
tier 5 → Terminate
Nudge (tier 1). A short user-role message that tells the agent what the watchdog observed and suggests a different approach. Nothing else changes, and the agent is free to take it or ignore it.
Redirect (tier 2). The second correction calls out the specific failure mode more directly and pushes the agent toward a new approach. Still no cap on remaining turns.
Force-wrap (tier 3). By the third correction the agent is told to stop probing and deliver on whatever it already has. No turn cap yet, because writing the output itself takes turns.
Hard stop (tier 4). Remaining turns are capped at turn + 10. Ten more attempts to produce something usable, then the budget is gone.
Terminate (tier 5). Cap of turn + 3. Whatever the agent tries next, it will not get past the third turn after this correction.
The structure trades autonomy for control as the tier rises. Tiers 1 through 3 intervene but leave the agent in charge, because the agent almost always has better local context than the watchdog does. By tier 4 the watchdog has been wrong too many times to keep trusting the agent's judgment, so the remaining budget gets cut. Tier 5 is just the recognition that continuing would be throwing good money after bad.
Signals, not interrupts
Corrections travel through the filesystem rather than through a signal handler or a callback. Each one lands as correction.signal in the agent's working directory, written atomically via a temp file and os.rename.
function write_signal(signal_dir, correction, max_turns_override):
signal_path = signal_dir + "/correction.signal"
tmp_path = signal_path + ".tmp"
payload = { correction: correction }
if max_turns_override is set:
payload.max_turns_override = max_turns_override
write payload as JSON to tmp_path
atomically rename tmp_path to signal_path
At the top of every turn, before firing the next LLM call, the main loop looks for the signal file. If one is waiting, it reads the payload, appends the correction as a user-role message, optionally reduces max_turns, and clears the file.
if correction_text is present:
append { role: "user", content: correction_text } to messages
log: "[WATCHDOG] correction injected at turn " + turn
if signal has max_turns_override:
new_max = signal.max_turns_override
if new_max < max_turns:
max_turns = new_max
Interrupting an agent mid-tool-call is how you end up with half-written files, dangling database transactions, and shell commands that ran partway through. The turn boundary is the only safe moment to inject a user message, because it sits cleanly between a completed tool call and the next LLM inference. Keeping the protocol on disk buys a second benefit for free. The watchdog and the agent never share memory, so if the watchdog crashes the agent just keeps running.
The read side is atomic too. The main loop renames correction.signal to correction.signal.claimed first, which stakes exclusive ownership of the payload, then reads and deletes the claimed file. Any new signal the watchdog writes during that window lands back at correction.signal and gets picked up on the next turn.
Traced in Langfuse
Every watchdog correction shows up in Langfuse as its own span, named watchdog-<tier>-turn-<N>. The drift-check LLM call nests inside that span, so its cost and its latency live in the same trace as the rest of the agent's spend. There is no parallel log file to reconcile when a run goes over budget.
Above, a correction pulled out of a live trace. watchdog-nudge-turn-20 is a tier-1 nudge that fired on turn 20, with roughly eleven cents of drift-check LLM call behind it.
Clicking into the span shows both sides of the intervention.
The Input field records which heuristic fired. In this run the rule caught an agent that had written no output files and was stuck exploring. The Output field below it is the actual user-role message that got appended to the agent's messages. Here the watchdog spotted that the agent was re-authenticating against a session it already held, listed the files it already had on disk, and told it to stop. On the next turn the agent did.
Every intervention stays on the record afterwards. Pick any correction in any run and you can reconstruct why it fired, what the agent was told to do about it, and, by scanning the spans that follow, whether the agent actually changed behaviour or shrugged the correction off and marched into the next tier.
What one 80-agent run looked like
Trace 2322eaddcf3294efc3094f77cc7b3997, captured 2026-04-18. Full-assessment run against a production target. Wall-clock 7 hours 40 minutes, 80 agents across the pipeline, 70 corrections fired.
Tier 1 Nudge #################################### 34
Tier 2 Redirect ################## 18
Tier 3 Force-wrap ############ 12
Tier 4 Hard stop #### 4
Tier 5 Terminate ## 2
The distribution matches the intent. Most interventions stay in tiers 1 through 3, where the agent keeps its autonomy, and only six of 80 agents ever escalated past force-wrap. Two hit terminate.
The worst drifters by correction count were business-logic and cross-origin with 8 each, websocket-testing and rest-api-testing with 7, and cryptography-testing with 6 (which hit terminate). All of those are test categories where the attack surface is broad and the expected behaviour is genuinely exploratory, so the line between "still searching" and "stuck" gets genuinely hard to draw.
Two outliers are worth calling out. xss-reflected and rce-validation each burned through six corrections and produced zero findings. Drawing six interventions without shipping anything is not the outcome anyone wants, even if the agent eventually exited cleanly, so both are on a watchlist for per-category review. The watchdog did its job here (nothing ran unbounded), but the agents themselves need their purpose and termination criteria rethought.
Zero agents ran unbounded. Every single one either produced its deliverable or hit a hard turn cap on the way out. The pre-watchdog failure mode, an agent quietly burning through a full eight-hour budget on 47 ls commands, cannot happen anymore.
Where it breaks
The remaining shortcomings fall into three buckets.
Legitimately repetitive work trips false positives. Authorization testing hits the same endpoint once per role by design, and replay-style vulnerability scanning does the same. The default "no file writes after 15 turns" rule is too aggressive for categories whose whole job is to explore before writing anything. Our workaround is a per-category minimum-turns floor that pushes the first check later for repeat-by-design agents, which works, but it makes the watchdog's behaviour depend on getting each agent's category right. A misclassified agent either gets checked too aggressively or not aggressively enough.
The drift LLM is fallible. A mid-tier model reading 10 messages of context is going to call it wrong sometimes. Manual review of the 2026-04-18 run flagged a handful of tier-1 nudges that turned out to be false alarms, agents stuck in a legitimate dead-end-and-retry cycle that the heuristics mistook for a loop. The 90-second rate limit has its own blind spot, which is that an agent that spins for 60 seconds and exits cleanly sits below the check interval, so the watchdog never gets a look at it.
The two terminate-tier events. Both involved agents that looked busy to every earlier check. They were writing files, their commands did not repeat, and the drift LLM rated them as making progress. Terminate only fired because the correction counter had marched all the way to 5. In both cases the agent was doing something that looked like work but was not actually producing useful output, and the watchdog only caught the runaway after four earlier corrections had been shrugged off. Catching them sooner would take either a stricter ignored-correction heuristic or a second-order check that notices when an agent accepts a correction without actually changing behaviour. Neither is in place today.
When to use this
Each alternative fails in its own way.
A static turn cap ("this agent gets 50 turns, no more") is blunt. It punishes careful slow agents at the same rate as fast pathological ones, so a thorough agent hits the wall while a lying one finishes well under budget and ships garbage anyway.
Output-only validation, the job EVA does, catches fabricated findings after they arrive but cannot refund the tokens the agent burned producing them. If you want to protect the budget and the output, you need detection that runs in flight.
No detection at all is the default for most agentic frameworks, and it is the worst of the three. A drifting agent consumes budget, produces nothing, and gives no outward signal that anything went wrong until somebody checks the output and finds it missing.
The watchdog sits between the blunt turn cap and the nothing-at-all default. A lightweight observation loop on one thread, an escalation ladder with five clear rungs, and a bounded cost of a few cents per check. Any long-horizon agentic system with heterogeneous agent types and a non-trivial token budget needs something like this.
Next
EVA and the watchdog are the two halves of a combined safety net. EVA watches for wrong answers after the agent finishes, and the watchdog watches for wrong behaviour while it is still running. A future post will cover how the rest of the agent fleet is orchestrated across an assessment, and how the signals from both systems feed back into scheduling decisions for the next run.