AI brokers are being judged by the standard of their solutions. That’s the unsuitable place to begin. Earlier than asking whether or not an agent hallucinated, misunderstood a request or made a foul determination, ask: did the work run in any respect?
I function scheduled brokers throughout roughly 18 companies on one machine. They learn telemetry, watch repositories and put together reviews. For 18 months, I anticipated critical failures to come back from the fashions. I anticipated a foul determination or a assured reply constructed on a misunderstanding.
I used to be unsuitable about the place the hazard lived.
Not one critical outage started with a foul mannequin output. Each one started within the plumbing across the mannequin. A job was registered with the scheduler incorrectly. A script known as a binary that was truly a shell alias. A timeout guard relied on a command the working system didn’t ship. Every failure regarded clear from the surface. Every returned exit standing zero. The agent had not failed in an fascinating manner. It had not run.
The outage that regarded like silence
The primary failure was a scheduled job that by no means loaded. The configuration existed, the command regarded appropriate and the encircling service was wholesome. The schedule was unsuitable, so the runner by no means launched the job. Nothing crashed. Nothing logged an exception. The anticipated report merely didn’t seem.
That could be a tough failure to note as a result of an absent output can imply a number of various things. The system could have run and located nothing. It could have run and did not authenticate. It could by no means have began. A dashboard that shows solely the newest successful-looking worth turns all three states into the identical clean area.
The second failure was a shell assumption. A script invoked a command that labored interactively as a result of it was an alias. The scheduler launched it in a non-interactive atmosphere the place the alias didn’t exist, but the wrapper nonetheless accomplished efficiently. The agent was blamed for producing no outcome though it by no means reached the work.
The third failure concerned a timeout guard constructed round a command obtainable on one machine however absent from the working system that truly ran the job. The safeguard didn’t safeguard something. The method once more exited cleanly, leaving a system that regarded idle slightly than damaged.
These are peculiar distributed-systems failures: incorrect registration, atmosphere variations and dependency assumptions. AI makes them tougher to see as a result of the mannequin output is essentially the most seen a part of the system. Groups examine the immediate and reply whereas the scheduler, shell and course of boundary stay outdoors the body.
Richard Prepare dinner’s essay “How Advanced Methods Fail” makes the broader level: complicated techniques hardly ever fail due to one dramatic part. They fail by means of interactions amongst peculiar elements whose habits made sense regionally. An AI agent is one other part in that system, not a alternative for the system round it.
A very good agent reviews unknown
Probably the most helpful design rule I reached got here from a unique incident. One in every of my brokers misplaced an API credential throughout a run. It may now not see the channel it was imagined to summarize.
The tempting response would have been an empty abstract: no new messages, no motion required. That reply would have been true in a slender sense. The agent had seen zero messages. It will even have been fully deceptive, as a result of zero messages was not the state of the channel. It was the restrict of the agent’s entry.
The agent reported unknown as an alternative.
That distinction is the boundary between a system that’s trustworthy about its proof and one which quietly manufactures certainty. A very good agent reviews unknown when it can not observe the supply. A nasty one reviews zero. The identical rule applies when a instrument name instances out, a scheduler has not loaded the job or a dependency can’t be resolved. “No outcome” and “no knowledge noticed” are usually not interchangeable.
The POSIX specification isn’t an AI doc, nevertheless it represents the specific contracts agent techniques want. A human’s interactive shell isn’t the identical atmosphere as a scheduled course of, and a profitable exit isn’t proof that the supposed work occurred.
Observability should embrace what didn’t occur
Most observability steerage begins after execution: gather logs, traces and metrics for the work carried out. That’s obligatory, however agent fleets want one other query: what work was anticipated however by no means noticed?
I now distinguish between a job that ran and produced a outcome, a job that ran and located no qualifying knowledge, a job that ran however couldn’t entry its supply and a job that didn’t run or whose execution state is unknown. These states want completely different alerts and completely different language in reviews.
“No new gadgets” ought to require proof that the supply was reachable and assortment accomplished. If that proof is lacking, the right result’s unknown, not zero. The OpenTelemetry observability primer is a helpful start line for fascinated about traces, metrics and logs as proof of habits. For agent techniques, I prolong that concept to execution receipts: the schedule that fired, the command that resolved, the credential or connection that was used and the supply that was truly noticed. A last mannequin response is simply the final hyperlink in that chain.
The lesson is to not add extra prompts. Make the boundaries testable. Confirm binaries. Take a look at the scheduler’s actual atmosphere. Deal with lacking credentials and dependencies as specific states. Add a heartbeat that proves the work ran, not merely that configuration exists.
AI brokers can produce dangerous solutions. That failure is seen and infrequently reviewable. The extra harmful class is a system that produces a believable empty reply as a result of its execution path disappeared earlier than the mannequin had an opportunity to do something.
Your brokers might not be failing. They might not be working. Earlier than evaluating the intelligence of the reply, confirm the existence of the work that was supposed to supply it.
How do I detect when an AI agent scheduled job by no means ran?
Distinguish between 4 execution states: job ran and produced a outcome, job ran and located no qualifying knowledge, job ran however couldn’t entry its supply, and job didn’t run or whose execution state is unknown. Implement heartbeat alerts or execution receipts — logging the schedule that fired, the command that resolved, and the credential used — so a lacking heartbeat triggers an alert slightly than silent absence.
Why does a scheduled agent return exit code 0 however produce no output?
A zero exit code confirms the wrapper course of accomplished, not that the supposed work ran. Frequent causes embrace shell aliases that resolve interactively however not in a non-interactive scheduler atmosphere, misconfigured cron or job-runner schedules that by no means set off, and lacking dependencies (corresponding to a timeout utility) that trigger the agent to skip its work silently.
What’s the distinction between ‘no outcome’ and ‘unknown’ in an AI agent system?
‘No outcome’ implies the agent ran, noticed the supply, and located nothing. ‘Unknown’ means the agent couldn’t confirm whether or not it noticed the supply in any respect — for instance, as a result of a credential was lacking or a instrument name timed out. Conflating the 2 causes techniques to report false certainty, making actual failures indistinguishable from genuinely empty knowledge.
How ought to observability be arrange for AI agent fleets?
Normal observability (logs, traces, metrics) covers what occurred throughout execution. Agent fleets moreover want expected-but-not-observed monitoring: alerts that fireside when a scheduled job produces no execution receipt inside a time window. OpenTelemetry gives a basis; prolong it with agent-specific execution receipts masking the schedule set off, resolved binary, credentials used, and the info supply truly queried.
