Method
How to tell whether an automation is actually working
A process that has quietly stopped doing its job almost never announces it. It keeps running on schedule, keeps returning success, and keeps looking exactly like a process that works. Distinguishing the two is a specific skill, and it comes down to what you decided to assert.
The question is not "is it running?" You can see that it is running. The question is: if it had stopped doing the thing you built it for, what would have told you, and how long would it have taken?
Most people check an automation by looking at whether it ran. That check is nearly worthless, because running is the part that almost never breaks. Schedulers are reliable. What breaks is the work in the middle — a credential expires, an input moves, a downstream system starts refusing politely — and none of that necessarily stops the run. The run completes. It reports success. The work does not happen.
This matters more once a process runs unattended, because the entire reason you automated it was to stop watching it. An automation nobody watches and nobody verifies is not a reduction in work; it is a deferral of it, and the bill arrives with interest when someone downstream discovers three months of missing output.
Running, reporting, and working are three different things
Separate them deliberately, because almost every silent failure lives in the gap between two of them:
- Running — the scheduler fired and the process executed. Easy to verify. Almost always true.
- Reporting — the process returned an outcome. Also easy to verify, and the one most teams mistake for the third.
- Working — the thing you wanted to happen in the world happened. Rarely verified directly, because it requires looking somewhere other than at the automation.
A green dashboard is evidence of the first two. It is not evidence of the third, and it is routinely read as though it were.
Failure shape one: it is alive but out of date
A process can execute faithfully every day while running a version of itself from months ago. The schedule fires, the heartbeat lands, every status is green — and the logic being executed is not the logic you reviewed. This happens whenever deployment and execution are separate steps and only one of them is monitored: the thing that pulls the new version fails, the thing that runs the old version does not.
Liveness is not currentness. A heartbeat proves something is alive, not that it is the right something. If you monitor only one of the two, monitor currentness — a stalled deployment with a healthy heartbeat is far more common, and far more expensive, than a process that simply stops.
Failure shape two: green means "I could not look"
This is the most dangerous shape, and the most common in anything that reads from another system. A check goes to look at a queue, a folder, an inbox, an API. The read fails — expired token, changed permission, renamed field. The code catches the error, finds no items to process, and reports the truthful-sounding result: nothing to do.
"Nothing to do" and "I was unable to find out whether there was anything to do" are completely different statements, and collapsing them into one green result is how a monitored process becomes invisible. It is worse than no monitoring at all, because the report actively reassures you. Every tick is a small false statement that the queue was empty, accumulating into a confident record that nothing needed doing.
The tell is a check that has been suspiciously clean for a long time. A process that reads real-world input and has reported nothing to do for two months is not necessarily healthy. It may never have read anything at all.
Failure shape three: every check passes and nothing happens
The third shape is subtler and is not a defect in the automation at all. Every check is correct, every component is healthy, and the outcome you built the thing for is not being produced — because nothing measures the outcome.
The pattern is that all the instrumentation points at supply and none of it points at result. Whether the documents were generated, not whether anyone acted on them. Whether the messages were sent, not whether anyone replied. Whether the pages are being served, not whether anyone arrived. Each of those first clauses can be perfectly, verifiably true while the second is zero, and a system instrumented only on the first half will report health indefinitely.
The rule: a check that cannot look must not report green
One rule collapses most of this. Every check needs three possible answers, not two:
- Pass — I looked, and it was fine.
- Fail — I looked, and it was not fine.
- Undetermined — I could not look, and I am telling you that rather than guessing.
Two-state checks are forced to fold "undetermined" into one of the other two, and they almost always fold it into pass, because the code path that could not read anything looks locally identical to the code path that read nothing. Adding the third state is a small change that converts a category of invisible failures into loud ones.
It pairs with a second rule: the thing that watches must not depend on the thing it watches. A check that writes its alarm into the system that has failed is a smoke detector wired to the breaker it is monitoring. It will work correctly in every situation except the one it exists for.
Four assertions worth adding
Concretely, for any unattended process, these four are usually the highest-value checks you do not already have:
- Assert on the artifact, not the dashboard. A control panel that says a feature is enabled is a statement about a control panel. Read the thing that was supposedly produced — the file, the record, the served bytes — because only that proves the work occurred.
- Count the outcome, not the run. Track the number you actually want to go up. If a process has run four hundred times and its outcome counter is still zero, that is the single most informative fact available about it, and it is invisible unless someone decided to count.
- Give every reading an expiry. Anything a human has to check by hand should go stale on a schedule and complain when it does. Silence is indistinguishable from health, so make absence of a fresh reading an explicit failure rather than a quiet gap.
- Separate "broken" from "bad". A guard that cannot see should escalate loudly. A correctly measured but unwelcome number should be reported, not escalated — if a true, stable condition raises an alarm on every cycle, people learn to dismiss the alarm, and the next real one goes with it.
A drill you can run this week
Pick your most trusted unattended process. Then, deliberately, break its input — revoke the credential it reads with, rename the folder it looks in, point it at an endpoint that refuses. Do it in a way you can reverse in a minute.
Now look at what it reported. If it reported success, or reported nothing to do, you have just reproduced the exact failure it would experience in production, and you know that when it happens for real nobody will find out. That is a much cheaper way to learn it than the alternative. Fix the check so the same drill produces a loud, specific complaint, and run the drill again to confirm it does.
The discipline generalises: a guard is only proven by watching it fail on purpose. A check that has never once gone red is not a check that has passed — it is a check nobody has tested. Whether the boundary you set is respected is the same question in a different register, which is what an AI agent should be allowed to do on its own, and what it should leave behind afterwards is the record every action ought to write.
If you have unattended processes and no answer to "how would we know if that stopped working", that is the gap we would close first. Bring us what is running and we will tell you what it should be asserting.
Work with us