· 5 min read
When Green Checks Lie
A reliable AI workflow has to prove that useful work happened.
Software teams are good at monitoring machinery. We check whether a service responds, a database accepts writes, or a scheduled job finishes on time. An LLM-driven workflow can pass all of those checks and still miss the goal. I ran into four cases of this while building a research platform for systematic trading strategies. In each one the checks were green and the result was wrong.
For agentic software I want monitoring to answer two questions:
- Did the intended result happen?
- Did it happen within the permissions and limits I set?
I call this outcome observability.
Four incidents
Implausible Sharpe ratios and future-dated rows
I audited the evidence behind the platform’s research gates: the stored performance results that a strategy needed in order to advance. Some rows reported Sharpe ratios above 10. Some carried performance dates in the future.
A Sharpe above 10 is not a good result. It suggests the number was not computed from real out-of-sample returns. A future-dated row cannot describe anything that happened. Neither shows up in a check on the job. Both showed up when I looked at the data the job produced.
A stale result overriding a failed one
Strategies had to pass a dual backtest, two runs that both had to succeed. In one case the dual backtest failed, but an older single-run result for the same strategy was still stored and the gate read that instead. The strategy looked like a pass.
The gate checked that a passing result existed. It did not check that the result was current, or that it came from the run the gate was about.
A correlation that was too good
A candidate signal showed a near-perfect correlation with forward returns. Real alpha is weak and noisy. I flagged it as probable label leakage, meaning information from the prediction target had reached the features.
No infrastructure check could have surfaced this. The pipeline ran cleanly and produced an excellent-looking number.
A crash-looping gateway behind green dashboards
The market-data gateway was killed with SIGKILL 7 times in one market day. Each restart forced strategies to re-hydrate their state, and while they did, they ran zero inference. The process was up after every restart, so health checks and dashboards stayed green. The platform was running and not doing its job.
The fix was a memory limit of at least 4x the JVM heap. What should have alerted was not process liveness but a count of inferences per market day, which was zero for long stretches.
Activity is not evidence
Most operational signals describe an attempt:
- a job started and exited successfully;
- an API returned a success response;
- an agent invoked the expected tool;
- a process is up; or
- a gate found a passing result.
None of these shows that the work was correct, timely, or authorized. Each incident above sat behind one of these signals.
Rule: a gate runs on fresh artifacts, with cost provenance
The first three incidents share a cause. A gate accepted a stored result without asking where it came from. My rule now:
- Fresh. A gate runs on artifacts produced by the run it is judging. An older passing result does not count, and a newer failing one always overrides it.
- Provenance. Every result records what produced it: the code version, the data window, and the cost and slippage assumptions. A backtest without cost assumptions attached is not evidence.
- Sanity bounds. A result outside plausible bounds (a Sharpe above 10, a date in the future) fails the gate and is flagged for review. It does not pass with a warning.
Three controls that make the green check meaningful
Prove the result from outside the agent
Start with the state a user or downstream system can observe: recent, valid records arrived; the artifact passed validation at its destination. This proof should not come only from the component declaring success. Where possible, the receiver verifies.
Constrain action before execution
Define the operations, resources, environments and limits available to the agent, and enforce them at the tool boundary. A useful record connects the requested action, the identity that authorized it, the policy that permitted it, and the effect that followed, without relying on the agent’s own explanation.
Compare reality with the reviewed design
Automation drifts even when each action looks reasonable. Check the resulting state against an explicit specification. A deployment command’s exit code is evidence about the command, not about the environment it produced.
What can be proved
Freshness and authorization are properties of the boundary and can be model-checked.
A gate that accepts any stored pass has a spec bug. The admissible result is the one from the judged run: older passes do not count and a newer failure overrides. A model checker finds the stale single-run as a counterexample. The same applies to authorization in front of a tool: a protected action runs only if a grant for that exact call is valid, single-use and unexpired. The agent proposes and a deterministic monitor decides. The model’s report that it succeeded is not an input.
A bound can fail a Sharpe above 10 or a future-dated row. It cannot fail a 9.7 that was fit to the bound, and it cannot tell leakage from alpha. Real signal is weak and noisy, so a near-perfect correlation calls for an audit of the sample. For those results, check that the cost assumptions are present and that the checker ran on this artifact. Do not ask a solver to certify that the result was useful.
A proof can pass for the same reason a job does when the spec is thinner than the claim. Liveness holds vacuously when the process is up and no inference arrives, so the property has to be on the work: inferences per market day above zero.
Pair attempts with proof
For every operational event, name the independent evidence that would make it meaningful.
| The system reports | Verify instead |
|---|---|
| Job completed | Required artifact passed validation |
| API accepted a request | Expected downstream state is visible |
| Data connection is open | Recent, valid records arrived on time |
| Process is up | Useful work per unit time is above zero |
| Gate found a passing result | Result came from the current run, with cost provenance, inside plausible bounds |
| Deployment passed | Live environment matches the reviewed design |
| Agent invoked a tool | Action stayed within policy and budget |
When proof is missing, the safest response depends on the stakes: pause the workflow, request review, reject the output, reduce permissions, or stop spending. Monitoring that cannot change behavior is only reporting.
Build the proof into the workflow
Start with the user-visible result and work backward. Decide what would convince a skeptical reviewer that the result happened, and place that evidence next to the action that claims to produce it. At each boundary, require that proof before more work is allowed.
The question I ask of any agent workflow is: what changed, who allowed it, and what proves the change was useful? That question survives model upgrades, prompt rewrites and new orchestration frameworks.