Only believe what you can validate: a verification framework for agentic AI

How to read this article

This article picks up where my January piece left off. My catch phrase is still the same: “only believe what you can validate”. This article is about validation.

If you are new to agentic AI (in modernization contexts), I would highly recommend reading The Realities of Application Modernization with Agentic AI first. Start there, then come back.

The problem with “looks good to me”

I have had a version of the same conversation many times over the last months. I work with a customer on an agentic application modernization. In this concrete example we are facing one of 150 COBOL modules with 10 related copybooks. The customer’s experts chose this module intentionally as it is “fairly isolated”. The agent works for five minutes and generates the reverse engineering documentation of the module for us: a markdown file with over 2500 words, longer than the draft of this article. I ask: “alright, is this what you would expect?” The room goes quiet. Blank stares. Then: “Looks good to me on first glance”, often followed by a “do you expect me now to read through everything?”

And the person is right. It does look good. They quickly scanned the first 10-20 lines. The record layout is described correctly. So they conclude the rest is probably correct as well. Reasonable conclusion at first glance. But what is easy to verify is usually also easy to get right, for the model too. So it probably is correct, but it does not hold any significance for the rest of the document.

The knowledge captured in that module took decades to develop, and the agent claims to have extracted it in five minutes. And in this moment they have no easy way to determine which of these 2500 words are correct, hallucinated, or misinterpreted, and which words are missing. They are facing this massive document and would need to read through it thoroughly, which would take hours. Just as a reminder: this module is one of the easier ones. What will happen if they feed the agentic AI larger modules with more dependencies?

My first impulse is to say: “well, of course I do. How else would we be able to determine if the agentic output is sufficient?” But obviously it is not that easy. It will never scale. It will overload the experts and create AI fatigue. And that’s fair enough. So the question is, how do we solve this problem?

The problem is that agentic AI produces a lot of output in minutes, and that output looks complete, coherent, and confident. And if we are honest, this is a problem we have with every agentic AI output. But in some use cases, and app modernization is one of them, accuracy matters more and the consequences of getting it wrong are more severe, especially when we talk about core business cases of organizations.

“Most things look right” in a mission-critical system is not the same as correct.

The tech can’t hold up to the expectations

As a model’s context window fills up, two effects kick in: Below 50% capacity, middle tokens degrade – the model retains the beginning and end but loses what is in between. Above 50%, even the beginning decays. The model continues to produce confident output.

These effects are widely known as context rot. If you want to learn more, I recommend: producttalk.org/context-rot

The other extreme is not providing enough context. Either way, the result is one of three specific failure modes.

1. The miss. The AI overlooked something that should have been included, resulting in incomplete output while nothing is technically wrong.

2. The hallucination. The AI produced something plausible that does not exist or is not true. Looks good, still wrong.

3. The misinterpretation. The AI tried to follow slightly vague instructions to the best of its knowledge and still produced the wrong outcome.

Now we know the core of the problem and the answer, obviously, is verification.

The three actors in any verification story

Verification involves three parties: humans, AI, and deterministic tools. Each pair has a specific failure mode.

“You are absolutely right.”: When AI misses something and a human is lucky and catches it, AI just fixes the found problem unless actively prompted to find the root cause.

“I miserably failed you.”: When AI produces something that compiles but is factually wrong, deterministic tools are unlikely to catch the problem, which results in a green pipeline and the problem getting shipped to production.

“I have to automate this?”: Automation has always been hard. And there is still this wrong mental model: if a task only needs 5 seconds to be done manually, why spend the effort to automate it. This technical process debt becomes visible way too late and is another root cause why agentic AI adoption stalls.

It became clear that none of these three actors can work alone and scale, so how do we combine them to become safe with real verification? How do we define the question mark in the middle?

While researching this matter, two things became clear to me: agentic AI output is only as good as its verification mechanism, but not every verification mechanism has the same significance in every context.

You can choose the best LLM with the best benchmarks, but its output does not matter. What matters is your ability to tell how good it is. And to determine this, I tried to create a framework/mental model to help simplify answering the question “what good looks like”.

An iterative verification framework

Verification should always be seen iteratively. If you skip the bare minimum the advanced checks lose reliability. Across all disciplines the six core questions you need to answer to verify the output of agentic AI are essentially the same:

• What is the check? The concrete thing you run, review, or measure.
• What does it cost to build? Money, time, expertise, access to production data.
• Who performs it? Deterministic tooling, AI, or a human. Often a combination.
• What does it actually prove? Be honest about what you are actually learning by performing this check.
• What is the result? A concrete, tangible and easy to act on artifact.
• What tools exist to help?

The output is not only the table. Most teams have never tried the exercise of actually writing down the de facto verification model that already exists implicitly in the expert’s heads. It was just never documented, and therefore probably inconsistent and invisible. At the same time, writing the verification model down names it – and naming it is the first step to improving it. That exercise is what makes it possible to identify that the value of certain bare minimum checks is “almost nothing”. The framework is just a helper.

Below, I show two examples of how I would build verification layers for reverse engineering and code generation output done by agentic AI (these are currently the most relevant in my work domain).

Disclaimer: This is an opinionated POV, your criteria might be different, your domain might be different.

Reverse engineering

When applying the framework to reverse engineering, the verification layers could look like the following:

Code generation

Applying the framework to code generation is way less fuzzy than reverse engineering, as the quality gates for good AI-generated code are the same as for traditionally written code. So in a perfect world, in a mature software engineering culture, this should already be in place, which makes it easier to “just apply” to AI-generated output.

*Tokens are usually never free

The uncomfortable truth about reverse engineering is that you always have to start from scratch, because documentation cannot be executed, which makes it harder to implement good quality gates, but not impossible.

A check without a result is not a check

The difference between determinism and non-determinism is the nature of the output. Deterministic checks produce a result and they also tell you if the result is actually good. Linting gives you a list of violations. A test run gives you pass or fail. Human (and AI) checks do not work like that. “Domain expert reviews the module” produces nothing, unless somebody decided upfront what should come out of it. And if nothing comes out, we are back at “looks good to me”.

So do not ask an expert to read 2500 words and tell you if it is correct. Give them the business rules the agent extracted and let them confirm or correct each one. The expert spends the same time, but the result is something you can act on.

And if a check cannot be broken down like that, then the check itself is the problem.

Closing

Most teams I talk to are currently operating even below the bare minimum. If technical maturity and AI maturity are low, organizations prioritize investing resources into innovation projects. They learn the hard way what non-deterministic means, then they panic and demand people like me to fix it. Truth is I can’t. That decision needs to be made consciously. And that is work. Sometimes it is a lot of work. Balancing out risk and opportunity is always hard. Same as taking responsibility and ownership.

Pick your discipline. Take the conscious decisions and create your own validation layers. Apply them.

Only believe what you can validate. And validate.