AI That Understands vs. AI That Guesses

The number everyone will quote

Across a dozen frontier AI models AWS tested, precision at telling a real vulnerability from a false alarm clustered in the mid-50s. When these models flagged something, they were, on the whole, about as likely to be wrong as right.

I do not read that as an argument against AI in security. I read it as the most useful thing anyone has measured about it.

Because AI is already doing real security work. Teams are using it to triage findings, review code, take a first pass at an investigation that used to sit in a queue for a week. That is happening, and for the most part it is good. Which makes one question the most important in the field: when the AI flags something, is it right?

For a long time we could not answer that cleanly. The benchmarks measured something else, whether a model could find a vulnerability, or exploit one. Useful, but only half the job. The other half is judgment: of everything it could flag, is this one real, and does it actually matter here?

This month AWS published research aimed squarely at that second half. They built a benchmark, and they say it is the first to measure not whether AI can find a vulnerability, but whether it can tell a real one from a false alarm. That is the question that decides whether an AI security tool saves you time or quietly buries you in false alarms. Their answer, across the dozen frontier models they tested, was that mid-50s precision: as a group, when the models flagged something, they were about as likely to be wrong as right.

The industry has talked about accuracy for a long time. Measuring it this directly, and this honestly, is the useful part.

But the number is not the interesting part. The interesting part is why the models miss.

Why the models miss

AWS built challenges where the code contains a genuinely vulnerable pattern, but the surrounding environment makes that vulnerability unexploitable. A Kubernetes network policy blocks the path. An identity and access boundary prevents the escalation. The dangerous-looking code is right there in the source, and it is also, in that environment, harmless.

The models saw the pattern and stopped there. The environmental facts that would have changed the answer were within reach, and the models did not connect them. In AWS's words, they see a vulnerability pattern and stop reasoning.

That single finding is the whole thing, and it is the failure mode I wrote about earlier this summer in a piece on what I called the four-second keyhole: an AI reasoning fluently over a view of an environment it has not actually connected. Here it is again, measured. Finding the bug was never the hard part. Knowing whether it can actually hurt you is. The vulnerability is in the code. Whether it is exploitable is in how the environment relates around it. A model that reasons over the code without resolving the environment will be confidently wrong a lot of the time, not because it reasons badly, but because it never connected the relationship that would have changed its answer.

No, a bigger model won't fix it

Here is why this matters for how you build, and why I am optimistic rather than discouraged. The obvious objection is that real tools do not work in a single pass, they use multi-step agentic loops, so surely those close the gap. AWS answers that directly, and it is worth sitting with. Agentic verification is proven mostly on offensive tasks, where success has an oracle: the exploit fires or it does not. Judging that code is safe has no such oracle. So extra passes re-sample the same judgment rather than confirm a negative, and the harness still inherits the base model's understanding. If the model cannot separate an effective mitigation from an ineffective one in a single look, more looks do not add the missing knowledge.

The environment is buildable

The missing piece is not reasoning horsepower. It is a resolved model of the environment. And that is good news, because that is buildable. Raw context is not enough, the AWS models often had the relevant facts in front of them and still did not use them. What works is structuring the environment so the relationships are already explicit: what is reachable, which identities touch what, where the controls sit, connected before the model reasons about any single finding. When the answer to "is this reachable" and "does this control neutralize it" is already resolved rather than left for the model to infer from raw material, the questions that trip up code-only models stop being hard. The line between AI that works and AI that guesses is not the model. It is whether it reasons over a resolved picture of the environment or over the code alone.

This is the bet we are making at Unizo, and it is why we are building the environment model first, so the reasoning runs on top of it rather than scrambling to assemble it. Not because AI is the wrong tool for security. Because it is the right one, and the right tool deserves to be built on something better than a keyhole.

How to evaluate an AI security tool

So if you are evaluating AI security tools, take AWS's advice, it is exactly right. Do not just ask whether the tool finds vulnerabilities. Ask how often it is wrong, and ask what it understands about your environment when it decides. AWS was careful that it measured general-purpose models, not finished security products, and so am I. But the line it draws is the one that matters: a system that reasons only over code, however much scaffolding wraps it, will keep producing confident noise. A system built on a real understanding of the environment is the one that reduces real risk.

The question was never whether AI belongs in security. It does. The question is whether your AI understands the environment the vulnerability lives in, or only the code in front of it.

Sudhanva Gnaneshwar, Co-founder & CTO, Unizo