The limits of AI agent verification

Pebble sells verification, so this page is written against our own interest. Every claim below is something our approach cannot do. Know the failure modes before you commit, not after.

What are the limitations of AI agent verification?

The limitations of AI agent verification are six, and they are structural rather than temporary: a model cannot reliably check its own work, learned graders carry biases and drift, optimization pressure finds holes in any grader, a check only catches what it encodes, a wrong check passes wrong work, and expert annotation costs real money. None is solved by a better prompt or a bigger model. Each has a mitigation that reduces the damage without removing the limit.

Can an agent verify its own work?

Not reliably, and not for the errors that matter. People expect otherwise because of a complexity-theory intuition that verifying a solution is easier than producing one, an argument that does not transfer to a system operating by approximate retrieval. When Stechly, Valmeekam, and Kambhampati tested GPT-4 on Game of 24, graph coloring, and STRIPS planning, they found "significant performance collapse with self-critique" and "significant performance gains with sound external verification".

The practical form of this ceiling: a model does not catch the error classes it consistently makes, because catching them requires the capability whose absence caused them. If an agent misreads a date format, asking it to check its own date parsing produces a confident pass.

Mitigation. External verification only. Any check sharing weights with the thing it checks is a self-check wearing a costume, which is why the check has to live in the environment rather than in the agent. That boundary is the subject of harness vs environment.

How reliable is an LLM judge?

Reliable enough to be useful and biased enough that you cannot trust an unmeasured one. The paper that made LLM-as-judge standard practice is also the best source on its holes: Zheng et al. found strong judges like GPT-4 reach over 80% agreement with human preferences, the same rate humans reach with each other, while documenting "position, verbosity, and self-enhancement biases, as well as limited reasoning ability". Later work isolates position bias across pairwise and list-wise settings and self-preference bias, a judge scoring its own outputs higher.

Drift is the harder half. A judge's agreement rate is measured at a point in time against a distribution of work, and both move. The process changes, edge cases shift, the underlying model gets updated, and the judge keeps returning verdicts at the same confident rate while meaning something different. Nothing in the output signals this. You find out when a human spot-checks and disagrees.

Mitigation. Score every learned check against held-out examples before trusting it, report the agreement number rather than calling the judge accurate, keep a human in any check where agreement is too low, and re-measure on a schedule instead of at launch. A judge with a stated agreement rate is an instrument. A judge described as "highly accurate" is not.

Can a verified agent still game the check?

Yes, and it gets more likely the harder you optimize against the check. This is reward hacking, and the people who build training environments treat it as the central quality problem rather than an edge case. Epoch AI's survey quotes practitioners directly: "Reward hacking is a big issue. The model might cheat by searching up a solution, or if you're not careful with how you script the repo, by checking out future commits", and "It takes many many iterations to check against reward hacking" (An FAQ on Reinforcement Learning Environments, Denain and Barber, January 2026).

Note what those examples have in common. Neither is a clever attack on the grader's logic. Both are sloppy environment construction leaking the answer into reach.

Mitigation. Push checks toward computed, where a deterministic assertion on system state is much harder to satisfy without doing the work than a model's opinion is. Then harden the environment itself, because a correct check in a leaky environment is still hackable.

What does a check fail to catch?

A check catches exactly what it encodes and nothing else. If the pass condition is "invoice total equals the sum of line items", the check is silent on whether the invoice was fraudulent, whether the vendor was right, or whether payment terms were misapplied. Every check is a decision about what to look at, and everything outside it passes by default.

This is the coverage limit, and it is why unknown-unknowns survive verification. You cannot write a check for a failure mode nobody has thought of. The first time a new failure appears in production it passes every check you have, because your checks encode the failures you already knew about. Verification narrows the space of undetected errors without closing it.

Mitigation. Audit runs, including passing ones. Sample real run records and have the process owner read them, not to catch a specific failure but to find the failure classes your step map has no check for. Every one found becomes a new check. This maintenance cost never goes to zero, because real processes drift and the check set has to drift with them. The broader case for treating each run as a recorded experiment is in our white paper, Scaling Experimentation.

What happens when the check itself is wrong?

The work fails and the record says it passed, which is the worst outcome available. A check is a written-down proxy for the actual standard, and a proxy under optimization pressure stops tracking the thing it stood for. Goodhart's law is the canonical statement of this, drawn from Charles Goodhart's 1975 observation that "any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes", popularly compressed to "when a measure becomes a target, it ceases to be a good measure".

Concretely: a tolerance set slightly too loose passes the discrepancies it exists to catch, and every one of those runs is now labeled clean in the audit trail and weighted as a positive example in training data. Because the same records serve as evidence and as training set, a wrong check does not just miss a failure. It teaches it.

Mitigation. Approval by the person who owns the process, not by the vendor, and approval of the specific pass condition rather than of the general idea. Then review the conditions when the process changes. Most specification gaps we see are a threshold nobody revisited after the business changed.

What does verification cost?

Judged checks cost expert time, and that is the binding constraint on how much of a process gets verified well. A judged check needs a few hundred pass/fail examples annotated by people who know the work, which means pulling your best operators off their jobs. They have the least slack, and re-measuring drift pulls them again later.

Computed checks are cheap to run and expensive to design, since somebody has to know the process well enough to state a deterministic condition over real system state. No version of this is free, and vendors who imply otherwise are planning to ship you unaudited judges.

Mitigation. Spend the annotation budget where computed checks cannot reach, and be willing to leave a step with a human rather than buy false coverage with a cheap judge. A process with four computed checks, one measured judge, and two steps still owned by a person is in better shape than one with seven confident checks of unknown quality.

What does this add up to?

Verification with known limits beats trust with unknown ones. Both approaches ship work you cannot fully guarantee. Only one tells you where the gaps are, gives you a percentage instead of a feeling, and leaves a record you can read when something goes wrong. When an unverified agent fails, you get a transcript and a guess. When a verified run fails, you get a step number, and when a verified run passes for the wrong reason the audit trail makes the wrong reason findable.

The methods, ranked, are in how to verify AI agent work. If you arrived from the identity-verification side of the term, that market is covered in agent verification, and terms are defined in the glossary.

If you want an honest read on which steps of your process can be verified and which cannot, that is what a scoping call is for: [email protected]. If an environment is the wrong tool, we say so on the call.