Approved checks: the unit of a verified run

This is Pebble's term. We use it in every proposal and every run record, so this page defines it precisely enough that someone else can use it too, or argue with it.

What is an approved check?

An approved check is a pass condition attached to one step of a process, written down and signed off by the person who owns that process, before any agent runs it. Three parts, all load-bearing. It is attached to a specific step, so a failure produces a step number. It is a condition rather than a score, so the verdict is pass or fail. And it was approved in advance by a named owner, which is what separates it from a check a vendor wrote and a check a model invented.

Drop any of the three and the term does not apply. A condition written after the run is a postmortem. A condition nobody signed is an assumption. A condition floating free of a step is a policy.

Why does the word "approved" matter?

The word "approved" matters because it assigns the correctness standard to the customer rather than to the builder, and that assignment is the entire difference between a verified process and a demo. When an AP controller signs the pass condition for the three-way match, including the tolerance, the resulting reliability number is a statement about her process. When a vendor picks the tolerance, the number is a statement about the vendor's product.

The practical consequence shows up in audits. A run record whose checks were approved by the process owner is evidence about the process. A run record whose checks came from a config file the customer never read is evidence about the config file. Under the EU AI Act, providers of high-risk systems must keep automatically generated logs "for a period appropriate to the intended purpose of the high-risk AI system, of at least six months" (Article 19), applicable from 2 August 2026. Six months of unapproved verdicts is six months of somebody else's opinion, retained.

What are the three kinds of approved checks?

Approved checks come in three kinds, ordered by how much you have to trust a model, and the design goal is always to move a check toward the first kind.

Computed. Code reads the state of the affected systems and decides. Do the totals reconcile, does the record exist, do the two systems agree, is the amount inside the approved limit. A computed check reruns to the same verdict forever and its error rate is zero by construction. More of a business process is computable than people expect once it is decomposed into small enough steps, which is why decomposition comes before any conversation about models.

Classified. A small model trained to make one narrow decision for one check, scored against held-out examples before it is trusted. The distinguishing feature is not the model, it is the number: a classified check ships with a holdout accuracy that gets reported alongside every verdict it produces. Without the number you do not have a check, you have a second model whose errors correlate with the first one's.

Judged. A judge trained on a few hundred pass/fail examples annotated by the operators who own the process, with judge-to-human agreement reported on held-out examples. Where agreement is too low to defend, a human stays in the check. The measurement is what makes this tier usable, not the model choice. An unaudited judge is not worthless, it is unmeasured, and an unmeasured check cannot support a reliability claim.

Which tier a given step belongs in is a decision, not a preference, and the procedure for making it is in verifier vs LLM judge.

What makes a check approved rather than just written?

Six conditions. We treat all six as required before a check goes into a run record.

1. A named owner. A person, not a team. The person who answers for the process when it goes wrong.
2. A written pass condition. Stated as a sentence about state, in the owner's language, not as code the owner cannot read.
3. A sign-off date. Recorded, because the condition will change and old records have to be readable against the condition that was in force when they were written.
4. A version. Stored with every verdict the check produces. Cheap at write time, unrecoverable later.
5. A stated error rate, or none needed. Computed checks state that they are deterministic. Classified and judged checks state holdout accuracy and agreement. A check with no error-rate line is not finished.
6. An asymmetry decision. The owner has said which direction of error is worse. Most business processes can absorb a false fail and cannot absorb a false pass, and the thresholds should reflect that rather than defaulting to whatever maximizes a balanced accuracy score.

How is an approved check different from a test, a guardrail, or an eval?

The rows are complementary. Keep your tests, keep your guardrails, keep your evals. None of them tells you whether last Tuesday's invoice was reconciled correctly.

Why does verification need to be external?

Verification needs to be external because the model cannot reliably catch the error classes it reliably makes. The intuition that checking is easier than doing comes from computational complexity and does not transfer to systems working by approximate retrieval. Testing across Game of 24, graph coloring, and STRIPS planning, Stechly, Valmeekam, and Kambhampati report "significant performance collapse with self-critique" and "significant performance gains with sound external verification" (On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks).

The model-as-judge alternative has documented biases from the paper that introduced it - GPT-4 judges reach over 80% agreement with human preferences while exhibiting "position, verbosity, and self-enhancement biases, as well as limited reasoning ability" (Zheng et al.) - which is why the judged tier is the last resort rather than the default.

And the failure this all guards against is quiet. Apica's description is the clearest version: "An AI agent returns HTTP 200. The JSON is valid... Every dashboard shows green. The agent hallucinated its findings, skipped a required tool call, and hid the failure behind a well-formatted answer" (Your AI Agent Passed Every Test. It Was Still Wrong). A well-formatted wrong answer passes every check except one that reads the system.

What does an approved check look like in practice?

Take the three-way match in accounts payable, which is the cleanest computed check in a common process.

The step: match the extracted invoice against its purchase order and its goods receipt. The pass condition, as the controller wrote it: for every line, invoiced quantity equals received quantity, invoiced unit price equals PO unit price, and the invoice total equals the sum of its lines within the tolerance the controller set. Owner: the AP controller. Kind: computed. Error rate: deterministic. Asymmetry: a false pass posts a wrong payable, so no tolerance widening without the controller's sign-off.

The same process has a step that will not compute. "Is this exception note clear enough for the AP clerk to act on" is a judged check, trained on a few hundred notes the AP team labeled actionable or not, with agreement reported on held-outs and a human reviewing whatever the judge is unreliable on. Two steps, two tiers, one signature each. Worked end to end, that is how to verify AI agent work.

When a check fails, the run stops at that step and resumes at that step once the problem is fixed. That is what makes long processes economical and what turns "the agent got it wrong" into "step 6 failed on 2026-07-14, here is the verdict and the check version". Accumulated, those verdicts are the verified agent audit trail and, separately, a training set the customer owns. How that becomes an engagement is on RL environments as a service.

Can you use this term?

Yes. Use it, and cite this page if it is useful - the definition above is the one we hold ourselves to, and we would rather the vocabulary spread than stay proprietary. If you use it differently, say so, because the failure mode we are trying to prevent is a term that means "we check things somehow".

Related definitions, including the computed/classified/judged hierarchy and run records, are in the glossary. If you want to know which steps of your process can carry a computed check, bring one process to [email protected]. If most of them cannot, we say that on the call.