Grader design: turn your QA checklist into code
Search "grader design" and you get motor graders and academic grading. The AI sense of the word is newer, it comes out of the eval and fine-tuning tooling, and almost everything written about it assumes you are training a model. This page is about the other use: you have a written process with a QA checklist, and you want the checklist to run.
What is grader design?
Grader design is the work of turning a stated standard for correct output into an executable check that returns a score, and then proving that the check agrees with the people who own the standard. Two halves. Writing the grader is the visible half. Proving it agrees is the half that decides whether anything downstream can be trusted.
OpenAI's API docs define a grader as "a way to evaluate your model's performance against reference answers" that returns "a grade in the range from 0 to 1" (Graders guide). That is the training-time framing and it is where the word got standardized. In a business process the reference answer is usually not a stored string, it is the state of a system after the step, so the design work looks different even though the vocabulary carries over.
Why did "grader" become the standard word?
"Grader" became the standard word because OpenAI shipped it as an API surface with named types, and named types travel. The documented types are string check, text similarity, score model, Python graders, and multi, the last of which composes the others. Once "score_model" and "string_check" existed as JSON, every eval vendor had a shared vocabulary to argue with.
Worth knowing before you build on it: the same doc now carries the notice "OpenAI is deprecating graders as part of the evals and fine-tuning workflows they support." The concept is not going anywhere, the specific endpoint might. Design your graders as your own code with your own tests, and treat any vendor's grader schema as an export format rather than the place your logic lives.
The other reason to take the word seriously is that the people who use it professionally say it is the hard part. From OpenAI's own reinforcement fine-tuning cookbook: "Designing an effective grader requires both principled structure and thoughtful domain insight, and is perhaps the most important task for successful RFT" (Exploring model graders for reinforcement fine-tuning).
How do you turn an SOP into graders?
Six steps. This is the sequence we run when a process arrives as a Confluence page and a spreadsheet of QA criteria.
- Cut the SOP into numbered steps with one output each. If a step produces two artifacts, it is two steps. Most checklists fail here, because a line like "reconcile and post" hides two different pass conditions.
- For each step, write the pass condition as a sentence about state. Not "the extraction is accurate" but "every line item in the extracted record matches the corresponding PO line on quantity and unit price". The sentence is the grader's specification and it is what the process owner signs.
- Mark which conditions are computable. A condition is computable when the values it compares exist in a system you can read. Do this before deciding anything about models. In our experience the computable fraction of a process is larger than the operators expect and smaller than the vendors claim.
- For each remaining condition, write the smallest decision it requires. "Is this exception note actionable by the AP clerk" is one narrow binary decision. "Is this note good" is not a decision, it is a mood. Narrowing is what makes the next step possible.
- Collect annotations for the non-computable conditions. A few hundred examples per check, labeled by the operators who own the process, split into train and holdout before anyone writes a prompt. The holdout existing before the prompt exists is the whole point.
- Attach the graders to the steps and version them together. A run record is only rereadable if you know which grader version produced each verdict. Storing the version costs nothing at write time and is unrecoverable later.
The pattern applied end to end on a real process is in AI invoice reconciliation.
What are the grader types and when do you use each?
| Type | Verdict from | Use it when | What it needs before you trust it |
|---|---|---|---|
| State assertion | Reading the affected systems and comparing | The condition is about what is now true in the ERP, ledger, CRM, or filesystem | Read access to real state and a test on known-bad cases |
| Numeric tolerance | Arithmetic against an approved threshold | Totals, balances, rounding, quantity matches | A tolerance the process owner set in writing |
| Existence and uniqueness | Query for the record | Something must have been created, or must not have been created twice | Cases covering both directions, including the duplicate |
| String or schema check | Pattern, enum, or schema match | Format, required fields, controlled vocabularies | A corpus of real outputs, not synthetic ones |
| Trained classifier | A small model built for this one decision | The decision is narrow, repeated, and annotatable | A holdout accuracy number stated out loud |
| Model grader (judge) | A language model applying a rubric | The condition is about meaning and a human would otherwise read every output | Measured agreement with the annotators, plus repeat-run stability |
| Human | A person | Agreement is too low to automate, or the stakes make automation the wrong call | Nothing. It is the honest floor. |
The rows are ordered by how much you have to trust a model, and the design goal is to keep pushing checks upward. The decision procedure for the boundary between the last two model rows is in verifier vs LLM judge.
What makes a grader hold up on real runs?
Five properties. A grader missing any of them will pass in development and mislead you in production.
- It reads state, not the agent's report. If the grader's input is the agent's own summary of what it did, the grader is grading prose. Read the system.
- It has a stated failure direction. You know what a false pass costs and what a false fail costs, and they are not the same. Most business processes can absorb false fails and cannot absorb false passes, which should make your thresholds asymmetric.
- It was tested on known-bad cases. A grader that has only seen correct outputs has never demonstrated it can fail. Feed it the historical exceptions before you feed it anything else.
- It is stable across repeat runs. For model graders this means running the same input several times and looking at the spread. Non-determinism you have not measured shows up later as unexplained variance in your reliability numbers.
- It has an owner who signed it. A pass condition nobody approved is a vendor's opinion about your work. This is the property that turns a grader into an approved check.
What does a bad grader look like?
Bad graders fail in a small number of recognizable ways.
The gameable grader. OpenAI's cookbook documents the canonical case: "The model soon uncovered a loophole and began reward-hacking. Scores shot up - sometimes by 20-30 percentage points - not because clinical accuracy improved but because the model padded its answers with synonyms, doses, and full management plans." A grader that rewards presence rewards padding. Their fix is worth copying: clarify expectations, enforce output constraints, and supply "contrastive examples of correct versus incorrect behavior".
The tautological grader. The grader checks that the output has the shape the agent always produces. It passes everything, including the failures, and it looks like a green dashboard.
The grader with no holdout. A model-based grader whose accuracy has never been measured against withheld examples. It has a number in the docs and no evidence.
The grader that grades the wrong step. Attached one step too late, so it validates a downstream artifact and cannot tell you which upstream step went wrong. This is what makes failure investigations become transcript reads.
The compound grader. One check enforcing four conditions, returning one pass or fail. When it fails you learn nothing. Split it.
How do you know a grader works?
You know a grader works when it disagrees with the process owner less often than the process owners disagree with each other, measured on examples neither of you chose. That ordering matters: measure human-to-human agreement first, because it is your ceiling. Research on rubric edits found human-autorater agreement improved significantly when the human raters themselves were more consistent (Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement, Huynh et al., CMU and Google Research, May 2026). If two of your controllers disagree about what "reconciled" means, no grader will resolve it and the fix is upstream in the SOP.
Three numbers per model-based grader, reported together: holdout accuracy, agreement with the annotators, and spread across repeat runs on identical input. Computed graders need none of these, which is the argument for them. The rubric-level version of this measurement problem is in rubric engineering, and the run-level method is in how to verify AI agent work.
Where do you start?
Start with the QA checklist you already have and mark each line computable or not, because that split is the whole design and it takes an afternoon. Bring the marked-up checklist to a scoping call at [email protected] and we will tell you which lines we would build first and which ones we think should stay with a person.