Rubric engineering: fewer criteria, sharper checks

A rubric is where teams put everything they could not turn into code. That makes rubric quality the ceiling on the reliability of the whole system, and it makes the most common rubric habit - adding criteria until it feels complete - the fastest way to lower that ceiling.

What is rubric engineering?

Rubric engineering is the practice of designing, testing, and revising the criteria a judge uses to score an output, treating the rubric as a measured instrument rather than a document. The instrument framing is the useful part. A rubric has an agreement rate, a spread across repeat runs, and a failure mode, exactly like a classifier, and none of those numbers exist until someone measures them.

The academic definition matches. A survey of rubric use across LLM evaluation defines rubrics as "explicit criteria sets that transform complex quality judgments into structured and actionable standards" and gives four properties (From Holistic Evaluation to Structured Criteria, Chen et al., June 2026):

• Explicitness - "criteria are articulated in natural language"
• Structuredness - "criteria are organized through explicit relationships"
• Decomposability - "criteria are partitioned into mutually independent units"
• Verifiability - "criteria are designed for independent and reproducible evaluation"

Read the fourth property again. Verifiability is in the definition of a rubric, which means a criterion that cannot be checked reproducibly is not a rubric criterion. It is a preference.

Why do 3 to 5 criteria beat 15?

Three to five criteria beat fifteen because judge reliability falls as the label space and the criterion count grow, and a rubric is only as good as its noisiest item. This has been measured. In a study of rubric-conditioned grading, moving from a binary pass/fail rubric to a five-way partial-credit rubric dropped agreement hard: "Accuracy dropped from 76% to 57%, while Kappa dropped from 0.51 to 0.34 from the binary task to the 5-way task", and performance was "inversely related to complexity" (Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness, Deng et al., Purdue, December 2025). The dominant error in the five-way condition was leniency, classifying partially correct answers as correct.

The rubric-modification study reaches the same place from the other direction: "higher criterion complexity and conservative aggregation methods tended to decrease human-autorater agreement", and "analytic rubrics do not consistently outperform holistic rubrics in aligning autoraters with human judgments" (Huynh et al., CMU and Google Research, May 2026). More structure is not free.

And the survey is blunt about the downside risk: "low-quality rubrics do not merely fail to provide useful signals but can actively mislead reward models and degrade evaluation accuracy". A vague fifteenth criterion is not a neutral addition. It is noise injected into every score the rubric produces, and if that score is a training signal it teaches the wrong thing.

The practical rule we use: each criterion must be binary, and there should be few enough of them that a single operator can hold the whole rubric in mind while annotating. In business processes that has meant three to five. If you need more than five, you almost certainly have more than one step.

What makes a criterion good?

Five tests. A criterion that fails any of them gets rewritten or cut.

1. Binary. Pass or fail, no five-point scale. Partial credit is where the agreement collapse in the Purdue numbers lives. If the underlying reality is genuinely graded, split it into two binary criteria at the thresholds you care about.
2. Independently checkable. Two annotators reading the criterion and the same output reach the same verdict without discussing it. If they need to discuss it, the criterion is underspecified and the discussion is the missing text.
3. Falsifiable by a real example. You can produce an actual output from your history that fails this criterion, and one that passes. Criteria with no failing example in the archive are usually restating the process rather than checking it.
4. About the output, not the effort. "Cites the policy section" is checkable. "Shows careful reasoning" is not, and it rewards verbosity, which is one of the biases the judge already has.
5. Owned. A named person signed off on the wording and the date is recorded. Otherwise you will not be able to reread last quarter's scores against the rubric that produced them.

When should a rubric criterion become a deterministic check?

A rubric criterion should be promoted to a deterministic check the moment its pass condition can be restated as a comparison between values that exist in your systems, and the promotion is the highest-value move in rubric engineering. Most rubrics contain two or three criteria that are already computable and were written as prose because the rubric was drafted by people describing their judgment rather than by people reading their databases.

Four promotion signals, in the order we look for them:

• The criterion mentions a number. "The total is correct", "the date is within the window", "the amount does not exceed the approval limit". Every one of these is arithmetic wearing rubric clothes.
• The criterion mentions a named field or artifact. "References the correct policy number", "attaches the receiving document". Existence and equality checks.
• Annotators never disagree on it. Perfect inter-annotator agreement on a criterion means it is not a judgment call. Perfect agreement is the promotion trigger, not a sign the criterion is well written.
• The judge's failures on it are all one shape. If the judge only ever gets a criterion wrong on documents in a particular format, that criterion is a parsing problem and code will do better.

What is left after promotion is the genuinely subjective residue - actionability, faithfulness, adequacy of a written response - and that residue is small enough to measure properly. The tier boundaries are defined in approved checks, and the per-step decision procedure is in verifier vs LLM judge.

How do you measure whether a rubric works?

You measure a rubric with three numbers, taken in this order, and the first one is the one people skip.

One: human-to-human agreement on the rubric. Two or three operators annotate the same fifty outputs independently. Their agreement with each other is the ceiling on anything automated. The Huynh et al. study found human-autorater agreement improved significantly where human raters were themselves more consistent, which is the empirical version of a plainer point: you cannot automate a standard your own team does not share. Snorkel's writeup notes even a heavily vetted benchmark like HealthBench landed at 55-75% agreement rates, so do not expect 95% and do not fake it.

Two: judge-to-human agreement on held-out examples. The judge scores examples the rubric authors never saw. Report per criterion, not just overall, because the aggregate hides the one criterion that is doing all the damage.

Three: stability across repeat runs. Score the same output several times and look at the spread. Pairwise LLM judging has been measured flipping "on average 13.6% of the time", with cross-judge agreement at 76% and kappa 0.51, and needing "11 repeated trials" for a majority vote to reliably recover a 50-trial reference verdict (The Coin Flip Judge?, June 2026). If you call the judge once and record the answer, that variance is in your results whether or not you measured it.

Rubric, checklist, or verifier?

Vendors selling rubric tooling mostly work the middle column. Labelbox builds rubric-based evaluation with domain-expert annotator networks for frontier-model work, and Snorkel has the most careful public treatment of rubric structure and scale design we have found. Both are aimed at model evaluation at lab scale. Our interest is narrower and is mostly about emptying the middle column into the right-hand one for a single company's process.

How do we build a rubric?

Five steps, and step four is the one that separates a rubric from a wish list.

1. Pull thirty real outputs from history, half of them ones the team rejected.
2. Ask two operators to write, independently, why each rejected one was rejected. The overlap is your criteria. The non-overlap is your disagreement, and it goes back to the process owner before anything else happens.
3. Draft criteria as binary conditions. Stop at five.
4. Have both operators annotate a fresh fifty against the draft. Compute their agreement. Rewrite any criterion below the bar and repeat until the humans agree with each other.
5. Only then build the judge, hold out a third of the annotations, and report per-criterion agreement and repeat-run spread next to every score the judge produces.

Grader-level mechanics for step five are in grader design, the run-level method is in how to verify AI agent work, and the definitions of rubric, judge model, and holdout set are in the glossary.

If you have a rubric that already exists and nobody trusts, bring it to a scoping call at [email protected]. The most common finding is that two of the criteria should be code and four should be deleted.