Reinforcement Learning with Verifiable Rewards (RLVR)

Most guides to RLVR stop at math and code. This one covers the part practitioners actually ask about: what to do when the work you want a model to learn has no unit test.

What is RLVR?

Reinforcement learning with verifiable rewards (RLVR) is a post-training method where a model is rewarded only when its output passes an objective check, such as a passing test suite or an exact-match answer, rather than being scored by a learned reward model or a human preference signal. The model attempts a task, a program checks the result, and the reward is a function of whether the check passed. The weights update toward whatever behavior passes the check.

The term comes from the Tülu 3 paper, published by the Allen Institute for AI in November 2024, which described RLVR as a training method alongside supervised finetuning and DPO. Two months later, DeepSeek-R1 showed that reinforcement learning against rule-based rewards on math and code, without human-labeled reasoning traces, was enough to produce strong reasoning behavior, including self-reflection and verification patterns that emerged during training rather than being demonstrated to the model.

The common thread in both papers is that the reward came from a check, not a judgment. A math answer either matches the reference or it does not. Code either passes the tests or it does not. That property, an answer that can be checked programmatically, is what "verifiable" means here.

Why do verifiable rewards beat learned reward models?

Verifiable rewards beat learned reward models because a computed check is much harder to game than a model's opinion. When the reward signal is itself a neural network, the policy being trained will find inputs that score well without doing the work. This is reward hacking, and people who build training environments for a living treat it as the central quality problem. In Epoch AI's survey of the RL environments industry, one researcher put it directly: "Reward hacking is a big issue. The model might cheat by searching up a solution, or if you're not careful with how you script the repo, by checking out future commits." The same survey notes that robust graders rarely work on the first pass and take many iterations to harden.

A learned reward model has a second failure mode: it degrades silently. When a policy drifts into a region the reward model was never trained on, the scores keep coming but stop meaning anything. A computed check does not have this problem. "Do the totals reconcile" returns the same answer no matter how strange the policy's behavior gets.

None of this makes verifiable rewards free. The Epoch survey's examples above show that even code-based graders get hacked when the environment is sloppy, for instance when a repo's git history leaks the answer. Verifiable rewards raise the cost of gaming the signal. They do not remove the need to build the environment carefully.

What happens when work is not natively verifiable?

When work is not natively verifiable, the practical answer is a hierarchy of check types ordered by how much you have to trust a model, and you push every check as far up the hierarchy as it will go. This is the pattern Pebble uses to build training environments for business processes, where most steps do not come with a test suite attached:

• Computed checks. Verified by code, no model grading anywhere. Do the totals reconcile, does the record exist, do the two systems agree. This is the gold standard, and more of a business process is computable than most people assume once the process is broken into small enough steps.
• Classified checks. A small model trained to make one narrow judgment, for one check, and scored against held-out examples before it is trusted. Not a general-purpose judge. A single-purpose classifier with a measured error rate.
• Judged checks. For steps where the pass condition is genuinely a matter of judgment, the customer's own operators annotate a few hundred pass/fail examples, and a judge model is trained on them. The judge's agreement with the human annotators is measured on held-out examples and reported as a number. Where agreement is too low, a human stays in the check.

The ordering matters. Every check moved from judged to classified, or from classified to computed, removes a place where reward hacking can start. Definitions for these and related terms are in the glossary.

How do you apply RLVR to a business process?

You apply RLVR to a business process by mapping the process into steps, attaching a verification check to each step, and then treating every run of the process as a training episode. Pebble's version of this: a company brings a process, Pebble maps it into steps, defines a check for each step using the hierarchy above, and delivers the environment as software the customer hosts or Pebble hosts. When a run fails a check, it stops at that step and resumes there once fixed, rather than restarting from zero. The run records accumulate into a dataset the customer can use to post-train an open-source model they own, with passing runs weighted more heavily.

Take invoice reconciliation as a concrete case. The process decomposes into steps like extracting line items, matching them against the purchase order, and posting to the ledger. "Do the extracted line items sum to the invoice total" is a computed check. So is "does the payment amount match the approved amount." A step like "is this discrepancy note written clearly enough to act on" cannot be computed, so it becomes a judged check trained on the operators' own pass/fail annotations. The result is an invoice process where every step has a defined pass condition, which is exactly the structure RLVR needs.

This is also why environments are the expensive part of applied RLVR rather than the training run. What building one costs is covered in our breakdown of RL environment pricing.

Is RLVR the same as RLHF?

No. RLHF trains a reward model on human preference comparisons and then optimizes the policy against that learned model, which is exactly the gameable component RLVR avoids. RLVR replaces the learned reward model with an objective check wherever one can be constructed. RLHF asks "did a model trained on human preferences like this output." RLVR asks "did this output pass the test." The two are compatible, and labs use both, but they fail differently: RLHF fails by being gamed, RLVR fails by not covering work that has no check, which is the gap the computed/classified/judged hierarchy exists to close.

Does RLVR require a simulator or game engine?

No. An RL environment is just a task, a set of actions, and a grader. For business processes, the environment is the process's own systems, real or replicated, plus the checks. No physics, no rendering.

Can reward hacking still happen with verifiable rewards?

Yes, if the check has holes. The Epoch AI survey documents models exploiting sloppy environment setup, such as reading future commits out of a repo's history to pass software engineering tasks. The fix is engineering discipline in the environment, not a better prompt for a judge.

Where should you start?

Start with one process and find its computed checks. If you want a second opinion on whether your process decomposes into verifiable steps, book a scoping call with the founder who builds the environments at [email protected]. If an environment is the wrong tool for the job, we say so on the call.