RL environments glossary

Plain definitions for the vocabulary of RL environments, agent training, and verified AI work. Written by Pebble, a machine intelligence lab that builds verified environments for business processes. Each definition stands alone, so you can link to any term directly.

Environments and tasks

RL environment

An RL environment is a software world in which an agent acts, observes the result of each action, and receives a reward signal that scores its behavior. In classic reinforcement learning the environment is a game or simulator. In modern LLM training the environment is increasingly a replica of real software or a real business process, where the agent's "moves" are tool calls and the reward comes from checks on its work.

Task

A task is one unit of work an agent is asked to complete inside an environment, with a defined starting state and a defined standard for success. "Reconcile this invoice against its purchase order" is a task. Environments are typically sold and priced per task or per taskset.

Taskset

A taskset is a collection of related tasks used together for training or evaluation, usually spanning a range of difficulty and edge cases so an agent cannot pass by memorizing one pattern.

Step map

A step map is the decomposition of a process into numbered steps, each with its own inputs, outputs, and check. It is the blueprint of a verified environment: when a run fails, the step map is what turns "something went wrong" into a step number. See how a step map works in practice in invoice reconciliation.

Sandbox

A sandbox is an isolated execution context where an agent can act without touching production systems. Sandboxes make it safe to let an agent run code, click through software, or write to a database, because everything can be reset.

Gym

A gym is an informal name for an RL environment or a collection of them, after OpenAI's Gym library (now maintained as Gymnasium). Vendors use "gym" loosely for high-fidelity replicas of real applications built for agent training. Rockie, the environment Pebble maintains for academic HPC research, is a live example.

Harness

A harness is the code that connects a model to an environment: it feeds observations in, executes the model's actions, enforces limits, and records what happened. The same environment can be driven by different harnesses for training, evaluation, or production. See agent harness for the full disambiguation, and harness vs environment for where the boundary sits.

Runs and records

Rollout

A rollout is one complete attempt by an agent at a task, from starting state to termination, recorded as the sequence of observations, actions, and rewards. Training pipelines generate many rollouts per task and learn from the difference between the ones that pass and the ones that fail.

Trajectory

A trajectory is the recorded path of a single rollout: every state the agent saw and every action it took, in order. Trajectory data is what post-training pipelines actually consume.

Run record

A run record is the audit-grade log of one run through a verified environment: each numbered step, the check applied at that step, and the check's verdict. Run records serve two jobs at once: they are the verified agent audit trail for compliance and postmortems, and accumulated run records become a training dataset. In Pebble's engagement ladder the run records are the durable asset the customer owns.

Resumable run

A resumable run is a run that, when a step fails its check, stops at that step, gets fixed, and continues from that step rather than starting over. Resumability is what makes long multi-step processes economical: one bad extraction at step 3 does not throw away steps 1 and 2.

Long-horizon task

A long-horizon task is one requiring many dependent steps over an extended interaction, where early mistakes compound. Long-horizon work is where unverified agents fail quietly and where step-level checks matter most.

Verification and rewards

Verifier

A verifier is the component that decides whether an agent's output at a given step meets the standard. Verifiers range from code that checks an exact condition, to trained classifiers, to judge models scored against human annotations. The reliability of an environment is bounded by the reliability of its verifiers. How to build one is grader design, and how to choose between a deterministic verifier and a judge is in verifier vs LLM judge.

Verifiable reward

A verifiable reward is a reward signal computed by a deterministic check rather than estimated by a learned model or a human rating. "The totals reconcile" and "the tests pass" are verifiable rewards. They are the foundation of reinforcement learning with verifiable rewards.

RLVR

RLVR (reinforcement learning with verifiable rewards) is post-training in which the reward comes from programmatic checks on the model's output, rather than from a learned reward model as in RLHF. RLVR removes the most common failure mode of learned rewards, which is the model exploiting the reward model instead of doing the work.

Reward model

A reward model is a learned model trained on preference or rating data to predict how good an output is, used as the reward signal in RLHF. Reward models are flexible but exploitable: an agent can learn to score well without being right.

Reward hacking

Reward hacking is when an agent finds a way to maximize its reward signal without achieving the intended outcome, such as deleting a failing test instead of fixing the code. Environments defend against it with computed checks, held-out evaluations, and judges audited against human agreement.

Rubric

A rubric is an explicit list of criteria a judge (human or model) uses to score an output. Rubrics turn "does this look right" into checkable parts, and make judge behavior auditable. Designing and testing those criteria, and knowing when one should become a deterministic check instead, is rubric engineering.

Judge model

A judge model is a model used as a grader, scoring outputs against a rubric or against examples annotated by humans. A judge is only trustworthy relative to a measured agreement rate with the humans it emulates on held-out examples. The trade-off against a deterministic check is in verifier vs LLM judge.

Computed, classified, judged checks

Computed, classified, and judged checks are the three-tier hierarchy Pebble uses to verify each step of a process. Computed checks are decided by code alone (do the totals reconcile). Classified checks use a small model trained for that one decision and scored on held-out examples. Judged checks use a judge trained on operator annotations, with judge-human agreement reported, and a human stays in the loop where agreement is too low. The order is a preference: push every check as far toward computed as the work allows. The full definition of what makes a check approved is in approved checks.

Holdout set

A holdout set is data kept out of training and used only to measure whether a classifier or judge actually works. Trusting any learned check without a holdout score is guessing.

Training

Post-training

Post-training is everything done to a model after pretraining to shape its behavior: supervised fine-tuning, RLHF, RLVR, and environment-based RL. In the environments economy, post-training is the demand that environments supply.

SFT vs RL

SFT (supervised fine-tuning) teaches a model by imitation of example outputs, while RL teaches it by reward on its own attempts. SFT is simpler and data-efficient. RL can surpass its examples because the model learns from outcomes, which is why environments with reliable reward signals matter.

GRPO

GRPO (Group Relative Policy Optimization) is an RL algorithm, popularized by DeepSeek's reasoning models, that scores each rollout relative to a group of rollouts on the same task instead of using a separate value network. Its practicality made large-scale RLVR training far cheaper to run.

Environment engineering

Environment engineering is the discipline of building environments: mapping the process, designing verifiers, replicating the software surface, seeding realistic data, and maintaining all of it as the real process drifts. It is the skill RL environment companies actually sell, and it is covered in full in environment engineering.

Agentic evaluation

Agentic evaluation is measuring an agent's ability to complete multi-step tasks in an environment, rather than scoring single responses. Benchmarks like SWE-bench and OSWorld are agentic evaluations. The same environment built for training doubles as the evaluation of record for the process it models.

Computer-use agent

A computer-use agent is an agent that operates software the way a person does, through the screen, keyboard, and mouse, rather than through APIs. Computer-use environments replicate real applications so these agents can be trained and evaluated without touching production.

Missing a term you expected? Write to [email protected] and we will add it. If you want to see these ideas running against a real process, start with RL environments as a service.