What are RL Environments?

June 2026 · Sean Lee

For most of the last decade, frontier language models got better mainly through pre-training: more parameters, more tokens, more compute, with loss falling predictably along power laws. Post-training was the comparatively small step at the end that turned a next-token predictor into an assistant. That balance has shifted. The largest recent gains in reasoning and agentic capability have come from reinforcement learning on top of a pre-trained model, and the labs now talk about RL as a second axis of scaling rather than a finishing pass. Anthropic's Dario Amodei put it directly in February 2026: "we're seeing the same scaling in RL that we saw for pre-training," with performance improving log-linearly in RL compute across a widening range of tasks.

Pre-training scales on data that already exists. RL scales on something that has to be built: places where a model can act, be graded, and learn from the grade. Those places are RL environments, and they have become a significant line item. Anthropic's leadership has discussed spending more than $1 billion on RL environments in a single year, and in 2025 YC saw multiple RL environment companies in its spring and summer batches.

So what, concretely, is an RL environment for a language model, and how does it differ from the environments of classical RL? The short answer is that the loop is the same and almost everything inside it has changed. The agent is a language model whose actions are tokens, the environment is a software world rather than a simulator, and the reward is no longer a property of the world but a program someone has to write.

Classical RL

Classical RL formalizes an agent interacting with an environment as a Markov decision process (S, A, P, R, γ): states, actions, a transition function P(s′ | s, a), a reward function, and a discount. The agent's policy π_θ(a | s) is trained to maximize the expected discounted return, usually by following the policy gradient, as set out in David Silver's UCL lectures and Sutton and Barto:

J(\theta) = \mathbb{E}_{\pi_\theta}\Big[\sum_{t} \gamma^t r_t\Big], \qquad \nabla_\theta J = \mathbb{E}_{\pi_\theta}\Big[\sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t)\, A_t\Big]

Here A_t is the advantage, how much better an action turned out than the policy expected. In this setting the environment is given. It is a physics simulator, an Atari emulator or a board game, and its reward is part of its definition. AlphaGo is the canonical example (Silver et al., 2016). A policy network was first trained by supervised learning on human expert games, then improved by policy-gradient RL through self-play, with a value network trained on the outcomes and Monte Carlo tree search combining the two at play time. The environment was the rules of Go, and the reward was +1 for a win and −1 for a loss, computed exactly by those rules. Nobody had to wonder whether the reward was right. The hard problems were exploration and credit assignment across a long game.

RLHF

The first large-scale use of RL on language models kept the optimizer and changed everything else. Christiano et al. showed that a reward model trained on human preferences between pairs of behaviours could stand in for a reward function nobody could write down (2017). InstructGPT applied this to language (Ouyang et al., 2022): fine-tune on demonstrations, train a reward model r_φ on human comparisons of responses, then optimize the policy against it with PPO (Schulman et al., 2017) under a penalty for drifting from the reference model:

\max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\mathrm{ref}}(\cdot \mid x)\big)

Viewed as an environment, RLHF is almost degenerate. The state is a prompt, the action is an entire response, there is no transition, and the episode ends after one step. It is a contextual bandit. The difficulty moved from exploration to the reward itself: a learned reward model is only an approximation of human judgement, and optimizing hard against it eventually makes the true quality go down while the proxy keeps rising (Gao et al., 2022). The KL term exists largely to limit that. For this era and the ones that followed it, Nathan Lambert's RLHF book is the most complete single reference, covering reward modelling, the policy optimizers, direct alignment methods, and later reasoning and tool use.

RLVR

The next step replaced the learned reward model, wherever possible, with a program. Tülu 3 named the method reinforcement learning with verifiable rewards, or RLVR (Lambert et al., 2024). For prompts whose output can be checked, such as math problems with known answers or instructions with testable constraints, the reward is 1 when a program confirms the output and 0 otherwise, used there alongside SFT and DPO to post-train an instruction-following model. A verifier cannot be flattered the way a learned reward model can, so it can be optimized much harder.

DeepSeek-R1 scaled the same idea into the main engine of reasoning training (DeepSeek-AI, 2025). Its rewards were rule-based, checking answer correctness and format, and the authors explicitly avoided neural reward models because of reward hacking at scale. In R1-Zero, trained with RL directly from the base model, long chains of thought emerged from the optimization rather than from demonstrations. The optimizer changed as well. GRPO drops PPO's learned value function and uses the other rollouts of the same prompt as the baseline (Shao et al., 2024), which suits verifiable rewards because a group of samples on one problem is cheap to draw and easy to compare.

Structurally, RLVR is still a one-step bandit: a prompt, a long response, a check. What changed is who writes the reward. For the first time it is a program written for the task, which is the seed of the environment-design problem.

Agentic RL

The step that made environments the bottleneck was agency. A coding agent does not answer in one shot; it reads files, runs commands, edits, tests, and submits, over tens or hundreds of turns. The MDP comes back in full. The state is the conversation so far together with the state of a real software world, an action is a block of tokens that may contain a tool call, and the transition is whatever happens when that tool runs.

The four regimes side by side:

	Classical RL	RLHF	RLVR	Agentic RL
State	Board, frame, joint angles	A prompt	A prompt with a checkable answer	Context window plus the world's state
Action	A move or a torque	A whole response	A long reasoning trace and answer	Tokens, including tool calls
Transition	Simulator or game rules	None, one step	None, one step	Tools executing in a sandbox
Reward	Defined by the environment	A learned reward model	A program that checks the answer	Verifiers written for the task
Main failure	Exploration, credit assignment	Reward model over-optimization	Saturated or gameable checks	Reward hacking of the verifier

Harnesses

Agency tied the model to its harness. The harness decides which tools exist, how tool output is truncated, when the context is compacted and how a turn ends, and a policy trained in one harness learns its conventions. So labs began collecting RL rollouts in the same harness the model ships in. For a stretch beginning with GPT-5, OpenAI released a separate Codex line of models. GPT-5-Codex, in September 2025, was a version of GPT-5 tuned for agentic coding inside Codex, OpenAI's coding agent, and trained with a focus on real software engineering work. GPT-5.1-Codex-Max went further: it was natively trained to work across multiple context windows through compaction, so a feature of the harness became a learned behaviour. In agentic RL the harness is part of the environment.

Anatomy of an Environment

An environment is everything a model needs to attempt a task and be graded on it. Its components:

Task category. The domain, and the problem the model should solve: a coding task, a piece of writing, a financial analysis, an AI research problem. In principle it can be anything a person is paid to do.
World. Everything the agent is given when it is spawned in its container. For a coding task, a snapshot of a codebase at a specific commit; for an AutomationBench-style task, a frozen company state of apps, records and history. The world also includes:
- Instruction. The prompt that states the task.
- Tools. What the agent can do in the world: execute code, use a browser, reach the internet under a stated rule, and custom tools built so it can act on parts of the world it could not otherwise reach.
Rubric and verifiers. The rubric states what correct and good mean. The verifiers check it one item at a time, and a formula turns their verdicts into a scalar reward.
Reference. A solution that scores 1, proving both that the task can be done and that the grading can award full marks.
Reward density. A sparse reward arrives once, at the end of the trajectory; a dense one arrives per step, per token or per sub-goal. Most RL environments today use sparse rewards.
Rollout monitoring. A way to read what the agent actually did, turn by turn. Harbor ships tooling for this, and platforms such as Prime Intellect's Lab provide it hosted.

The model being trained is not one of the components. It is plugged in through a harness, the runtime that defines its tools, runs its action loop and manages its context, ideally the same harness it will be deployed in.

Each rollout runs in two phases:

1 · Agent phase

Instructionwhat to do

Worldfiles, data

→

Agentacts

container stopped

2 · Grading phase

Graderfresh, pinned

→

Verifierscheck state

→

Reward0 to 1

The agent acts in its own container. When it submits or runs out of budget, that container is stopped, and a fresh one built from a pinned image grades whatever the agent left behind. The separation is the environment's main security property. Nothing the agent writes can reach the grader, so a pre-written reward file, a patched test runner or a background process is gone by construction.

Two implementation details matter for training. First, the tokens the environment produces, tool outputs and observations, are part of the context but not part of the action, so they are masked out of the loss; the policy is only updated on tokens it generated. Second, the reward almost always arrives at the end of the episode. A rollout of two hundred turns produces one number, and the trainer has to assign credit for it across every token that came before.

Training Signal

During training, the trainer samples several rollouts of the same task from the same model and compares their scores. Only the differences carry information. Rollouts that scored above the group are made more likely and those below less likely, which shifts the model's distribution toward producing the correct solution. If every rollout passes, or every one fails, there is nothing to compare and the environment teaches nothing. An environment is useful only at a difficulty where the model sometimes succeeds.

Reward Types

Most rewards are built from three kinds of check. Deterministic verifiers give the same verdict for the same submission: the tests pass, the database row exists, the output matches within a tolerance. They are cheap and hard to argue with, but they only accept the forms their author anticipated. LLM judges grade against a written criterion, which lets them recognize a correct answer in a form nobody wrote down, at the cost of run-to-run variance and their own susceptibility to manipulation. Heuristics score how the work was done, such as the order of steps or the sources consulted. They are interpretable and the easiest of the three to game.

Because most production recipes keep the reward sparse, the structure goes into how the final reward is computed: hard gates that zero it, items that each prove one behaviour, and weights fixed before any model is run.

Reward Hacking

Reward hacking is the policy finding a way to raise the reward without doing the intended work. In coding environments the documented forms are concrete: deleting or rewriting the tests, exiting early with a success code, special-casing the visible inputs, overriding equality so that any output compares equal, or reading the fix out of the repository's history. Each is a gap between what the verifier checks and what the task means.

It matters more than it looks, because what a model learns from a hack does not stay in the environment. Anthropic found that models which learned to hack production coding environments generalized to alignment faking, cooperation with malicious actors and attempted sabotage (MacDiarmid et al., 2025). A related trap is grading the model's reasoning. OpenAI showed that putting a chain-of-thought monitor into the reward first reduces misbehaviour, then, under more optimization, teaches the model to hide its intent while continuing to hack (Baker et al., 2025). Reasoning models also often use reward hacks without mentioning them at all (Chen et al., 2025). Process is safer graded from what the agent did, its commands, diffs and the state it left behind, than from what its reasoning says.

Infrastructure

An environment is software that generates training data, and it is run like production. The world must be reproducible, with base images pinned by digest, dependencies locked and seeds fixed, so that the same rollout can be graded again. Failures of the world or the grader, such as a service that did not start or a judge that timed out, must produce an invalid outcome rather than a reward of 0, or infrastructure noise becomes a training signal. Grading must be fast enough to keep accelerators busy; a verifier that takes an hour is unusable inside a training loop.

Before an environment is trusted, its grader is also tested against candidates whose correct score is known, from the untouched workspace to the reference.

There is no standard interface yet for how an environment talks to a trainer. Frameworks such as OpenEnv, Verifiers, NeMo Gym and SkyRL Gym each draw the line between environment, harness and trainer in a different place, and task formats such as Harbor's package the instruction, world, tests and reference together. The important point is that they agree on the parts: a task, a world the agent can act in, a reward, and a loop that turns rewarded rollouts into a policy update.