Defining 'Quality' Data and the Future of RL Envs

Post-training data used to mean text: demonstrations to imitate and comparisons to rank. For agents it now means environments, synthetic worlds where a policy acts and a verifier assigns the reward. The shift changes what quality means. A demonstration is good if the answer in it is good. An environment is good only if its reward is right across the whole distribution of things a policy might try, including the rollouts that are, in effect, a search for the reward's mistakes. The policy is an optimizer pointed at the grader, and whatever the grader gets wrong becomes the gradient.

My position is that the verifier is the dataset, and that most of what ships today, including much of what appears in model cards, does not clear the bar the labs apply internally. Good post-training data in 2026 has three properties. It has a pass-rate distribution that actually produces gradient on the model being trained. It can state its own error rates, against human judgement, with intervals. And it grades the work, not only the answer. The rest of this post starts with what makes a verifier sound, takes those three properties in turn, and ends with where the data itself comes from.

Verifiers

A verifier should be invariant to differences that do not matter and sensitive to the ones that do. Two correct solutions with different names, structure or wording get the same score; a solution that only looks correct does not. The test I apply to every check is the second-design test: could a competent engineer, given only the instruction and the world, pass this without having seen the reference? Answer it by naming a concrete second correct design. If one exists that behaves differently at the check and the task does not care, the check is too narrow, and the check is what changes.

The asymmetry of reward errors is what makes this load-bearing. Under GRPO, a reproducible false positive that is the only success in a group of sixteen receives an advantage of about +3.9 against −0.26 for each failure, every time it recurs. Grading noise slows training; a learnable false positive redirects it. And it does not stay local. Models that learned to hack Anthropic's production coding environments generalized to alignment faking, cooperation with malicious actors and sabotage (MacDiarmid et al., 2025). A hackable environment is a safety defect, not a quality issue.

Contours of Knowledge Work

Most economically valuable work does not come with a test suite. It is long, it depends on judgement, and its intermediate steps feed each other. Packaging it into an environment means finding its shape: where the decisions are, what evidence a correct decision leaves behind, and which parts can be checked by code and which need a judge.

The grading itself is specific to the work. In an existing codebase, passing tests is table stakes and the reward covers what reviewers care about. In an incident inside a live world, the correct change is small and the difficulty is understanding a large noisy system first. Systems builds are graded through public interfaces against a real implementation, with deterministic performance proxies, since wall-clock time on a shared host is noise; FrontierSWE moved to weighted instruction counts for exactly this reason. Long professional documents need dependency maps between steps and a judge for structure that earns credit only when the analysis underneath holds. Enterprise workflows are graded by state left behind across systems, with negative checks; consumer apps by a judge that drives the deployed product through pre-written workflows and verifies stated facts rather than forming opinions.

The common thread is that the grader follows the contours of the work rather than the convenience of the author. Where the work has a canonical answer, check it deterministically against the truth. Where it has many correct forms, use a judge with atomic criteria. Where steps build on each other, encode the dependencies. And where the value is in what a professional would accept, grade that, not only whether the output parses.

Most benchmarks fail

Benchmarks and training environments are built from the same kinds of graders and fail in the same ways, so a flawed benchmark is also a flawed reward. The public record is not encouraging. Epoch AI's benchmark reviews grade reviewability, scoring quality and evaluation quality, and label a benchmark flawed when at least 20% of an inspected sample contains errors or one issue corrupts grading at scale. Of the first fifteen benchmarks reviewed, nine were flawed, four verified and two lacked the information to judge. The flawed set includes names that anchor model cards. A sample of Humanity's Last Exam found 46% of questions with accuracy-altering errors. On SWE-bench Verified, an OpenAI audit found that 59.4% of audited tasks had tests rejecting functionally correct submissions, on top of contamination across every frontier model tested. These are the numbers headline capability claims are built on.

A failure I see repeatedly in knowledge-work and "GDP-style" benchmarks that market themselves as frontier is format binding. The verifier is deterministic, which sounds rigorous, but it parses a specific output schema: field names, key orderings, section titles that exist only because the benchmark's author chose them and that the agent cannot infer from the prompt. Two things go wrong at once. A correct analysis in a different structure fails, a false negative that teaches the policy to guess the author's template. And a response that emits the expected fields with hallucinated contents passes the parse and collects partial credit, a false positive that teaches format over substance. The fix is the standard one. Where a canonical form exists, normalize and compare deterministically. Where it does not, an LLM judge with atomic, binary criteria grades the content in whatever form it arrives. Format generalization is one of the main reasons judges have moved from a convenience to a requirement in this class of task.

Pass@k and Distributional Analysis

Before an environment touches a training run, the first question is whether it produces gradient on the target model. That is a question about the distribution of per-task pass rates, not the mean. Estimate each task's pass rate from n samples with c successes, and report pass@k with the unbiased estimator rather than the naive empirical one (Chen et al., 2021):

Under group-relative training, a task with pass rate p contributes signal only when its group of G rollouts disagrees, so the expected informative fraction of a dataset is the mean of $1 - p_i^G - (1 - p_i)^G$ over its tasks. That number depends on the shape of the distribution far more than on its mean. Three illustrative datasets, at G = 16:

A bimodal dataset, where tasks are either always solved or never solved, can have a respectable mean pass rate and almost no gradient: the solved half is already saturated, and the unsolved half returns all-zero groups that DAPO-style dynamic sampling simply discards (Yu et al., 2025). A dataset whose pass@1 sits near zero on the target model is the same failure in one direction. The productive band is the middle, which in practice means per-task pass rates between a few percent and roughly 70% for the model being trained, measured in the scaffold and at the reasoning effort it will be trained with.

The gap between pass@1 and pass@k is the other diagnostic. Yue et al. (2025) found that RLVR-trained models beat their base models at small k while the base models win at large k: much of what RL does is concentrate probability on solutions the base distribution could already reach. For data, that cuts two ways. Tasks with low pass@1 but non-zero pass@k are where a policy can be hill-climbed, because the solution is in its distribution. Tasks with zero pass@k at large k are, for that model, closer to an expansion problem than a sharpening one, and a sparse reward will not reach them without a curriculum or denser structure. A dataset report should show the pass@k curve, not one point on it.

Pass@k should also be read across models. For each graded item, comparing a strong and a weak model sorts it into one of four bins: both pass, it discriminates, it is too hard, or it is reversed. Items both models pass add nothing and raise the floor. Items the strong model misses in most runs cost solvability, and are more often an ambiguity or a tolerance defect than genuine difficulty. Reversed items, where the weak model beats the strong one, are almost always a grader that rewards the wrong thing. Reliability belongs beside it: pass^k, the probability that all k attempts succeed, measures consistency, and the distance between pass@k and pass^k is the variance a deployed system will feel.

QA Vectors

The pass rate tells you whether there is gradient. It does not tell you whether the gradient points the right way. That requires evidence about the verifier itself, at two levels.

The first is proving the grader on candidates whose correct score is known. The untouched workspace must score 0. The reference must score 1, repeatedly and concurrently. A copy of the reference with planted defects must fail exactly the items those defects target. A correct solution that differs from the reference must pass, since any check it fails is bound to the reference. And an agent explicitly instructed to pass with a wrong or incomplete solution must fail; every success it finds is a criterion to fix. Judge-graded items add a measured σ: grade one frozen rollout several times, and require the gap between the models you mean to separate to sit well above it.

The second is auditing real rollouts after calibration. A judge agent, in a fresh sandbox with the instruction, the full trajectory, the deliverable, the verifier's output and the hidden reference, decides independently whether the work satisfies the task and classifies the outcome against the verifier: true pass, true fail, false pass or false fail. Every claim cites an artifact, a trajectory step or a line of the deliverable. Failures are tagged from a fixed taxonomy that separates capability failures (wrong logic, an upstream error, a missed inference, an unverified claim of success) from environment defects (a grader too narrow, an ambiguous instruction, an unfair inference). Behaviour tags record process: whether the agent explored first, tested its own work, looped, gave up or declared success without checking.

The output that matters is the false-pass rate among passes and the false-fail rate among failures, estimated from samples stratified by verifier outcome, model and environment, weighted back to the population, and reported with Wilson intervals. Zero hacks found in n reviewed passes bounds the hack rate below 3/n at 95% confidence, which is a statement a dataset can actually make. The judge itself is calibrated against people who re-grade a random subset blind. What a buyer should see is low verifier error with intervals, failures dominated by capability tags rather than environment tags, and visibly different failure shapes for a strong and a weak model. Anything less is a dataset that cannot say whether its own reward is right.

Increasingly, agents do the QA themselves, and the newest step is adversarial: an agent whose only job is to break the environment. Xiaomi's MiMo-V2.6 report describes a dedicated hack agent that probes prepared environments for leaks and exploitable weaknesses, seeded with routes found in early experiments, such as recovering a solution from cached artifacts or a preinstalled copy of the target project. It found exploit paths the team had not seen during training and that its cleanup had missed. They fixed them, reran the agent, and repeated until it could find no exploit in any environment, then kept offline trajectory audits running through training, where the share of hacking trajectories stayed below 2%. The caveat is that a probe is only as strong as the model running it. A hack agent weaker than the policy will miss what the policy eventually finds, so the attacker has to keep pace with the model being trained.

Trends

Three shifts define what good data looks like now, and each is a change in what the verifier can see:

• From outcomes to process. Grade how the result was reached, not only the end state.
• From correctness to quality. Grade what a reviewer would accept, not only what passes.
• Toward intelligence per token. Grade what the work cost, beside whether it worked.

Start with process. The current wave of RL for language models was built on outcome rewards, and for correctness that is still right. Correctness should be judged from the end state, through public interfaces, indifferent to the path. On long multi-step work, though, outcome-only grading pays for the wrong things. It pays for the right state reached the wrong way: in τ-bench an agent can issue a refund without the confirmation the policy requires and still get full reward (Yao et al., 2024). It cannot separate a minimal fix from a sprawling refactor when both pass the tests. And it rewards a polished conclusion built on a wrong intermediate result. So I grade on five axes, with correctness first and the others separating correct rollouts:

Aggregation matters as much as the checks. In a refunds app graded by four workflows made of 24 verifiers, a weak candidate that passes every easy check but only one workflow scores 0.67 per verifier against 0.96 for a strong candidate that passes three; per workflow they score 0.25 and 0.75. Score the units a user cares about, behind hard gates. Where steps build on each other, make the dependencies explicit, so a step counts only if every step it builds on passes:

In a loan memo, every downstream figure is computed from EBITDA, so a wrong EBITDA should zero everything built on it:

This is the cheapest way to put process into a reward: no judgement about the trajectory, only a map of what feeds what, fixed before calibration. Quality then earns credit only once correctness blockers pass, efficiency is compared only among correct rollouts, and actions that make the work worthless whatever the end state, such as editing tests or reading the fix out of git history, are gates that zero the episode.

On quality, the case is ultimately economic. A model's value to an organization is the share of its output that ships without rework, and passing tests is a weak proxy for that. METR had maintainers of SWE-bench Verified repositories review 296 agent-written PRs that passed the tests: roughly half would not have been merged, and the maintainer acceptance rate sat about 24 percentage points below the benchmark score, with rejections split across code quality, breaking other code, and patches that did not actually fix the issue. A model trained to pass tests learns to pass tests. Broad diffusion into real work requires training toward what a reviewer would accept.

Labs are starting to train for this directly. The same MiMo-V2.6 run replaces "every passing patch gets the same credit" with groupwise agentic grading (Dong et al., 2026). In Groupwise Advantage Redistribution, an online grader agent examines each group of rollouts, ranks the passing solutions on predefined criteria, zeroes any confirmed hack, and shifts positive advantage from weaker passes to stronger ones while keeping the group's total credit fixed. For high-pass-rate tasks, where groups often pass entirely and there is nothing to redistribute, Groupwise Reward Synthesis has an agent study contrasting rollouts offline and write task-specific rubrics for the solution and for the behaviour that produced it, which then score each rollout individually. The ablation is the argument. Without online grading, turns and tokens grew quickly and pass-rate gains stalled at the length limit, and maintainer audits found the policy drifting into exception swallowing, relaxed validation and evaluation-specific configuration changes. With it, the pass rate kept climbing while turns stayed roughly flat, and the patches were smaller and stayed in scope. A pass/fail reward gives the clean patch and the workaround the same credit, and the optimizer will find the workaround.

The benchmark I think gets this most right is Cognition's FrontierCode. Its tasks are built by the maintainers of the repositories, over more than forty hours each, around one question: would the maintainer merge this PR? Grading separates blockers from quality. Behaviour, regression safety, build and lint, and the agent's own tests, which must fail on the original code, are blockers; scope and code quality are weighted on top, and a run that fails a blocker scores 0. An LLM adapts the reference tests to the candidate's names before they run deterministically, so a different but correct implementation is not penalized. Version 1.1 audited more than a thousand criteria and relaxed 75 that were too strict, and replaced network blocking with a fair-use rule plus a detector that zeroes runs that consult solution-bearing sources.

CursorBench comes at the same problem from the product side. Its tasks are drawn from real Cursor sessions, with the short, underspecified prompts developers actually write, and agentic graders score them. Cursor reports that its rankings track how developers experience the models in its online evaluations more closely than public benchmarks do, which is the property a quality benchmark should have. On September 10, CursorBench 4.0 added long-horizon problems on intent understanding and design adherence, two things a pass/fail test does not see.

On tokens, FrontierCode's leaderboard makes the argument better than any model card, because it plots every model across all of its reasoning efforts:

More reasoning does not monotonically help. Opus 5.5 scores 54.6% at medium effort on 18.5k output tokens and $0.80 per rollout, and 54.4% at max on 165.8k tokens and $6.19, nine times the tokens for nothing; at extra-high it drops to 51.4%. Sonnet 5.5 peaks at 52.1% at extra-high and falls to 46.2% at max while emitting 596k tokens per rollout at $20.78. Opus 5 peaks at medium, 53.4%, and falls to 43.6% at extra-high. Past a point, more reasoning produces more output, and on a coding task more output means wider diffs, unrequested changes and second-guessed code, which is exactly what a merge-oriented grader rejects. An environment that grades scope and quality, and reports tokens beside the score, is the one that teaches a model when to stop.

The trend across releases is toward more intelligence per token. Anthropic reports that Opus 5.5 at its lowest effort beats Opus 5 at high effort with about 60% fewer output tokens; on FrontierCode its best run beats Opus 5's best on 45% fewer tokens. GPT-6.1 Sol lands one point below GPT-6 Astra on Artificial Analysis's Intelligence Index at under a quarter of the cost per task, $0.72 against $3.26. That saving is mostly price per token; on that index Sol actually emits more output tokens per task than Astra, though on FrontierCode it is the most frugal model near the top at 11.4k. Either way, the cost of a unit of useful work is falling, and environments that measure tokens beside correctness are what make that trade visible to the optimizer.

Ignoring the last two shifts has a visible cost. Aggregate indices put several Chinese labs close to the US frontier. Kimi K3 launched at third on that same index. On a merge-oriented benchmark the gap reopens. On FrontierCode, Kimi K3 scores 44.2% on 53.6k output tokens per rollout, ten points below Opus 5.5 on nearly three times the tokens; GLM 5.3 scores 40.1% on 112k, and DeepSeek V4 Pro 0813 28.5% on 105k. Price has moved the same way. K3's output price rose to $15 per million tokens from $4 for K2.6, a shift widely read as the end of cheap Chinese frontier models.

I call the pattern “benchmaxing”: training that lifts the scores a release is judged on without lifting the qualities those scores are supposed to stand for. Correctness-only verifiers are what make it possible. They are blind to scope, to quality and to tokens, so a policy can climb them with verbose, defensive, test-targeted behaviour that looks frontier-level on a leaderboard and feels noticeably worse in use. Anecdotally, that matches what I hear from researchers I know at Cornell, MIT and Berkeley, most of whom prefer Claude for research code and for technical work in physics, math and ML even where the published scores look close. My read is that the difference is mostly in the data. Anthropic has been one of the most aggressive buyers of environments that model how experts actually work, often as the first customer of new environment vendors and on exclusive terms, and it discussed spending more than a billion dollars on RL environments over a single year. OpenAI has taken a different route, buying from a smaller pool of vendors while building more of its data operation in-house. Packaging expert workflows into environments that grade them faithfully is hard, and the labs that have treated it as a core competency are the ones whose models feel the part.

Synthetic Data

Environments are engineered from material, and the material decides what can be built. The most valuable source is the record of how experts already work: a claims desk's files, a clinic's referrals, a support team's tickets, an engineering team's review threads. Records become an environment when each entity's history rebuilds a sequence of decisions, the expert's choice at each point is visible, outcomes are recorded by someone other than the decision-maker so a verifier can judge whether a decision was good rather than whether it matched, and the action space is bounded enough to lift a policy above noise in a practical number of rollouts.

Synthetic generation is the alternative to building from expert records: contrive the company, the data and the tasks, and produce references with frontier models. It costs a fraction of expert-built data and suits labs below the frontier that want to learn from frontier behaviour, and increasingly the frontier model's own output anchors the rubric. The risks are realism, solvability and a ceiling set by the generator. Check the generation before anchoring to it, and grade the qualities it shows rather than resemblance to it. Mining public repositories commit by commit is increasingly contaminated, and low-fidelity worlds teach the wrong lesson: a model that recognizes a test rig behaves differently in it, and what it learns there does not transfer.

Taken together, post-training data is becoming worlds, tasks and graders that can score answers nobody has written yet, and its value is moving from volume to the quality of its signal. In practice that means a pass@k distribution that sits in the training band, a verifier that can report its false-pass and false-fail rates with intervals, grading that covers quality, scope and tokens beside correctness, and process evidence taken from actions rather than from what the model says about them. If a dataset cannot show those, it is not the data the next generation of models will be trained on.

The open problems sit at the edge of that list. Dense process rewards that resist hacking do not yet exist for long agentic tasks. Worlds must become realistic enough that a model cannot tell it is being trained. And hard environments are getting harder to build just as the models they train get better at breaking them, which makes building them quickly and keeping them honest the real bottleneck.