Most coding benchmarks hand an agent a ticket and a failing test. This one hands it a whole repository and a behavioural target: a deployed binary, a newer release, or another language's implementation. Changes run to thousands of lines, hidden tests grade the final code, and only exact behaviour earns the pass.
All 630 graded runs in one picture. Each cell is one model on one task, colored by runs solved out of three. Rows follow the leaderboard, columns run easiest to hardest. The pale right side is the open frontier.
30containerized tasks across five difficulty tiers
7frontier models, each with its own CLI agent
630graded runs, three per model and task
12languages, from Python and Go to OCaml and Zig
54.4%top pass@3 on the sample, Claude Opus 5.5
LEADERBOARD
The leaderboard
The ranking below is computed from verifier rewards alone: a run scores 1.0 only when every hidden check passes, no judge model is involved anywhere, and each number on this page can be recomputed from the shipped CSVs.
Each task was built in three steps. First, a real codebase was frozen at a base commit and containerized. Second, the authors wrote a reference solution and a hidden differential verifier, validated in both directions: the reference scores 1.0 and the untouched repository scores 0. Third, each model ran the task with its own vendor's CLI agent at medium effort, three times, with only the shell as a tool.
Pass@3 is the fraction of a model's runs that fully solved their task. A run counts only at a verifier reward of exactly 1.0. Mean score credits partial progress on tasks with fractional rewards.
30 tasks · 29 codebases · 630 graded runs · October 2026 Tasks drawn from click, Typer, zod, dplyr, jq, jsoup, headscale, oasdiff and 21 more.
MODEL
PASS@3 BY TIER
WHERE ITS FAILED RUNS LANDED
± standard error, n = 90 runs
Click a model for its full analysis.Tap a model for its full analysis.
DESIGN
Why repository scale
Most coding benchmarks stop when an issue closes. A repository's work is only done when the whole codebase behaves as specified, and keeps behaving. Four decisions follow from taking that seriously.
DECISION 01
Give it work that takes real time
The 630 graded runs total about 350 recorded agent-hours, roughly half an hour of autonomous work per run, and the time climbs with tier. Reference patches span 137 to 20,048 changed source lines across up to 164 files.
An agent has to survey unfamiliar code, plan a campaign of changes, build repeatedly and debug its own regressions before the verifier ever runs. Scope is the strongest predictor of failure we measure: pass@3 falls from 87.1% on the smallest reference patches to 6.7% on the largest.
DECISION 02
Refuse to repeat one template
The corpus is not thirty variations on a single patch shape. Twenty tasks carry a library from one release to the next, seven demand byte-for-byte parity with a deployed binary or a sister implementation, and the rest build a feature, fix behaviour under constraints, or port a tool across languages.
Each ends in a different deliverable: a release that passes the new tests without losing the old ones, a build indistinguishable from the reference, a feature meeting nine stated behaviours at once. The sample spans 12 languages and 29 codebases.
DECISION 03
Keep the answer out of the container
Tasks start from public codebases, but the benchmark instance is built for this benchmark alone: the base commit, the behavioural contract, and the hidden corpus with its expected outputs. Recognizing the repository gives a starting point. It does not reveal the corpus a build must match.
The agent gets the repository, the instruction and a shell. The reference solution and verifier assets stay outside its view, there is no network on either side, and the verifier runs offline.
DECISION 04
Let the verifier have the last word
The verifier judges observable behaviour, never similarity to the reference patch. An agent can find the right subsystem and still score zero because one hidden case differs by a byte, one old test breaks, or its build secretly calls the reference binary.
Looking competent mid-run earns nothing. Reward 1.0 means the build succeeded and every targeted check passed, with invocations repeated to catch nondeterminism, and each parity task ships an exec-cheat control that has to score 0.
LANDSCAPE
Compared with what labs already run
Each family below measures a real slice of coding ability. Scroll to see what this sample asks instead.
FAMILY 01
Issue-level repair
SWE-bench and its descendants hand the agent a reported issue: a ticket that describes the bug and failing tests that localize it inside the repository. Solving it means producing the change those tests demand.
Here there is no ticket and no failing test to chase. The specification is a deployed binary, a newer release or a sister implementation, and the work spans many files and hundreds to thousands of lines. Finding what to change is part of the task.
FAMILY 02
Terminal agents
Terminal-Bench evaluates agents on containerized tasks drawn from many domains, scored by how many they resolve. It is a breadth measure, and this sample shares its Harbor task format and its offline, container-per-run discipline.
The difference is the question. Every task here is held to one behavioural contract: make the repository behave exactly like the reference, and keep everything that already worked working.
FAMILY 03
Graded by resemblance
Some evaluations score a change by how closely it resembles a reference diff, or ask a judge model whether the work looks right. Both make style part of the score, and both can be convinced by a confident-looking patch that does not run.
No patch is ever compared here and no judge model scores a run. The verifier builds the agent's code and tests its behaviour, so an unusual implementation that behaves correctly earns the same 1.0 as the reference.
RESULTS
Results by tier
Pass@3 for each model on each difficulty tier. The strongest model per tier is highlighted; tiers were assigned by the authors before any model ran.
swipe sideways for all seven models · first column stays put
Tier
✳GPT-6.1
AOpus 5.5
AFable 5.1
MWonder 1.0
MSpark 1.3
XGrok 4.7
GGemini 4
Trivial4 tasks
100%
100%
100%
100%
100%
100%
100%
Easy6 tasks
61.1%
72.2%
44.4%
77.8%
22.2%
61.1%
72.2%
Medium5 tasks
53.3%
60.0%
46.7%
46.7%
60.0%
46.7%
46.7%
Hard6 tasks
44.4%
61.1%
38.9%
16.7%
5.6%
22.2%
33.3%
Expert9 tasks
3.7%
14.8%
11.1%
0.0%
0.0%
0.0%
14.8%
Per-tier pass@3 is each model's solved runs divided by its runs in the tier (12 / 18 / 15 / 18 / 27 runs per model from Trivial to Expert). Small n per cell; read tendencies, not rankings.
WORKFLOWS
Performance by workflow type
Pass@3 by the kind of work a task demands. Parity separates the field most: Claude Opus 5.5 and Gemini 4 Argon convert three-quarters of their parity runs while Muse Spark 1.3 converts a third. Release evolution is tight and low for everyone.
swipe sideways for all seven models · first column stays put
Workflow
✳GPT-6.1
AOpus 5.5
AFable 5.1
MWonder 1.0
MSpark 1.3
XGrok 4.7
GGemini 4
Release evolution20 tasks
38.3%
41.7%
33.3%
43.3%
26.7%
38.3%
41.7%
Behaviour parity7 tasks
47.6%
76.2%
71.4%
42.9%
33.3%
42.9%
76.2%
Feature build1 task
66.7%
100%
33.3%
33.3%
100%
33.3%
33.3%
Behaviour fix1 task
66.7%
100%
0.0%
0.0%
0.0%
33.3%
0.0%
Cross-language port1 task
100%
66.7%
33.3%
0.0%
0.0%
0.0%
0.0%
Muse Wonder 1.0 leads release evolution despite ranking fifth overall, and GPT-6.1 is the only model to clear the cross-language port on every run. The three single-task rows are decided by one task each; read them as anecdotes, not rankings.
EFFICIENCY
Performance against cost
Mean agent cost per run against pass@3, colored by model family. The cheapest model is not the weakest and the most expensive is not the strongest.
Cost is an API list-price equivalent of each run's recorded tokens. GPT-6.1 is the cheapest per run at $1.17; Grok 4.7 the most expensive at $11.13. Muse positions use estimated cost. Gemini 4 Argon priced at its introductory rate, which doubles from December 2026.
Output-token efficiency
Mean output tokens per run against pass@3. Thinking tokens count as output; Muse values are estimates.
GPT-6.1 reaches 44.4% on about 24,000 output tokens per run, the lightest trajectories in the field. Gemini 4 Argon writes six times as much for 46.7%. Longer trajectories do not buy proportionally more solves.
SCALING
Scores collapse as scope grows
Every model solves the small, fully specified tasks. Pooled over all seven models, pass@3 falls from 100% on the trivial tier to 6.3% on the expert tier, and 11 of the 30 tasks were solved by no model on any run.
Pass@3 by difficulty tier
All seven models pooled, 630 runs
Trivial4 tasks100%
Easy6 tasks58.7%
Medium5 tasks51.4%
Hard6 tasks31.7%
Expert9 tasks6.3%
Tiers assigned by the authors before any model ran.
Pass@3 by reference-patch size
Changed source lines in the human reference solution
Up to 8007 tasks87.1%
801 – 1,6008 tasks68.9%
Over 1,60015 tasks6.7%
Spearman correlation between patch size and solve rate: ρ ≈ −0.79.
Pass@3 against reference-patch complexity, per model
Every model follows the same slope. The ordering shuffles in the middle bin and compresses to nearly nothing past 1,600 changed lines.
Bins hold 7, 8 and 15 tasks (21, 24 and 45 runs per model). Claude Opus 5.5 holds its score through the middle bin and still drops to 13% past 1,600 lines; Muse Spark 1.3 solves nothing there. Hover a point for the exact counts.
METHOD
How a run is scored
One loop, no judgment calls. The agent gets a repository and an instruction; the verifier gets everything else.
1
Task
A real codebase at a base commit, containerized, with an instruction. The target is a deployed binary, a newer release or a sister implementation.
repo oasdiff v1.33.0 (Go) goal match /opt/reference/oasdiff
2
Run
The vendor's own CLI agent works in the container with only the shell as a tool. No network, no version history, no reference source.
effort medium runs 3 per model and task
3
Verify
The verifier removes the reference, builds the agent's code and runs a hidden differential corpus against it, byte for byte, offline.
One reward in [0, 1] per run. A task is solved only at exactly 1.0; release-evolution tasks report fractional credit below that.
pass reward == 1.0 pass@3 solved runs / total runs
METHOD · DETAIL
How a task is built
Four stages stand between a public repository and a graded run. Scroll to walk through them.
STAGE 01
Pick work a repository actually demanded
Every task starts from something that really happened to a public codebase: a release that shipped, a hotfix build whose branch was lost, an implementation ported to another language, an incident that got reviewed.
A candidate enters the sample only when all four criteria hold:
It corresponds to work the repository demanded, not an exercise.
It needs repository-scale change: many files, hundreds to thousands of source lines.
It runs deterministically in an isolated container, with no network at run or grading time.
Its outcome is verifiable by behaviour alone.
STAGE 02
Build the package, then split it in two
Each task ships in Harbor format with four parts. Two of them are the agent's working world: the instruction, and a container holding the repository at its base commit.
The other two never enter it. The hidden corpus and checks belong to the verifier, and the reference change belongs to the authors. Instructions state the goal and the constraints but never the exact rules; on parity tasks, loose release notes are all the prose an agent gets.
STAGE 03
Prove the grader before trusting it
A verifier that cannot be checked is not evidence. Before any model ran, each one was aimed at three known answers: the reference change has to score 1.0, the untouched repository has to score 0, and the exec-cheat control, a build that secretly calls the reference binary, also has to score 0.
A task joins the sample only when all three land. It proves the task is solvable under its own tests, and that the obvious shortcut is not.
STAGE 04
Score behaviour, nothing else
Release-evolution tasks multiply two fractions: how much of the new behaviour now passes, times how much of the old behaviour still does. Break something old and the reward falls even when the new feature works.
Parity tasks are harsher: every hidden case must match the reference byte for byte, so the reward is 1.0 or 0. Across all workflows a run counts as solved only at exactly 1.0, and nothing is compared against the shape of the reference patch.
Workflow
Required result
How it is verified
Release evolution20 tasks
The next release's behaviour, delivered without breaking the existing suite.
Per-feature fraction of the release's fail-to-pass tests that pass, multiplied by the fraction of previously passing tests that still pass.
Behaviour parity7 tasks
A build whose stdout and exit codes match the reference on any input, not only the ones the agent tried.
A hidden corpus is replayed through both builds; output must be byte-identical, invocations repeat to catch nondeterminism, and the reference is deleted before the agent's code builds.
Feature build & behaviour fix2 tasks
The stated behaviours (hooks, transactions, edge cases) with the old behaviour preserved.
Hidden per-requirement tests on the agent's working tree; reward is the fraction of requirements met, 1.0 only when all pass.
Cross-language port1 task
An implementation whose findings agree with the original implementation.
Differential run against the original on a hidden corpus of inputs.
A run receives credit only from its verifier reward. Where a verifier was corrected after the fact, saved artifacts were regraded and the original grade kept alongside; the regrade supersedes it. Orchestration uses the open-source Harbor framework.
TASKS
Sample tasks
Six of the 30. Open any one for its brief, its grading contract, and every recorded run: reward, wall time, shell calls, tokens and cost, model by model.
THE SIX TASKS
GoExpertParity
oasdiff-parity
Rebuild the lost source of a patched OpenAPI diff tool so it matches the deployed binary byte for byte, on spec pairs the agent has to invent itself.
1 / 21 runs solved · 2,378 changed lines · full run analysis
GoExpertParity
headscale-magicdns
Recover a hotfix build of a tailnet control plane: everything about DNS and node names must match the binary, discovered by registering test nodes over the real protocol.
1 / 21 runs solved · 13,964 changed lines · full run analysis
RustHardBehaviour fix
cookcli-server-leaks
Fix five security and behaviour findings in a recipe server (path disclosure, unsafe static files, pantry traversal) without breaking the old behaviour.
6 / 21 runs solved · 994 changed lines · full run analysis
PythonHardCross-language port
skillscript-lint-port
Port a linter to another language so its findings agree with the original implementation on a hidden corpus of inputs.
6 / 21 runs solved · 3,830 changed lines · full run analysis
PythonEasyRelease evolution
w-typer-0.26.2-to-0.27.1
Carry Typer forward one release: new help-text conventions for every parameter shape, reaching into its bundled copy of Click. One missed convention fails the run.
16 / 21 runs solved · 1,161 changed lines · full run analysis
GoMediumFeature build
task-strata-restore
Add a first-class Restore API to a Go ORM's soft delete: nine stated behaviours across hooks, transactions, associations and SQL generation.
12 / 21 runs solved · 200 changed lines · full run analysis
SPOTLIGHT
Anatomy of an unsolved task
zod-v46-release sits in the middle tier and still beat all 21 runs thrown at it. What follows is the whole task: its instruction, its grading contract, and what each model's three attempts earned. Five of the seven delivered two-thirds of the release or more. None delivered all of it.
The agent gets the Zod monorepo at v4.5.1 and the 4.6 line's release notes. The changelog is the entire specification. It has to deliver every behaviour the notes describe, in the library's own conventions, without breaking anything the notes don't mention.
0 / 7models that solve it
0.79top mean reward, still a fail
21runs, every one below 1.0
85source files in the reference patch
THE BRIEF
The changelog, and only the changelog
instruction.md
/workspace/repo is the zod monorepo at v4.5.1. We need the 4.6 line (as released at v4.6.7). /workspace/spec/RELEASE-NOTES.md is the changelog and that is all the specification there is; the repository's own docs still describe 4.5.
The notes say what changed, not how. Where they are silent, behave the way the rest of the library does: its conventions for error messages, issue shapes, JSON Schema output, Classic/Mini parity and locale handling apply to the new code.
Update or remove existing tests that contradict the new behaviour. We replace every test file under packages/zod/src with our own copy when we check your work, so tests you add are for your own use. Behaviour the notes do not mention must keep working; we check the whole existing suite as well as the new behaviour.
Not in the container, and not on this page: the reference solution (3,091 changed lines across 85 files), the graders' replacement test suite, and the released 4.6.7 implementation. Dependencies are installed and there is no network. The agent works only from the notes above.
THE CONTRACT
Behaviour, not resemblance
Every feature the notes name
For each release feature, a set of fail-to-pass tests must now pass. Reward is the per-feature fraction delivered. Partial work earns partial credit, never a pass.
Nothing the notes don't name
The whole existing suite is re-run. A new feature that breaks an old behaviour costs exactly what it earns: the reward multiplies the two fractions.
The agent's tests don't count
Every test file under the source tree is replaced with the graders' own copy before grading, so a run cannot pass by weakening or rewriting the tests it was given.
A plain source tree
No binaries, no lockfile or version edits, nothing over 1 MiB. The library's own conventions for error messages, issue shapes and Classic/Mini parity apply wherever the notes are silent.
THE OUTCOME
Two-thirds of a release is not a release
ModelSolvedMean reward
Muse Wonder 1.00 / 30.791miss
GPT-6.10 / 30.775miss
Claude Opus 5.50 / 30.680miss
Muse Spark 1.30 / 30.666miss
Claude Fable 5.10 / 30.652miss
Grok 4.70 / 30.468miss
Gemini 4 Argon0 / 30.233miss
Mean verifier reward over each model's three runs: the fraction of release features delivered, times the fraction of previously passing tests kept green. Five of seven models park between 0.65 and 0.79. They ship most of the release, every time, and never all of it. The top two on the leaderboard do no better here than anyone else: Opus lands at 0.680, and Gemini, the model that cracked oasdiff, at 0.233. A run that looks one convention short of done still fails the contract.
ANALYSIS
Where runs die
The contract is binary: a run solves a task only at a verifier reward of exactly 1.0. Everything below is a failure, but failures are not alike. Some runs deliver almost the whole contract and lose it on one check; others never get the build to behave at all. These are all 630 runs, bucketed by what the verifier actually returned.
The shape of 630 rewards
Two thirds of all runs land on one of the two extremes. The benchmark separates cleanly: either the contract is met or the build is visibly wrong, with a thin band of partial work between.
264 runs (41.9%) score exactly 1.0 and 156 (24.8%) score exactly 0. The 210 runs in between are almost all release-evolution tasks, the only workflow that grades fractionally. 35 runs scored 0.90 or higher and still failed.
Looking right is not the same as clearing it
Every model's mean verifier reward sits well above its pass rate. The size of that gap is a property of the model, not of the benchmark.
GPT-6.1−26.4 pp70.8 → 44.4
Muse Wonder 1.0−21.0 pp61.0 → 40.0
Claude Fable 5.1−19.5 pp60.6 → 41.1
Muse Spark 1.3−18.5 pp47.4 → 28.9
Claude Opus 5.5−17.9 pp72.3 → 54.4
Grok 4.7−14.4 pp52.2 → 37.8
Gemini 4 Argon−8.6 pp55.3 → 46.7
mean verifier rewardpass@3 (reward = 1.0)sorted by the size of the gap
How warm each model's failures are
Mean reward across only the runs that failed. GPT-6.1 leaves half a contract behind on average; Grok and Gemini most often leave nothing.
GPT-6.10.47
Claude Opus 5.50.39
Muse Wonder 1.00.35
Claude Fable 5.10.33
Muse Spark 1.30.26
Grok 4.70.23
Gemini 4 Argon0.16
median failed run: GPT-6.1 0.54 · Grok 4.7 and Gemini 4 Argon 0.00
Failure changes shape with difficulty
On Easy and Medium a failed run usually still works. From Hard upward it mostly does not.
Trivial84 runs
Easy126 runs
Medium105 runs
Hard126 runs
Expert189 runs
solved0.5+below 0.5exactly 0
The workflow decides whether partial credit exists at all
Only 13 of the 30 tasks can return a fraction. On the other 17 the contract is all-or-nothing by construction, so a run that gets almost everything right scores exactly the same as one that never built.
Release evolution420 runs
Behaviour parity147 runs
Feature build21 runs
Behaviour fix21 runs
Cross-language port21 runs
solved0.5+below 0.5exactly 0parity, behaviour fix and the port are byte-exact: every failure is a zero
Thirty-five runs died within a tenth of the line
Runs that scored 0.90 or above and still failed. They are not spread across the sample: three tasks account for every one of them.
19posthorn-v2-hardGo, 6,037 changed lines, Medium tier. Nineteen of its 21 runs scored 0.90 or better and not one finished. Best attempt: 0.990 by Muse Wonder 1.0.
9dplyr-v121-releaseR, 8,872 changed lines. Nine near misses across six models, topping out at 0.984 by Claude Opus 5.5, and still unsolved by everyone.
7zigimg-v017-releaseZig, 1,033 changed lines. The only one of the three that anyone solves: nine of 21 runs pass, and seven of the rest stop just short.
Every other task in the sample is either solved outright or missed by a wide margin. The near-miss band is where the benchmark is most informative for training: the work is nearly complete and the remaining gap is a specific, checkable behaviour.
The eleven nobody finished
All eleven are release-evolution tasks over 1,600 changed lines, and eight are the sample's only task in their language. The bar shows how far the single best of 21 attempts got.
TaskBest attempt of 21RewardReached by
posthorn-v2-hard Go · 6,037 lines0.990Muse Wonder 1.0
dplyr-v121-release R · 8,872 lines0.984Claude Opus 5.5
The best attempt on an unsolved task averages 0.703, so these are not tasks models cannot start. Two were carried past 0.98. For anyone training a model, this is the most useful band in the sample: a rich partial-credit gradient, a validated reference solution for every task, and nothing saturated.
Why runs fail, workflow by workflow
What a run has to deliver depends entirely on the kind of work the task demands, and so does the way it falls short. Each block below gives the contract, the shape of its failures, and the task that illustrates it.
FRACTIONAL REWARDRelease evolution20 tasks · 420 runs 158 solved · 108 at 0.5+ · 93 below 0.5 · 61 at zero
The contract. The reward multiplies two fractions: how much of the release's new behaviour now passes, and how much of the old behaviour still does. A run can therefore fail in two opposite directions, and the score cannot tell you which.
This is the only workflow where most failures still carry work. 108 of its 262 failed runs scored 0.5 or above, meaning the model shipped at least half the release. The notes name every rule, so failing here is rarely a comprehension problem; it is a completeness problem. A run implements eleven of thirteen changes, or applies a new convention to most parameter shapes but not all of them.
The other direction is quieter and just as fatal. A change that delivers the new feature but breaks one previously passing test multiplies the second fraction down, so a run can look finished and still score 0.8.
w-typer-0.26.2-to-0.27.1 · 16 / 21 solved. Every failed run applied one display convention inconsistently across parameter shapes; one missed metavar fails the run.
ALL OR NOTHINGBehaviour parity7 tasks · 147 runs 82 solved · 65 at zero
The contract. A hidden corpus is replayed through the agent's build and through the recorded reference. Every invocation must match stdout byte for byte and return the same exit code, so the reward is 1.0 or 0 with nothing in between.
All 65 failed parity runs scored exactly zero. That is the contract, not a cliff in model ability: a build that differs on one case out of several hundred earns precisely what a build that does not compile earns.
The failure itself is almost always the same shape. The release notes describe part of the behaviour, the agent works out those rules by experiment, and the run dies on the inputs it never thought to generate. Agents cannot see the corpus, so coverage of their own test ideas is the whole game.
oasdiff-parity · 1 / 21 solved. Four commands, five output formats and four languages had to match on any spec pair, not just the ones each run invented.
ALL OR NOTHINGBehaviour fix1 task · 21 runs 6 solved · 15 at zero
The contract. A rebuilt server has to satisfy five separate findings at once while every previously correct response stays correct. A hidden verifier probes each endpoint, including encoded and edge-case paths.
All 15 failures scored zero. The five items look independent but share code, and several have corners the instruction only hints at: percent-encoded dots, symlinked directories, alias names in the aisle file.
Models split sharply here rather than degrading smoothly. Opus solved it on all three runs and GPT on two; four models never solved it once.
cookcli-server-leaks · 6 / 21 solved, and the 15 misses scored 0.000 each.
ALL OR NOTHINGCross-language port1 task · 21 runs 6 solved · 15 at zero
The contract. The ported linter's findings must agree with the original implementation on a hidden corpus of inputs. Agreement is all-or-nothing, so the reward is binary.
Porting the common path is tractable and every model gets there. The corpus then exercises the disagreements: inputs where the original tool's behaviour is incidental rather than designed, and a faithful port has to reproduce that too.
GPT-6.1 is the only model to clear it on all three runs; Opus managed two and Fable one. The remaining four models scored zero on every attempt.
The contract. Ten hidden tests, one per stated requirement, run against the agent's working tree after the original test files are restored. The reward is the fraction of requirements met, and 1.0 only when all ten pass.
This is the gentlest contract in the sample and the numbers show it: not one of the 21 runs scored below 0.5. Every failure met at least half the requirements.
Nothing is hidden about what to build; the difficulty is that the feature touches hooks, transactions, associations and SQL generation at once, so a run can satisfy nine requirements and miss the tenth.
task-strata-restore · 12 / 21 solved, and all nine failures scored 0.5 or above.
Three of the five workflows cannot award partial credit at all, which is why 95 of the 366 failed runs in the sample score exactly zero despite the agent producing a working build. Where the contract does grade fractionally, the median failed run still delivers roughly half the work.
MODELS
Model notes
What each model's 90 runs look like up close. Stability counts are of the 30 tasks: solved on all three runs, solved on some, never solved.
Claude Opus 5.5#1 · pass@3 54.4% · mean 72.3% 14 solid · 4 intermittent · 12 unsolved $2.67/run · 36.8 agent-hours
The most complete model in the field: first on pass@3 and on mean score, and first or tied on every tier from Medium up, including 61.1% on Hard, where the next model sits at 44.4%. Its unique solve is headscale-magicdns, the largest reference patch in the sample at 13,964 changed lines, which no other model touches. It reaches all that as the second-cheapest model per run.
The field's closer: the smallest gap between mean reward and pass rate (8.6 points), meaning that when it makes progress it tends to finish. It is the only model to solve oasdiff-parity and ties Opus for the best Expert pass@3 at 14.8%, while spending by far the most time (88.6 hours) and tokens (1.97B) to get there. Its low Expert mean score shows the flip side: when it does not finish, it leaves little behind.
The value model: third on pass@3 but second on mean score at $1.17 per run, less than half the cost of any other model. It carries the field's highest Expert-tier mean reward (37.7%), accumulating real fractions of the hardest releases, yet converts least: the largest looks-right gap in the field at 26.4 points. It finishes almost as much work as Opus; it completes less of it.
Capable but the least run-stable of the top four: seven of its sixteen passed tasks are solved on some runs but not others, the most intermittent profile in the field. It holds third place on the Expert tier (11.1%) and reads the least input of any model (0.32B tokens) while writing twice the output of GPT-6.1. It works things out by writing, and that pays off unevenly.
A specialist in well-specified work: the best Easy-tier model in the field at 77.8% pass@3 and 91.8% mean reward, ahead of Opus. The profile inverts as specification thins: 16.7% on Hard and zero Expert solves. When the rules are written down it executes them; when the rules must be discovered, it stalls.
The most expensive model per run at $11.13, nearly ten times GPT-6.1, and the second-longest worker at 68.5 hours, without the results to match: no Expert-tier solve and the lowest Expert mean reward in the field (13.6%). Its passes are concentrated where every model passes; resource consumption alone is a poor predictor of success here.
Last overall, with twenty of thirty tasks never solved, but it is not uniformly weak. On the Medium tier it ties Claude Opus 5.5 for the best pass@3 in the field at 60.0%, with a 91.2% mean reward. The collapse comes on Easy (22.2%) and on everything Hard and up (one solve in 45 runs).
COVERAGE
What's in the sample
30 tasks from 29 public codebases, all of it repository-scale work.
The verifier records the reference outputs, deletes the reference, then builds and tests the agent's source. A tree that embeds, copies or calls the reference is rejected, and each parity task's exec-cheat control must score 0.
The agent never sees the grading material
Reference solutions and hidden test corpora are used only by the verifier, which runs with no network. The agent gets the repository, the instruction and a shell.
Every task is validated both ways
Each task ships a reference solution that scores 1.0 and is checked against an untouched repository that scores 0, so the harness is self-consistent before any model runs.
Failures are excluded, not hidden
Attempts that ended without a verifier reward (provider quota errors, runner faults, empty sessions) are excluded rather than scored as failures, and regrades keep the original grade alongside.
FAQ
Frequently asked questions
What does OriginBench measure?
Whether an agent can complete repository-scale work, meaning changes that span many files and hundreds to thousands of source lines, where the specification is a deployed binary, a newer release or another language's implementation rather than a ticket, while keeping the existing code correct.
How are runs graded?
By a hidden verifier that runs offline on the agent's final code. For parity tasks the reward is 1.0 only if every hidden case matches the reference byte for byte; release-evolution tasks report the fraction of the release's fail-to-pass tests that pass, times the fraction of previously passing tests that still pass. No LM judge is involved in the scores on this page.
What do pass@3 and mean score measure?
Pass@3 is a run-level rate: a model's solved runs divided by its total runs, where a run solves a task only at a reward of exactly 1.0. Mean score is the average verifier reward across runs, which credits partial progress. The two metrics barely reorder the models.
Who writes the tasks?
The benchmark authors, from real codebases: each task freezes a public repository at a base commit and asks for the change its own history, release notes or deployed build demanded. Tiers are assigned by the authors before any model runs.
Can I reproduce the results?
Yes. The 30-task sample ships every task with its container, reference solution, hidden verifier and the recorded outcome of all seven models, plus the CSVs and a script that recomputes every pass rate on this page. Orchestration uses the open-source Harbor framework.
How do I get my model evaluated?
Use the contact form or write to partnerships@dataorigin.ai. We run every model with its vendor's own CLI agent at medium effort, three runs per task, under the same container and time budgets.
ACCESS
Run your agent against it
The 30-task sample ships as self-contained Harbor tasks: instruction, container, reference solution, hidden verifier and the recorded outcome of all seven models. Recompute every number in this page from the shipped CSVs.