FairBakeHow the Signature bake is run
What a task is, what counts as solved, where each number comes from, and what I keep private.
The tasks
Each of the 30 tasks is a change somebody really made to a codebase: 20 bug fixes and 10 features, in JavaScript, TypeScript, Python and Rust. The repositories come from a mix of sources: primarily private, with some recent open source as well. The agent gets the code as it stood before the change, and the problem described the way its owner would have put it.
Solved
Tests the agent never sees do the grading. A run is solved when all of them pass and nothing that worked before is broken. Anything else is not solved.
The agents
An agent is a model in the harness its maker ships, or in pi when it has none, at one effort level. Every agent's page shows its harness, version and effort level.
Runs
The standard is 2 runs of every task. An agent with a different number is tagged with its count, and is ranked the same way, by the share of its runs that solved.
Cost per fix
Runs in Claude Code and Codex are priced at list API prices for the tokens each run used. Runs through OpenRouter are priced at what OpenRouter charged. List prices use each model's base rate, so a model that charges more for long prompts costs more than shown. Cost per fix is everything an agent's runs cost, divided by the runs that solved their task, so the failed runs are paid for too.
Time per run
How long the agent worked, as the median over its runs. Several agents ran at once on one machine, so read a small gap as noise.
Lines to review
Lines added plus lines removed in the agent's change, any tests it wrote included, as the median over its runs. It's what a reviewer has to read.
What stays private
The codebases, the prompts and the tests. Once a benchmark's tasks are public they end up in training data, and it starts measuring memory.
Tokens in and tokens out
Each run's detail gives two token counts. Tokens in is everything the run sent to the model. An agent sends its context again on every request, so this is a sum over all of them, and most of it is read from cache. Tokens out is what the model wrote. Each harness reports its own counts.