Signature: 12 coding agents on 30 real bug fixes and features
The Signature agent leaderboard is live at fairbake.dev/signature: 30 unpublished real tasks, 12 coding agents, 720 runs - and will grow from here. It’s the benchmark I’ve wanted for the last year to help high-performing teams pick a coding agent. It focuses on solve rate, cost, and, importantly, lines left for code review as your engineers struggle with the current flood of LLM-generated code.
Every agent runs each task twice at its harness’s default effort level, and tests it never sees do the grading. The prompts and tests stay unpublished: once a benchmark’s tasks are public they end up in training data, and it starts measuring past performance rather than future returns.
What the leaderboard advises today:
- Pick the cheapest agent that fits your work. Eight agents solve 80-90% of tasks, with a 30x difference in prices.
- For routine work, agents are all quite good - they solved nearly a third of the tasks every time, including Luna that costs $0.025 a fix and left you the least code to review.
- If you’re not sure about the complexity of your work, start with GPT-6.2 Astra. It tied Fable 5.2 for the most solved, 90% of runs, at $1.11 a fix against $1.78, and left less code to review.1
The site has the full leaderboard including efficient frontiers, pages with all tasks and for each agent, and how each bake was run. I’ll add models as they ship (Fable 5.2, GPT-6.2 Astra, and GPT-6.2 Sol were on the board the day they shipped). Gemini models are next on the list to add. Go have a look at fairbake.dev/signature!
These are my tasks. Yours will rank the agents differently, and a private bake runs the same method on your commits: book 30 minutes at fairbake.dev/call. Which agent should I add next?
A quick preview of what’s coming next to FairBake:
Notice that the 10 newest tasks on the leaderboard do most of the separating. On my first 20 tasks, 11 of the 12 agents solved 83% to 98% of their runs (a dull leaderboard), so I had agents spend 24 hours mining commit history for difficult changes. On those 10, solve rates run from 20% (GPT-6 Luna) to 85% (Opus 5.5). Look out for a writeup on the challenge loop in an upcoming update!
Footnotes
-
I’ve seen negative user reviews of GPT-6.2 Astra that call it “lazy” or “lacking ambition”. They seem related to its style choices around terseness: next to Fable 5.2, it says less about what it did and didn’t do. This leaderboard doesn’t score that. ↩
