Signature: 12 coding agents on 30 real bug fixes and features

The Signature agent leaderboard is live at fairbake.dev/signature: 30 unpublished real tasks, 12 coding agents, 720 runs - and will grow from here. It’s the benchmark I’ve wanted for the last year to help high-performing teams pick a coding agent. It focuses on solve rate, cost, and, importantly, lines left for code review as your engineers struggle with the current flood of LLM-generated code.

Every agent runs each task twice at its harness’s default effort level, and tests it never sees do the grading. The prompts and tests stay unpublished: once a benchmark’s tasks are public they end up in training data, and it starts measuring past performance rather than future returns.

Chart titled What a fix costs: the share of runs each of the 12 agents solved against its cost per fix on a log scale, with a line through the efficient frontier, the agents no other beats on both. On the line: GPT-6 Luna, 55% at $0.025 a fix, Haiku 5.5, 75% at $0.046, GPT-6.1 Terra, 80% at $0.10, GPT-6.2 Sol, 83% at $0.31, Opus 5.5, 88% at $1.04, and GPT-6.2 Astra, 90% at $1.11. Off it: GLM-5.3 Flash, 68% at $0.078, GPT-6.1 Sol, 80% at $0.28, Sonnet 5.5, 73% at $0.37, GPT-6 Astra, 83% at $1.76, Fable 5.2, 90% at $1.78, and Fable 5.1, 83% at $3.14.

What the leaderboard advises today:

  • Pick the cheapest agent that fits your work. Eight agents solve 80-90% of tasks, with a 30x difference in prices.
  • For routine work, agents are all quite good - they solved nearly a third of the tasks every time, including Luna that costs $0.025 a fix and left you the least code to review.
  • If you’re not sure about the complexity of your work, start with GPT-6.2 Astra. It tied Fable 5.2 for the most solved, 90% of runs, at $1.11 a fix against $1.78, and left less code to review.1

The site has the full leaderboard including efficient frontiers, pages with all tasks and for each agent, and how each bake was run. I’ll add models as they ship (Fable 5.2, GPT-6.2 Astra, and GPT-6.2 Sol were on the board the day they shipped). Gemini models are next on the list to add. Go have a look at fairbake.dev/signature!

These are my tasks. Yours will rank the agents differently, and a private bake runs the same method on your commits: book 30 minutes at fairbake.dev/call. Which agent should I add next?

A quick preview of what’s coming next to FairBake:

Slope chart of the share of runs each of the 12 agents solved on the first 20 tasks and on the 10 newest. First 20 tasks: Fable 5.2 and GPT-6.2 Astra 98%, GPT-6.2 Sol 95%, Opus 5.5, Fable 5.1, GPT-6.1 Terra, and Sonnet 5.5 90%, GPT-6 Astra and Haiku 5.5 88%, GPT-6.1 Sol 85%, GLM-5.3 Flash 83%, and GPT-6 Luna 73%. 10 newest tasks: Opus 5.5 85%, Fable 5.2, GPT-6.2 Astra, and GPT-6 Astra 75%, Fable 5.1 and GPT-6.1 Sol 70%, GPT-6.2 Sol and GPT-6.1 Terra 60%, Haiku 5.5 50%, Sonnet 5.5 and GLM-5.3 Flash 40%, and GPT-6 Luna 20%.

Notice that the 10 newest tasks on the leaderboard do most of the separating. On my first 20 tasks, 11 of the 12 agents solved 83% to 98% of their runs (a dull leaderboard), so I had agents spend 24 hours mining commit history for difficult changes. On those 10, solve rates run from 20% (GPT-6 Luna) to 85% (Opus 5.5). Look out for a writeup on the challenge loop in an upcoming update!

Footnotes

  1. I’ve seen negative user reviews of GPT-6.2 Astra that call it “lazy” or “lacking ambition”. They seem related to its style choices around terseness: next to Fable 5.2, it says less about what it did and didn’t do. This leaderboard doesn’t score that. ↩

← All updates