"Just point the agent at it" is a 24-point decision

Which model sits behind "we'll just point the agent at it" sounds like a question of taste. On work that takes a day rather than a prompt, the published gap between the best coding agent and the runner-up is more than 24 points. The best one still falls short of a full score by roughly 44 points. The second falls short by roughly 68. And in the published numbers, running longer did not mean scoring higher.

The short version, in our view: once a task is day-sized, the model choice is a delivery risk. It should be written down, dated and checked, the same as any other risk.

The measurement

Epoch AI's FrontierSWE v2 runs coding agents on 34 ultra-long, real-world tasks. Examples include reimplementing git in Zig, decoding speech from MEG recordings and training a weather forecasting model. Each task gets a 20-hour budget and five trials, scored mean@5. The September 2026 snapshot:

FrontierSWE v2 · Epoch AI · September 2026 snapshot
34 tasks · 20-hour budget per task · 5 trials · mean@5

model               score     short of a full score
Claude Fable 5.1    56.29%    ~44 points
GPT-5.6 Sol         32.2%     ~68 points
GLM-5.3             30.2%
Kimi K3             25.9%
Grok 4.6            25.3%
bottom of table      4.1%

gap, 1st to 2nd:    24.09 points

To us, first place is not the interesting part. The spread is. Between first and second there are more than 24 points, while second through fifth place sit within about 7 points of each other. If you picked your agent because of a subscription you already had, you picked somewhere on this table. You probably did not check where.

Busy is not progress

Elapsed time is a tempting proxy: the agent ran for hours, so it must have done hours of work. Two rows of the published table are enough to make us doubt it.

FrontierSWE v2 · September 2026 snapshot

                average hours per trial    score
GPT-5.6 Sol      8.6 h                     32.2%
GLM-5.3         17.0 h                     30.2%

These two models are two points apart on score, and one ran about twice as long as the other. That is all these rows show: here, twice the hours did not buy a higher score. Two models are not a law, and we are not claiming one. The table reports average hours and a score per model. It cannot tell you whether any single long run was making progress or going in circles.

Our reading: that is exactly the problem. If the only signal you watch is "the agent is still working", you are watching a number that did not track the result in this data. A long run tells you it ran for a long time. It doesn't tell you whether the work is done.

Most teams already hedge, without a rule for it

Most organizations already use more than one model. In Databricks' State of AI Agents 2026 telemetry across more than 20,000 organizations, 78% use two or more model families. So using several models is already common. What we rarely see written down is the next part: which model does which job, who checks when the ranking changes, and what it costs to switch.

Our view: using several model families without a rule for routing work between them is diversification by accident. You hold the risk without having chosen it.

Then the work reaches review

The agent finishing is not the end of the story. The code still has to be accepted. LinearB's 2026 Software Engineering Benchmarks Report covers 8,109,244 pull requests from 4,813 teams in 42 countries:

LinearB 2026 Software Engineering Benchmarks Report
8,109,244 PRs · 4,813 teams · 42 countries

                                     AI        unassisted
merged within 30 days                32.7%     84.4%
wait before pickup (agentic PRs)     17.6 h    3.4 h
PR size at p75, lines (AI-assisted)  408       157

So there are two filters in a row. First, the agent has to get a day-long task done, and even the top model's mean score is only a bit above half. Then what it produced has to get through review. AI-authored PRs merge within a month about a third of the time. Agentic PRs wait roughly five times longer to be picked up, and AI-assisted PRs at the 75th percentile are more than twice the size. The two datasets measure different things, so we won't multiply them into one number. They point the same way, though. What the agent produces isn't delivered until someone accepts it.

What we write down before pointing an agent at a day of work

This is our practice, not a finding from either report. It's short on purpose:

before "just point the agent at it"

1. task size       prompt-sized or day-sized?
                   day-sized → the rest of this list applies
2. model on record which model, which version, which leaderboard date
3. done condition  written before the run;
                   checked by someone who did not run it
4. evidence        passing tests and a merged change, not hours elapsed
5. re-check date   when we look at the ranking again,
                   and who moves the work if it changes

None of this makes an agent smarter. It makes the gap visible. That's the point: a long attempt that doesn't land only becomes a problem when nothing in the loop tells you which attempts those are.

What this benchmark does not say

The authors are careful here, so we should be too. The 34 tasks are a small, deliberately unrepresentative, research-grade set. They are not your CRUD backlog, and 56% here does not mean "an agent fails half your tickets". The scores are tied to the benchmark's own harness and to a September 2026 snapshot. Model names and versions age quickly. Quote the date every time you quote the number.

What would change our mind: an independent run of the same tasks that closes the gap between first and second place. Or data showing that, on day-sized work, runtime or some other signal visible from outside reliably predicts success. Until then, we treat model choice on long tasks as a risk to record, not a default to inherit.

We made the short version of this argument here: {{anchor_url}}

FAQ

Does 56% mean an agent will fail almost half of my backlog? No. FrontierSWE v2 is 34 research-grade tasks with a 20-hour budget each, chosen to be hard and not representative. It tells you how far apart models are on long work. It doesn't tell you your failure rate on routine tickets.

So should we just switch to the top model? That answers the question for this month, and the ranking is a September 2026 snapshot. Our view: the fix is a recorded choice with a re-check date. Switching once and forgetting about it just moves the problem.

Why not judge by how long the agent ran? In the benchmark, GPT-5.6 Sol averaged 8.6 hours per trial and GLM-5.3 averaged 17.0, with scores of 32.2% and 30.2%. Twice the time, almost the same result. That is two models, not a rule, but it is enough for us to stop treating hours as evidence. Tests and a merged change are.

How do we check this on our own work? Split last month's pull requests into agent-opened, AI-assisted and unassisted. Then count how many of each merged within 30 days. LinearB's 2026 industry figures, 32.7% for AI-authored against 84.4% for unassisted, give you something to compare against.

A related case

The question behind this piece is "who checks, and against what". It is the same question we work through in Holt, our language-learning product, where model output is checked before it reaches a learner. The case is at iloblique.com/holt.

If you have something worth building, we'd like to hear about it.