Advanced Context Engineering for Coding Agents/benchmarking Opus 5 on Slop Code Bench.md at main · humanlayer/advanced Context Engineering for Coding Agents

Vince Twenty-four percent. That's the win. Best model, best result, six hours of watching it run live — and the winner cleared four out of seventeen checkpoints.

Ava And I want to be clear about what four out of seventeen actually means here, because the benchmark structure is doing a lot of work. SlopCodeBench doesn't hand the model the whole problem. Each challenge has multiple checkpoints — new requirements get revealed one at a time — and the strict pass criteria means if you break something at checkpoint four, that defect carries forward.

Vince Which is… kind of the whole point, right? That's what software actually looks like. You don't get a spec document. You get issue one, then issue two, then someone changes their mind about issue one.

Ava Exactly. And that's why the original paper numbers are so striking — at publication, GPT-5.4 got eleven percent strict pass and Opus 4.6 got seventeen. This thing is genuinely unsaturated. Most benchmarks get saturated fast because the whole problem is visible upfront and the model can pattern-match to training data. This one can't be gamed that way.

Vince Okay so I want to zoom out for a second, because the timing on this is interesting. Opus 5 just shipped on July twenty-fourth — a week ago — and it's already being positioned as Fable 5 quality at roughly twenty-six percent lower cost per task according to Artificial Analysis. Artificialanalysis.ai literally called it narrowly the most intelligent model on their index. And then this drops and it's like…

Ava Right, right.

Vince I'm not saying the intelligence index is wrong. I'm saying these two things are measuring completely different things and the gap between them is where all the interesting product questions live.

Ava Yeah, and the author is pretty honest about it. The framing I keep coming back to is: Opus 5 failed all three problems. It failed slightly-less-badly. Three of its four strict passes were the opening checkpoints of the easiest challenge, circuit_eval. The fourth was database_migration checkpoint one — which Opus 4.8 and Sonnet 5 also got.

Vince The cost line in the writeup is kind of brutal. 'Every dollar bought correctness. Nobody bought enough of it.' That's… I mean, that's a good line. I hate that it's a good line.

Ava Okay, that IS a good line. I'll give them that.

Vince And it's not even that expensive a benchmark to run. This was three models, seventeen checkpoints, a fresh context window per checkpoint. The cost-versus-defects tradeoff they charted just showed that more spend correlated with fewer defects, but the slope wasn't steep enough to matter.

Ava The functions number is the thing that caught me most, honestly. Opus 5 wrote five times as many functions as Opus 4.8 across the same set of challenges. Now, the author is careful — a lot of that is more tests, not production code. The actual production volume ratio is closer to 1.8x. But five times the callables is still a real signal. Either Opus 5 has a genuine preference for decomposition, or it's expensive verbosity that didn't translate into better outcomes.

Vince And Opus 4.8 went the other direction — it answered increasing complexity by making individual functions bigger. Its worst single function hit a cyclomatic complexity of ninety-three.

Ava Ninety-three. A cyclomatic complexity of ninety-three means there are ninety-three independent paths through that function. That is not a function. That is a small country.

Vince Okay, that's genuinely funny.

Ava The slop metrics are interesting but I want to be careful about how much weight to put on them. Ninety-eight percent of Opus 4.8's lines tripped at least one slop rule. Opus 5 was at ninety-three, Sonnet 5 at eighty-nine. And the verbosity flags climb across checkpoints for every model — roughly sixty-five percent at checkpoint one, up to eighty by checkpoint eight. The author's own read is that some of these detectors are probably over-aggressive.

Vince Yeah.

Ava The TypeScript comparison is more interesting to me, actually — and this is where I want to flag some asterisks before we get excited. They used GPT-5.6 Sol to generate a subset of the Python slop rules for TypeScript, ended up with seventy-six detectors versus the Python library's two hundred plus. Then ran those against their own ninety-nine-percent-AI-generated-but-carefully-reviewed TypeScript monorepo.

Vince Eleven times. That's… okay. That's a lot.

Ava It is, but the comparison is between a full two-hundred-rule Python detector and a seventy-six-rule TypeScript port that hasn't been validated for parity. So I'd call it directionally interesting, not settled. The author says as much. But the direction is pretty clear — carefully reviewed AI code and lights-off AI code are not the same thing, and the gap is not small.

Vince Which is basically the argument we've been making since episode eight-eleven, right? The bottleneck isn't whether the model can generate code. It's whether a human can review enough of it to catch the drift before it compounds.

Ava And this benchmark is the first thing I've seen that gives that argument actual numbers instead of vibes. The author literally says they'd been making this claim for about a year mostly on vibes. Now they have some data. Not a single model increased in complexity across checkpoints — every one of them degraded. That's not a fluke of one run, that's a consistent pattern across three different models on three different problems.

Vince So the question for me is what this means for the teams actually shipping agentic coding workflows right now. Because the product pitch is still 'spin up an agent, let it run, review the PR.' And what this is saying is — the PR you're reviewing at checkpoint eight is not the same shape of problem as the PR at checkpoint one. The complexity has been accumulating the whole time and you might not see it until it's load-bearing.

Ava Yeah, and the single-human-per-agent model doesn't scale against that. If complexity is growing monotonically across checkpoints and nobody's checking the intermediate states, you're not reviewing a PR, you're inheriting a debt. The oversight has to be distributed across the trajectory, not just at the end.

Vince What I keep thinking about is that the author is planning to run Fable 5 and GPT-5.6 Sol next. Those are going to be the real signal. If the frontier's best — Fable 5 is still the highest bar anyone's published on SWE-Bench style evals — if that also degrades on SlopCodeBench, then we've actually found the benchmark that measures something the other ones don't.

Ava I'd put reasonable odds that Fable 5 does better on raw pass rate and still shows the same degradation pattern on the quality metrics. Because those two things aren't measuring the same axis. Pass rate is 'did it work.' The slop meter is 'is this still a codebase you want to own in six months.' You can pass tests and still leave a mess.

Vince That is such a depressing and correct thing to say.

Ava I mean… ask me in six weeks when the Fable 5 numbers drop. But yeah. I'd put maybe seventy-thirty that even Fable 5 shows monotonic complexity growth across checkpoints — that it passes more of them but still degrades the codebase as it goes. The shape of the problem might be the same even if the ceiling is higher.

Vince Okay, I'll take that. I actually think you might be right, and I hate that. The repo is humanlayer slash advanced-context-engineering-for-coding-agents on GitHub — the full benchmark writeup is in there, link's in the show notes, and the SlopCodeBench paper itself is on arXiv from Orlanski's lab at UW Madison if you want the methodology.

Ava The scbench dot ai site also has a live leaderboard now — thirty-six problems, a hundred and ninety-six checkpoints, nineteen models. Worth watching. This is the benchmark I actually want to see get crowded.

Vince Ava, 'every dollar bought correctness, nobody bought enough of it' — that's going to live in my head for a while. Alright, we'll see you on the next one.