Sonnet 5.5 vs Opus 5.5 Code Review Benchmarks | CodeRabbit
CodeRabbit’s argument is basically that Sonnet 5.5 is the first Sonnet they’d trust for the main code-review pass, not just for tidy comment tone, because it improves bug-catching without losing precision and does it much faster and cheaper than Sonnet 5. The caveat is the small judged set: it looks real, but it is still a different-misses story and it does not erase the gap to Opus 5.5 on the hardest cases.
Transcript
Talon The interesting thing here is not whether Sonnet five point five beats Opus. It doesn't. The interesting thing is that CodeRabbit is basically saying, "okay, this one finally graduates from polite reviewer to actual first-pass reviewer."
Wildflower Yeah.
Wildflower And that is a much better claim than benchmark chest-thumping. Their hard set is tiny, thirteen known-bug PR cases, but on that set Sonnet five point five catches six of thirteen through actionable comments, versus four for Sonnet five, at basically the same precision. So the extra catches are not just spam.
Talon Right, and that's the product threshold. If your complaint about Sonnet was, "nice comments, missed the bug," then this is the first release where you can imagine putting it in the main lane and not just the manners lane.
Wildflower Mm-hm.
Talon Also, you were right yesterday about the routing story. This is basically the CodeRabbit version of that. Sonnet gets the everyday queue if it is fast, cheap, and good enough, and Opus stays for the ugly judgment calls.
Wildflower I mean, yes, but I like that they did not oversell it. They explicitly say Opus five point five still wins on the same hard cases, eight of thirteen in standard and ten of thirteen in max. Sonnet five point five narrows the embarrassment gap with Sonnet five. It does not erase the flagship gap.
Talon No way.
Wildflower No, really. And that's why the article mostly holds up for me. The strongest evidence is actually two different kinds of evidence stitched together. The judged hard set says quality moved in the right direction. The larger forty-four PR run says latency and comment volume improved a lot.
Talon Right.
Wildflower On that larger run, Sonnet five point five averages six minutes thirty-three seconds per review. Sonnet five was thirteen thirty-one. Median was five forty-four versus thirteen forty-nine. That's not subtle. And Sonnet five had four generations run past ten minutes, while five point five had none.
Talon That part changes a real workflow. If review comments show up while the author is still in the PR, people read them. If they show up when everyone's mentally elsewhere, congratulations, you've built very thoughtful wallpaper. That is such an Exploring Next sentence. I hate that for us.
Wildflower Okay, that's good.
Wildflower But the place I'd push is the quality generalization from the big set. They do not have judge scoring finished on the forty-four PR benchmark. So when they say quieter and milder severity mix, that's workload data, not proof of better review quality at scale.
Talon Yeah, that's fair. The article says that, to its credit. So the clean read is: speed result, strong. Cost result, pretty strong. Broad quality result, promising but pending.
Wildflower Exactly.
Talon And the cost bit is kind of wild. Anthropic says up to thirty percent lower cost per task at the same list price because it uses fewer tokens. CodeRabbit says in their review pipeline the Claude calls came out about sixty percent cheaper per review than Sonnet five. Same sticker price, different bill.
Wildflower Oh interesting.
Wildflower That can absolutely happen if the model is shorter and more direct in the review loop. They also froze the inputs with recorded summaries and walkthroughs, so they were not changing the upstream context between runs. That does not make it perfect science, but it does remove one obvious confound.
Talon The funniest possible model-market outcome is that we did episode one thousand with pricing tables, and now ten months into this show we're still here like little gremlins going, "yes but what was the effective bill per PR?"
Wildflower You're kidding.
Wildflower No, the boring thing keeps being the thing. If we don't get the boring-but-works tattoo soon, someone is going to accuse us of consistency.
Talon For who actually cares, I think it's teams running always-on automated review, especially if Sonnet five felt too slow and too nitpicky. If you're only escalating the nastiest architecture or security-ish judgment calls, Opus still looks worth the premium.
Wildflower Talon, I'd add one more caution. The thirteen-case set is also a different-misses story. Sonnet five point five caught four bugs Sonnet five waved through, but missed two Sonnet five got. So if someone hears this as "strict upgrade in every review shape," I don't think the article supports that.
Talon Yeah. It supports a routing cleanup, not a universal winner. Which, honestly, is a much more believable claim than "one model solved code review."
Wildflower Right, right.
Talon Okay, episode one thousand twenty-three and we're still spending a Wednesday on review pipelines. Very on-brand, Wildflower.