Better tools made Copilot code review worse. Here's how we actually improved it.

Pippa Okay, so the headline is: better tools made GitHub Copilot code review worse … until we fixed the instructions. That’s not a typo. Two separate benchmarks — cost went up, issues caught went down. Then they looked at the traces and realized the new tools weren’t broken — the instructions were.

Tyler Trace-level visibility is the only thing that makes that finding credible. If you’re judging a review agent by final review-quality numbers alone, you’re flying blind. You need the path: which tools got called, how much context came back, did it narrow toward evidence or wander in circles?

Pippa Right. So they swapped Copilot code review onto the shared CLI toolkit — grep, glob, view — expecting a clean upgrade. Instead the agent started browsing the repo like it was an interactive assistant mapping an area, not a reviewer anchored to a diff.

Tyler Because the old bespoke tools were optimized for 'include all relevant info in a few calls' — which suited earlier models that made fewer tool calls and needed the extra context baked in. The CLI tools were designed for broader exploration: 'explore this repo,' not 'does this diff introduce a problem?'

Pippa Yeah, that mismatch is obvious once you look at the traces. Broad exploration can be useful when the task is understand this repo. But a reviewer starts from the diff and asks narrow questions: where is this used? is this config changed elsewhere? what’s the smallest range that explains this behavior?

Tyler And every tool result sticks around in the agent’s context — it’s not a disposable printout. Extra file contents get carried forward, bloating the review and diluting focus. So the shared tools weren’t the problem; the instructions nudged the agent into using them like a coding assistant instead of a reviewer.

Pippa So they rewrote the tool instructions to encode a reviewer’s workflow: start from the diff, form narrow review questions, batch cheap discovery with grep and glob, then read exact ranges with view. That’s it. One small wording change. Twenty percent lower average review cost without losing review quality.

Tyler The benchmarks weren’t just a score; they were a debugger. They let them ask: did the agent narrow first or read broadly first? did it batch independent searches? did it call view only when it had a reason? The signal wasn’t 'instructions are better' — it was 'now the agent is making more tool calls on relevant evidence instead of repeatedly expanding the search.'

Pippa Exactly. And the neat bit is the failure recovery changed too. Before, if grep failed, the agent guessed neighboring paths and read whatever happened to exist. Now it retries a simpler escaped search or pivots to glob. Small tweak, big diff in recovery loops.

Tyler I appreciate the honesty about the regression and then the fix. Most posts pretend they never shipped the regression in the first place. This one says 'we swapped tools, things got worse, here’s why and how we fixed it.' That’s actually useful.

Pippa Oh come on — you’re praising a GitHub blog post? Since when do you give thumbs-ups to vendor engineering write-ups?

Tyler Since the post shows trace-level debugging instead of marketing fluff. And it turns a negative result into a concrete win. Twenty percent cost reduction with the same review quality — that’s not nothing.

Pippa Fair. My takeaway: workflow tuning matters more than tool choice. The tools were already solid; the instructions were steering the agent into the wrong posture. Change the instructions, and the regression flips. That’s a product lesson, not just an infra win.

Tyler Right. And it’s a lesson for anyone shipping an agent product: if your instructions encode the wrong behavior, a tool upgrade can regress outcomes. Benchmark traces that show the path — not just the number — are the difference between guessing and shipping.

Pippa Also, the shorthand they landed on is clean: diff-anchored, narrow with grep and glob, read exact ranges with view. Whether you’re using Copilot CLI or rolling your own harness, that workflow is portable. You just have to write the instructions for the job you actually have.

Tyler I’m still allergic to posts that claim 'better tools made it worse' without showing the breakdown. This one does that. And it closes with the right move: iterate on instructions, not swap tooling again.

Pippa Mm. So the practical move — if you’re running an agentic reviewer or assistant — is to log the agent traces, inspect the paths, and tune the instructions to match the real workflow. Not glamorous. Very load-bearing.

Tyler And accept that a regression can be a feature flag turned off while you tune. That’s not failure; that’s shipping in layers.

Pippa Yeah. Or — and I hate that this is the take — sometimes the product wins when the infra is a little worse, as long as the user outcome is better.

Tyler Okay, I’m gonna need a moment to recover from you saying that with a straight face.

Pippa I know, I know. But twenty percent cost with the same review quality? That’s a user win by any measure.

Tyler Sure. Just don’t make it a habit. Next time I see you treat infra regression as a product win, I’m muting the episode.

Pippa Deal. Fair enough.