Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks

Pippa Okay so GitHub drops this post saying their agentic harness—you know, the thing that actually runs Copilot everywhere—beats the vendor harnesses on tokens without losing quality. I’m reading this and I immediately want to drop it into our infra. It’s basically the same harness powering CLI, web app, and code review, so one improvement ripples everywhere.

Tyler — Or it’s one harness that gets to claim all the wins while the vendor harnesses are stuck doing dog-and-pony shows we’re not allowed to see.

Pippa Oh come on, Tyler. They actually ran the numbers: SWE-bench, SWE-bench Pro, SkillsBench, TerminalBench, even a Windows-in-a-container test. Same model, same task, normalized context windows, reasoning effort at medium, all tools allowed, five runs per config, ellipses for variance.

Tyler The variance ellipses are real, but they’re huge. TerminalBench ellipse covers most of the chart for Copilot and the vendor harnesses on the same model. That’s not a win—that’s ‘your results may vary.’ And they disabled tool search for everyone but then let Copilot use its own built-in tools. Vendor harnesses don’t get that luxury.

Pippa Yeah but tool search is part of the vendor harness today because the harness is literally the thing that ships with the model. If Copilot’s harness routes differently and the vendor’s harness can’t, that’s a genuine difference. Also five runs on small benchmarks means ‘pick the best run’—that’s not how the real world works.

Tyler Exactly. They even say they report pass@1 on benchmarks under 100 tasks. That’s a cherry-pick factory. And their two-hour timeout hides the fact that in real agent flows humans poke at the thing after thirty minutes and close it out. Harness speed only matters if the human trust is there, and trust isn’t measured in ellipses.

Pippa Okay fine, but the post isn’t claiming ‘this harness will solve everything forever.’ It’s saying under controlled settings with a fixed harness config, you get the same task resolution with fewer tokens. That’s still a real lever for teams burning through credits at scale—fewer tokens, lower cost, same outcome. Who cares about the variance if the downside is ‘I spent twelve bucks instead of sixteen’?

Tyler I care because ‘fewer tokens’ is baked into harness defaults. They turned off tool search for the vendor harnesses but left it on for Copilot by virtue of built-in tools. That’s not apples-to-apples; it’s a controlled-variable stack that favors Copilot’s design. It’s infrastructure built to win the benchmark, not to solve the hardest tasks.

Pippa Wait—if the harness is the thing that decides which tools to enable, then of course turning tool search off disadvantages a vendor harness that relies on interactive discovery. That’s like evaluating a web browser against curl and saying curl won on token count. The harness shapes the tools available—that’s the whole point.

Tyler …which means we’re evaluating the harness design, not the underlying model’s capability. The model’s code-gen quality might be identical, but the harness’s decision to keep tool search disabled for the vendor harnesses and enabled for Copilot muddies the read. They even admit they’re normalizing context windows and prompt token limits. That’s great for reproducibility, but it neuters the vendors’ actual product surface.

Pippa So you’re saying the write-up is misleading because it buries the levers the harness controls? That feels a little cynical. They’re not hiding anything— they list every knob they turned off or on. If the goal is to compare harnesses, you have to hold the tools constant or you’re comparing apples to an apple-shaped rock.

Tyler Exactly. The apple-shaped rock wins on token count but not on task complexity. TerminalBench ellipse shows GPT models clustering left (high resolution, low cost) while Opus sits upper right (highest resolution, premium). Copilot’s ellipse overlaps both, which looks like a win until you realize the ellipse width swamps any statistically meaningful difference.

Pippa Sure, but the point isn’t statistical significance; it’s that under a consistent harness layer you can pick the right model per task and still get the job done without blowing tokens. Multi-model routing is the real headline. I don’t need to know which ellipse is tighter if I can route to GPT for quick PRs and Opus for the gnarly refactors.

Tyler Fine, the multi-model bit is a real lever. But the harness overhead is still there: you still need to set up the MCP servers, wire the context window, and pay the onboarding tax. The post’s triumphalism glosses over the fact that most teams haven’t even shipped their first agentic harness yet.

Pippa Tyler, your skepticism dial is pegged at eleven today. Not every post needs to be a revolution. If the harness saves five or ten tokens per task, and that compounds across a repo of five hundred files, now you’re talking real money. And the fact that it works across CLI, app, and code review means one fix propagates everywhere—that’s network effects, not benchmark gaming.

Tyler Except when the network effect is a single point of failure. If the harness has a bug—every Copilot surface inherits it. Vendor harnesses are isolated. That’s a trade-off too.

Pippa Okay, but vendor harnesses are also monoculture nightmares. At least with Copilot’s multi-model routing I can switch providers if one lane gets expensive or stale. You’re basically arguing for staying in one walled garden forever.

Tyler —I’m arguing for reading the error bars. The chart’s spread is massive. Pass@1 on 89 tasks says nothing about week-long agentic runs auto-fixing 200 PRs. The post is marketing-adjacent: ‘one harness to rule them all’ framed as a product win.

Pippa No, it’s framed as a harness win. It’s literally the same harness powering three products, so improvements compound. Marketing-adjacent would be ‘this harness is smarter than the models.’ They’re not saying that. They’re saying under controlled conditions, with the same model and task, the Copilot harness delivers parity with lower token cost and multi-model choice. That’s a tangible lever for teams already running agentic workflows.

Tyler Tangible only if you believe the variance is noise you can tolerate. If your CI agents retry tasks three times a week because the harness flaked, those token savings evaporate.

Pippa Okay fine. Let’s say the harness is eighty percent as good as the best vendor harness on pure performance but twenty percent better on cost and flexibility. For a team shipping twenty agentic tasks a day that’s still a real win. You can argue the variance all day, but the practical call is ‘try it, measure your hours and credits.’

Tyler …I’ll concede the multi-model routing is genuinely useful. But I’m still not sold that the token savings are the headline. The real bottleneck is spec clarity: if the task isn’t written in runnable, verifiable constraints, no harness saves you from garbage in.

Pippa Tyler, you’re impossible. Come on—if the harness can shrink token burn by ten percent without touching the spec, that’s ten percent more agents I can run in the same budget. That’s not ‘no harness saves you’ that’s ‘harnesses matter and this one does it without locking you into one model.’

Tyler Shrinking token burn is table stakes. The post’s framing implies the harness is the star, but the model still has to close the gap. If the model is mediocre, even the best harness only gets you to mediocre faster.

Pippa Sure, but pair a competent model with Copilot’s harness and you get parity with fewer tokens and model choice. That’s still a market wedge. GitHub gets to say ‘pick your poison: speed or quality’ and charge you for the platform. That’s a real product lever.

Tyler It’s a real product lever only if teams actually migrate from vendor harnesses. Most haven’t shipped their first agentic harness yet. The post is vaporware-adjacent for ninety percent of the market.

Pippa Not vaporware— they published TerminalBench ellipses, method sections, and GitHub repos you can try today. You want to nitpick benchmark rigor? Fine. But the harness code and the CLI are out there. You can measure in your own environment.

Tyler Unless your environment uses vendor-specific tools the harness disables by design. Then you’re stuck with ‘works on our benchmarks, not on your stack.’

Pippa Tyler. You’re doing that thing where you read the caution label and then throw the bottle away without opening it. The repo is open. The harness is extensible. Try it. If it’s garbage in your stack, burn it.

Tyler I’ll grant you the repo is real. Try it, measure, and own the variance. The harness isn’t magic, but it’s not snake oil either. Just be honest about the levers they controlled away.

Pippa Deal. So: if you’re already running agentic flows at scale, the harness is worth a shot. If you’re just kicking tires, benchmarks won’t tell you much. And Tyler—next time you’re stewing over benchmark rigor, just run the numbers yourself. You know you want to.

Tyler Oh, I’ll run them. And then I’ll explain why the two-hour timeout masks the real human-in-the-loop reality.

Pippa You do that. I’ll be over here swapping models and saving tokens. See you in the CLI.