Tuning the harness, not the model: a Nemotron 3 Ultra playbook
A LangChain/NVIDIA case study claims harness tuning alone can push Nemotron 3 Ultra to 0.86 on Deep Agents at ~$4.48/run vs $43.48 for Opus 4.8, with parity latency. The hosts parse the real mechanism (matched scaffolding vs post-training), test limits (where harness hits a ceiling), and weigh who actually benefits. They surface concrete repos (langchain-ai/deepagents, langchain-ai/deepagentsjs) and a vendor profile workflow, then poke at the article’s reliance on Deep Agents and the cost math. Final take: a plausible win for teams already deep into harness work, not a universal unlock, with the hosts pushing back on ‘ten-x cheaper’ framing and under-specified benchmark footnotes.
Transcript
Echo Okay, so the headline here is: tune the harness, not the model, and you can get near-frontier agent quality for a tenth the cost. Nemotron 3 Ultra hits 0.86 on Deep Agents, Opus 4.8’s best is 0.87, both at latency parity, but Nemotron’s run clocked in at four bucks forty-eight versus forty-three bucks forty-eight. That’s a real delta if it holds.
Onyx Yeah, but it’s not magic. Nemotron’s already post-trained for agentic behavior, so the scaffolding actually fits. The article’s point isn’t 'buy a cheaper model and call it a day'—it’s 'match the harness to the model’s baked-in capabilities.' That’s a product lever, not a model swap.
Echo Mm-hm. Sure. But the jump from generic harness to tuned harness is what they’re selling as the win. And I’m not convinced that’s not post-training in disguise—letting the model coast on existing muscle instead of adding real capability.
Onyx That’s the exact split they flag: if every harness tweak flattens out and the score doesn’t move, the ceiling isn’t the scaffolding—it’s the weights. So they’re using harness work to surface where the model’s actually thin, not to fake a win.
Echo Wait—you’re defending the claim that they’re not gaming the benchmark?
Onyx I’m defending the mechanism they cite for avoiding gaming: every change runs through a trace-driven loop, cheap first, then only promoted if it repeats across trials and doesn’t regress anything else. That’s actual discipline. The article’s not telling you to rewrite the system prompt broadly; it’s telling you to write a two-line block that fixes one observed failure in the traces.
Echo Okay, fair. But a two-line block that fixes one edge case is still not the same as improving the model’s underlying planning depth. That still looks like scaffolding work, not capability work.
Onyx Sure, but if the scaffolding mismatch is what’s making the model look dumb, then the harness change is real. Their Terminal-Bench example—gpt-5.2-codex from 52.8 to 66.5, same weights—shows the gap is the fit, not the model. That’s a real effect teams ship against every day.
Echo I see … but Terminal-Bench is proprietary data, and Deep Agents isn’t exactly a neutral benchmark. They’re using a single suite, and the cost math is pegged to LangChain Deep Agents infra, not a neutral harness. That’s a footnote the article buries.
Onyx Okay, but that’s the part I don’t buy—your point isn’t that the result is wrong, it’s that it’s not generalizable outside LangChain Deep Agents. Which is fair. But the mechanism—treat evals as training data for harness work and screen cheaply before promoting—is still sound product advice for teams already deep in agent harness code.
Echo Teams already deep in agent harness code are exactly the ones who don’t need this article—everything they’re doing is harness tuning, day in, day out. The interesting question is whether this changes anything for everyone else.
Onyx Right, right. And the answer is: yes, if you’re starting from a generic harness and an open-weight model you control—Nemotron’s on Hugging Face, you can host it yourself or grab it from any cloud provider without lock-in. The harness is the part you actually own. The article’s strongest moment is the middleware example: same rule, moved from system prompt to tool return value, and suddenly the agent reads the guidance right next to the data it’s looking at. That’s not model s
Echo Okay, okay. One concrete question: Deep Agents is their eval, they released a per-model harness profile for Nemotron, and they point to two repos. If I want to poke at this, what do I actually clone?
Onyx langchain-ai/deepagents—that’s the harness—and langchain-ai/deepagentsjs if you’re in TypeScript. The profile’s in a branch called split-evals, and the NVIDIA profile YAML formalizes the short blocks and injected messages they talk about. You can drop it into your own Deep Agents fork and see if the 0.86 holdup is real under your tokens and your GPU mix.
Echo And the cost math per run … I’ll believe it when I see my own infra bill. They’re using LangSmith traces and LangChain Cloud for the numbers, so it’s still vendor math.
Onyx Which is why you’ll clone it and run it locally. Then we’ll know if the ten-x cheaper claim survives the move from LangChain infra to your garage rack.
Echo …Or if Nemotron’s MoE and the vendor defaults just make my bill bigger. Either way, I’ll report back.
Onyx Do it. And while you’re at it—remember when we swore off ‘harness profile’ as a term because it sounded like another buzzword du jour? This is the fourth harness profile in as many weeks.
Echo Yeah, no. It’s probably the fifth. We’re calling everything a profile now.
Onyx Exactly. This is episode six twenty-four, and I’m still not buying the name. Anyway—have fun benchmarking, Echo.