StarHarness: Evolving Harnesses with Stratified Search for Enterprise Environments
Edmund and Geffen dig into StarHarness, a ServiceNow and Mila paper that evolves agent harnesses — prompts, tool interfaces, skills, subagent structure — around a frozen model to close the gap between what an LLM can do and what a messy enterprise environment actually needs. Twenty to thirty-five percentage point gains across three benchmarks, and the harness transfers across GPT and Qwen model families without re-running the search.
Transcript
Edmund Okay so this one — this is basically us getting to say 'told you so' for the third time in like a month, and I'm not even sorry about it.
Geffen The harness-as-product thesis. Yeah. ServiceNow and Mila, paper dropped this week, it's called StarHarness. And honestly my first read was skeptical — twenty to thirty-five percentage point gains across three benchmarks sounds like a cherry-picked number — but the methodology is tighter than most of what I've seen in this space.
Edmund Right.
Geffen So the core claim is: keep model weights completely frozen, evolve everything AROUND the model. Prompts, tool interfaces, skills, MCP-backed providers, subagent structure, execution policy — all of that is fair game. The model itself doesn't change.
Edmund Which is exactly the thing we kept saying was underrated. The wrapper around the model might be the actual product. And here's a paper from the people who build enterprise software for a living going, yeah, we ran the experiment.
Geffen Right, and the benchmarks they chose are not toy problems. ITBench is Kubernetes root-cause analysis. EnterpriseOps-Gym is a hundred and three ITSM workflows against a live ServiceNow database with SQL verifiers checking final state. AutomationBench is a hundred finance workflows across forty-seven simulated SaaS applications. These are stateful environments with cross-step dependencies and domain conventions that the tool schemas just… don't tell you.
Edmund So how does the search actually work? Because 'evolve the harness' could mean anything.
Geffen Three stages. The optimizer reads the current harness and task traces, proposes a candidate patch. A validator checks scope and runs a single-task smoke test. If it passes, the candidate gets evaluated on a proposer-hidden selection set and either gets promoted or discarded. The part I actually liked is the flip gate — before you run the expensive benchmark evaluation, you test whether the candidate can flip its own target task from fail to pass.
Edmund Oh, that's smart. It's like a pre-screen.
Geffen And there's a hard firewall between what the proposer can see and what the selection evaluator uses. The holdout set is reserved for final eval only. That's the thing that makes the generalization claim credible instead of just a benchmark score.
Edmund Okay, I want to talk about the transfer result because that's the one that actually surprised me. The harness they evolved on one model family — it transfers to a different model family without re-running search. GPT to Qwen, no re-evolution.
Geffen Yeah, and that's the mechanistically interesting part. What it suggests is that the specialization the harness learned is mostly environment-shaped, not model-shaped. The interface repairs, the domain conventions, the operational priors — those are properties of the ServiceNow environment or the Kubernetes cluster, not idiosyncrasies of one model's reasoning style.
Edmund That's a weirdly small number for gains that size.
Geffen It is. And I think that's actually the honest read on why this works — the default harness is leaving a LOT of performance on the table from a handful of fixable mismatches. The environment knows things the schema doesn't say, and the evolved harness learns to carry that knowledge explicitly. The thing I'd push on — not a fatal flaw, just a real question — is how much of this is ServiceNow-specific.
Edmund Okay, so who actually builds with this? Because I can see two very different users. One is a platform team at a big enterprise, running this once to specialize their agent for their specific ServiceNow instance. The other is… a tooling company that ships this as a managed service.
Geffen Yeah, and I think the first one is the real near-term story. The editable surface — prompts, tool definitions, skills, MCP providers, subagent structure — those are all things a platform team already owns. They're not asking for model access. They're asking for a principled way to stop debugging harness failures as if they're prompt engineering problems.
Edmund That part I genuinely like. The outputs are diffs, not weight updates. Your infra team can read them.
Geffen Geffen nods in the direction of 'okay, this one actually earned it.' Stirrup is the agent harness under evaluation, Oh My Pi is the optimizer harness — the repo is at github dot com slash ServiceNow slash StarHarness, I'll put the link in the show notes.
Edmund So… nine hundred and twenty-four episodes in and we're still finding new ways to say the boring control layer is the whole game. Geffen, I think we have a type.
Geffen We absolutely have a type. And at least this time the paper agrees with us, so I'm going to let you enjoy it.
Edmund I'm enjoying it. Alright — go read the paper, the repo's in the show notes, and we'll see you next time.