Harness S: Learning from Sibling Harnesses for Generalizable Agent Training

Asteria Okay, this one is basically asking whether your fancy agent wrapper can teach the model anything, or if it's just babysitting it better.

Draco Which is a real stuck point, right? We've had a bunch of harness papers now where the execution setup clearly matters… and then you put the policy back under the original setup, and a lot of the magic evaporates.

Asteria Every single time.

Asteria And nobody wants to ship twelve fragile orchestration branches forever. You want the training trick to cash out as a simpler product later.

Draco So the paper's move is pretty clean. Instead of hunting for one universally best harness, they keep a pool of sibling harnesses.

Asteria Siblings meaning variants built from the same base system, each with its own repair specialty.

Draco Right, and during training, if the base harness fails a task on all initial tries, they selectively re-run that task under a few sibling variants and learn from those rescue trajectories.

Asteria What I like is they start from the annoying reality, not the heroic version. Harness edits are local. They fix one class of mistake and break something else.

Draco Mm.

Asteria Which is way more believable than 'we found the one wrapper to rule them all.' Also, tiny side note… you've been on this sibling-harness kick for a while. One of those times where your weird fixation was, annoyingly, useful.

Draco Oh, that's good. You said it, not me. But it lines up with what we covered before, the field keeps rediscovering that harnesses have distinct strengths. This paper's another entrant, except here the harness complexity is mostly a training asset, not necessarily the thing you want to serve.

Asteria Honestly that's the practical consequence, right up front. If you're building an agent around a stable workflow, you may be able to spend complexity during training and buy simplicity back at deployment.

Draco Yeah. Mechanically they ground it in three properties. Locality… a harness tweak repairs some failures and regresses others.

Asteria Complementarity. Different siblings cover different mistakes, so the union beats repeated sampling from the base harness at the same rollout budget.

Draco And nonstationarity. The best sibling today might stop being the best after the policy learns a bit.

Asteria That middle one is the part product people will feel immediately. More retries inside one setup is not the same as retries with different execution support. Same model, different scaffolding, different path through the task.

Draco And the router's doing the hard part there. They build these, uh… what do they call them, the behavior… the—

Asteria Harness Behavior Profiles.

Draco Those, yeah. Basically evidence about when each sibling helps or hurts relative to the base harness. Then voting from multiple LLMs picks the top-k rescue harnesses for a failed task.

Asteria Top-k meaning just the few most promising variants, not the whole zoo.

Draco A small shortlist. And because harness utility shifts as the policy changes, they periodically re-anchor those profiles using fresh trajectories from the current policy.

Asteria So where's the evidence?

Draco AppWorld, SWE-Gym, and DBBench. Quick gloss, those are agent benchmarks for app-like tasks, software engineering tasks, and database tasks. The claim is the trained policy beats policy-only training when you put it back under the ORIGINAL base harness.

Asteria Which is the whole test. Otherwise it's just another harness win wearing a model-training hat.

Draco And they say held-out analyses support real transfer of harness-elicited behaviors into the policy itself. That's a plausible read to me. It doesn't fully close the attribution question, though… how much moved into the weights versus how much still depends on routing at inference.

Asteria Though they don't only pitch training. Reusing the router for rank-one inference gives extra gains at the same executor rollout count. So there are kind of two product modes. Train with siblings, deploy simply. Or train with siblings and keep a little routing layer in prod, if the economics work.

Draco Huh.

Draco That's where I'd be picky, though. If I were extending this, I'd want a cleaner ablation on the transfer story. Fixed base harness only, no router at inference, held-out task families… then show what behaviors persist.

Asteria Still, this feels more shippable-adjacent than a lot of harness work. Not for a one-shot chat product. But a team with a stable agent, repeated failure traces, enough volume to justify training? You build a library of specialized helpers, use them as temporary teachers, then try to collapse that into one calmer system.

Draco Mm.

Draco And the sibling pool stays fixed during policy training. I like that. It avoids the cost spiral of evolving both harness and policy together every round. You pay to build the pool once, then reuse it.

Asteria Fixed the whole time?

Asteria Which is very funny, because the field keeps trying to invent one perfect adaptive super-harness, and this paper is quietly going, maybe keep your weird little specialists and stop forcing them into one trench coat.

Draco Stop it. Trench coat.

Draco No, that's basically right. And they released code, which helps. The GitHub repo is DeepExperience slash Harness-S. Useful, because this is the kind of paper where the training loop and routing details matter more than the slogan.

Asteria All right, enough sibling drama for one episode. Go stare at the router code… I'll be over here wondering which teams actually have the failure data to make this worth it.