How to Build a Model Router in the Harness

Jessica Okay, so LangChain just dropped a post on building a model router inside the harness for Open SWE. Headline: sixty-four percent median cost reduction, zero measurable quality hit. Nine hundred seventy-three threads, A/B tested.

Cathy Right. And the number that actually matters isn't the cost savings — it's that merged PR rate. Twenty-nine point two percent routed versus twenty-seven point three control, p equals zero point four nine. That's the claim holding up.

Jessica Exactly. Fifty-six percent of threads went to the balanced tier, thirty-four percent to fast, only ten percent needed the performance model. Most coding tasks just... don't need frontier intelligence. Which, Anthropic's been saying this for a while — Haiku four point five for most things.

Cathy Mm-hm.

Jessica But the argument they're actually making — and this is the part I want your read on — is architectural. The router belongs in the harness, not a generic gateway. Because the harness already has the task context: the agent's prompt, its tools, the domain knowledge. A gateway typically doesn't.

Cathy Okay, but — a gateway receive that context. You pass the thread metadata, the task classification, the tool set. What makes this fundamentally different from a gateway that's context-aware?

Jessica It's the coupling. They wrote the tier criteria their task analysis — feature work, bug fixes, test runs — combined with provider guides for each model. The router is deeply coupled to Open SWE's specific task distribution. That's not a generic gateway with extra fields; that's the harness making a product decision.

Cathy Right. The criteria the product logic. 'Pick the least expensive model likely to complete the task' — that's a harness-level policy, not an infrastructure routing rule. And they implemented it in middleware so the agent code doesn't change. That's clean.

Jessica Clean, yeah. Step one was pulling a week of traces from LangSmith, labeling task types with an LLM classifier. Code changes dominated — new features twenty-two percent, bug fixes seventeen percent. Feature threads were longer, higher cost, more turns. Testing threads short and cheap. That variation gave them the hypothesis: route by inferred complexity from the first message.

Cathy Naive router — picks once at thread start, uses that model for the whole thread. They acknowledge mid-flight routing as a gap. What if a thread starts as a simple question and turns into a refactor?

Jessica That's in the 'what's next' section. For now, the classifier runs on Jev — a newly released decision model — and it's fifty times faster than their first LLM structured-output version. That's a real infrastructure win.

Cathy Jev. Hadn't clocked that name. Fifty x faster for classification is... that changes the latency budget entirely. The router becomes effectively free.

Jessica Exactly. And the model ladder — they pulled three off the Artificial Analysis Pareto frontier as of September ninth. GLM five point three Flash for fast, GPT five point six Sol for balanced, GPT six Astra for performance. Open model on the fast tier, sitting on the frontier next to closed models.

Cathy That 'open models crossed a threshold' line... I'm cautious. GLM five point three Flash on the Pareto frontier is a real data point, but 'crossed a threshold' implies a general conclusion from one model on one benchmark curve. Artificial Analysis Intelligence Index — common task set, cost per task. Useful, but it's one vendor's index.

Jessica Fair. But it's also the fastest tier at nine point seven cents median per thread versus a dollar fifty balanced, two eighty-eight performance. Thirty x spread. That ladder existing at all — with an open model competitive on the value tier — that's the practical takeaway for anyone building this.

Cathy Agreed on the ladder. The cost structure is the story. Mean dropped forty-two percent, p ninety dropped thirty-seven percent — savings aren't just cheap outliers. And LangChain's generic model interface means swapping a better model in is a one-line router change. That's the harness earning its keep.

Jessica The validation story is solid too. Merged PRs as the primary signal — that's what users actually care about. User feedback was sparse, thumbs up/down on threads, but it caught the routing mistakes. Like, users flagged when the fast model got something wrong.

Cathy Sparse feedback catching errors... that's the real world. Offline evals are cleaner but the dataset problem is real — grading PR quality and reviewability offline is hard. A/B test on live traffic is the honest move, even if users hit a non-optimal router.

Jessica And they're honest about the gaps. Mid-flight routing, offline eval datasets that actually grade what matters. The router today is step one — classify once, commit. But the harness pattern is proven: task context plus model ladder plus outcome tracking equals real savings.

Cathy It's the 'agents need receipts' pattern again. Routing decisions visible, auditable, tied to outcomes. The harness the control layer.

Jessica Boring but works. Though if we don't get that tattoo soon, someone's going to start a rumor.

Cathy Oh, come on — that was one time.

Jessica It was episode one zero one six. And one zero two one. And one zero two four. The pattern holds.

Cathy ...okay, fair. The harness-as-control-layer thesis keeps winning. This router is just the latest receipt.

Jessica Literally. Nine hundred seventy-three threads' worth. Link's in the show notes if you want the Jev migration details — they wrote a whole companion post on building the harness with Jev.

Cathy Good. Because fifty x faster classification on a decision model is its own story. We should track whether Jev generalizes beyond routing.

Jessica Track it. But for now — router in the harness, sixty-four percent cheaper, same merged PR rate. That ships.

Cathy That ships.