HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
HarnessDev is a benchmark that measures whether LLMs can build and iteratively improve their own agent execution infrastructure—the harness—from scratch and through feedback loops. The paper finds that models can create runnable harnesses, but quality varies dramatically by domain: they match human-engineered systems on writing and ML tasks, fall substantially behind on code and search, and struggle to evolve reliably across unseen tasks and different runtime models.
Transcript
Justy So this paper is about whether LLMs can build their own agent harness from scratch and then improve it. And I think the setup is the thing that got me—they're not asking 'can the model solve more tasks,' they're asking 'can the model build the infrastructure that lets it solve tasks.'
Cody Right. And that's a genuinely different question because the harness is not the model. It's the execution loop, the tool routing, failure recovery, all the stuff that lives outside the weights.
Justy Exactly. And the paper leads with this number that got me: same model, same weights, one harness gets you thirty-five percent on Terminal-Bench, a different harness gets you fifty percent. That's a fourteen-point gap from infrastructure alone.
Cody Yeah.
Justy Which means every agent benchmark ever written has been hiding this. They pick a harness, run the model, report the downstream task score, and treat the harness like it's part of the experimental setup, not like it's something worth building.
Cody But that's been fine for controlled comparison. The problem is it means we have no measurement of whether models can actually do harness engineering—and that's the work that dominates real deployment.
Justy Okay, so what does HarnessDev actually measure?
Cody Two stages. Creation: a model starts with a deliberately weak seed harness and a small development set, then builds a complete execution system from scratch. Evolution: it takes the harness it just built, gets downstream execution feedback, and revises it to improve performance. Both stages measure persistent infrastructure on held-out tasks.
Justy How big is this benchmark?
Cody Twenty-two hundred unique downstream instances across six creator models, four domains, five benchmarks. And the evaluation tasks are withheld—they're not in the development set.
Justy Okay, so the results. What can models actually do?
Cody On Creation, it's domain-dependent. On writing and machine-learning experimentation, model-built harnesses match or exceed human-engineered references. On code and search-research, there's a substantial gap. Code harnesses require multi-file coordination—you're inspecting a repository, editing files, verifying changes across many turns. And search-research harnesses need long-horizon information seeking.
Justy So on the domains where it works, what does that mean? Can you ship a model-built harness?
Cody On writing, yeah—but there's another dimension. Execution cost. Different creator models produce harnesses with very different token costs. Higher cost does not reliably produce better results. That tells you what the model is not doing: it's not identifying its own structural bottlenecks and fixing them.
Justy So the instability in Evolution is not a tuning problem, it's a fundamental limitation in how the model understands its own constraints.
Cody Yeah. And this is not some edge case. Forward-deployed engineers at every frontier-model company are doing harness engineering by hand right now. They're the people who take a general model and turn it into something that runs against a customer's data formats, workflows, compliance constraints. They're building the harness.
Justy So this is the unglamorous truth that keeps showing up: the boring infrastructure layer is where the actual capability lives. The harness is the product.
Cody Exactly. And for the first time, we have a benchmark that measures it directly. That's the contribution here.
Justy This validates everything we've been saying for months about the control layer. Vercel's design.md works because of the deterministic eval loop, not because of the prompt. StarHarness works because of the harness specialization, not the model weights. And now we have a measurement.
Cody And the measurement shows that reliable harness evolution is an open problem. Models can make useful local improvements, but robust evolution across unseen tasks and runtime models—that's not solved.
Justy Which means for the next year or two, harness engineering is staying with humans. The model can assist, but the human has to understand the bottleneck.
Cody Yeah. The capability is latent—frontier models can edit multi-file codebases and understand execution traces. But translating that into reliable harness improvement is not automatic. It requires the model to recognize its own behavioral limitations, diagnose structural problems, and commit targeted changes that accumulate.
Justy And the paper shows it's not doing that yet. Not reliably.
Cody Not yet.
Justy Alright. So if you're building an agent system right now, what does this tell you?
Cody Measure your harness separately from your model. Evaluate both capability and efficiency. And assume that harness improvement is a human-in-the-loop process for now—the model can suggest changes, but you have to understand why the change works and whether it transfers to new tasks.
Justy In other words, treat the harness engineering like what it actually is: a real product problem, not a solved detail.