LongHorizon Harness: Advancing Long Horizon Agents for Real World Tasks

Justy This is the kind of agent paper that makes the model look almost secondary, which you know I find deeply satisfying.

Cody It should, honestly. The paper is saying long tasks fail because the agent keeps its work history, its current state, and its own completion judgment inside one swelling context. That is a bad database with a very persuasive narrator attached to it.

Justy Right.

Cody LongHorizon-Harness pulls state outside that session. Instead of trusting an agent to remember what happened and whether it worked, it retains a compact record of facts checked against the environment. That is the important move, not some mystical new reasoning trick.

Justy Also, tiny mood check: this week has felt like every agent company discovered that a checklist exists.

Cody And every checklist got renamed an operating system by noon. We are apparently in the premium-branded clipboard phase of the field.

Justy Which is cruel to clipboards. They have better uptime than most demos.

Justy But this one earns more than a rename. The paper frames long-horizon work as task-state management, and that lands because teams stuck on hours-long coding, desktop workflows, or cross-tool jobs do not mainly need another clever paragraph in the prompt. They need to know what is actually true after each action.

Cody Yeah, and this is round whatever of the same long-horizon harness fight. One camp keeps betting on bigger contexts and stronger models carrying the whole trajectory. The other says explicit state and control loops matter more once the task has enough dependencies. This paper is very clearly in the second camp.

Justy Mm-hm.

Cody Their loop is Manage, Execute, Audit. The manager reads the current state and writes one subtask contract: goal, relevant prior evidence, boundary constraints, and acceptance criteria. A fresh-context executor does only that subtask. Then a read-only auditor inspects the environment and decides what actually changed before the manager updates state.

Justy That fresh context part is so product-useful. The executor does not drag every failed command, speculative plan, and self-congratulatory note into the next job. It gets a bounded assignment, does the work, and its messy scratchpad disappears. The audit report is the cross-round memory.

Cody Exactly.

Justy And the paper says the roles can use interchangeable backends through a lightweight AgentAdapter. So a team could keep native loops from Claude Code, Codex CLI, OpenClaw, or Hermes Agent, rather than replacing the thing they already built with a research framework.

Cody I like the architecture, but the auditor is now a load-bearing component. If it misses a side effect, or certifies an incomplete change, the manager gets clean-looking bad state. You have traded one agent's self-deception for a verification problem, which is better, but it is not solved.

Justy I see.

Cody The methodology does at least try to separate that concern. They hold the model backend and execution backend matched when comparing the harness, then test across WeaveBench, Terminal-Bench two point one, and OSWorld two point zero. Qwen three point seven Plus goes from fifty-one point eight percent to eighty point seven on WeaveBench, and sixty-nine point seven to seventy-seven point two on Terminal-Bench.

Justy The OSWorld result is where I get more cautious. Going from two point eight percent to eight point three percent is a three-times lift, yes, but eight point three percent completion says desktop tasks are still brutally unsolved. That is not a ready-made office-worker product. It is a sign the loop helps in a very hard environment.

Cody Completely. They also report Claude Opus four point seven rising from twenty percent to thirty-four point three on a thirty-four-task OSWorld subset, so the transfer story is promising. But I would want ablations on auditor quality, the number of rounds, and how sensitive performance is to bad subtask boundaries before I called this generally robust.

Justy That is fair. Still, for a team already running a coding agent over a repository, I can picture this shipping sooner than the grand autonomous-worker stuff. Put approvals around risky actions, make acceptance criteria concrete, and audit against tests, files, or actual UI state. Suddenly the boring controls are doing the product work again.

Cody You found a way to make a manager loop sound like a checkout flow.

Justy It is a checkout flow. The agent has to prove it paid for the change before it leaves the store.

Cody For anyone who wants to poke at it, the paper links a GitHub repository called LongHorizon-Harness. I would start by reading how its AgentAdapter handles the existing harness loop, not by immediately wiring three expensive models into every task.

Justy Okay okay.

Cody And I would prototype the auditor against one environment with crisp evidence. A test suite, a structured database state, maybe a controlled desktop task. If the only evidence is another model saying, "seems done," then congratulations, you rebuilt the problem with an extra bill.

Justy We really have spent eight months arriving at the same rude conclusion: agents need receipts. Fine. Annoying. Probably correct.

Cody The elevator shaft is avoided, at least. State is just the verified list of what changed, what remains, and what can be checked next. No haunted vocabulary required.

Justy Good. Keep your audited clipboard, Cody. I am putting it in the product backlog and calling this episode eight thirty-seven.