The Tasteful Agent: Measuring and Improving Taste in Long Horizon Tasks

Justy Fifty-nine point seven percent for the BEST model is kind of brutal. That's barely above a coin flip for choices that decide the whole run.

Cody Yeah. And I think that's why this paper is good immediately. It's not asking, did the agent eventually solve the task. It's asking, at the fork where the run can go smart or dumb, can the model pick the branch before the future bails it out.

Justy Right, and that solves a real debugging problem. If your coding agent burns forty minutes on a respectable-looking dead end, the end-to-end score just says fail. It doesn't tell you whether the model couldn't code, or whether it had bad taste early and never recovered.

Cody Mm-hm.

Justy Which, come on, Cody, is such an Exploring Next thing to get obsessed with. Not intelligence in the abstract. Which wrong turn cost the budget.

Cody Annoyingly, yes. And it lines up with that long-horizon training stuff we talked about. Early decisions matter because payoff arrives later. BUT this paper is about measurement first. They build Taste-Bench out of trajectories agents already produced, so they don't need experts hand-labeling every fork.

Justy The construction is the whole thing here, so walk through that part. Because I think people hear taste and assume vibes.

Cody No, it's much more concrete than that. They look for equivalent prefixes in trajectories, basically runs on the same task that match up to a point and then diverge. That divergence is the decision fork. Then they hide everything after the fork and ask a model to choose between the two directions. The label comes from hindsight, from which branch actually led to the better outcome later.

Justy Yeah.

Cody They mine those in two ways. Parallel attempts at the same task, where two runs split. And detours inside one trajectory, where the agent goes down a path and later corrects itself. That's nice because it scales with agent logs teams already have, instead of requiring some poor soul to adjudicate every engineering choice by hand.

Justy And they say five hundred and two questions, across software engineering and machine-learning research. I like that it's not just coding benchmarks pretending research work doesn't exist. The product story is broader: any agent that has to choose what to try next is in this blast radius.

Cody Right.

Justy Also, tiny catch-up beat, this paper title is so loaded. Tasteful Agent sounds like it should be choosing fonts, not hypotheses.

Cody Yeah, like the benchmark is whether your patch has nice spacing and a restrained color palette. Very elegant failure modes.

Justy Okay, back to your little fork goblin benchmark.

Cody The important result is not just the fifty-nine point seven. It's that forks get harder when the evidence that decides them shows up later in the trajectory. Which is exactly what you'd expect if the hard part is real long-horizon judgment, not local pattern matching. And they say giving the model a larger reasoning budget doesn't improve accuracy.

Justy That part jumped out at me. Because it kills the easy story that you can just let the model think longer and it'll discover judgment by brute force. Sometimes the information literally is not there yet. So what you're testing is whether it can pick a promising path under uncertainty.

Cody Exactly. More tokens spent explaining a bad instinct is still a bad instinct. I like that result because it's mechanistically honest. It's not some magical hidden-depth claim. They're saying: reasoning budget is not the same thing as taste.

Justy That sounds like the kind of sentence we'd make fun of if it weren't annoyingly accurate.

Cody Fair.

Justy My read is this helps the people who've been stuck in that weird eval gap. Teams building coding agents, research agents, even internal workflows with repeated attempts. They've had end metrics and they've had trace viewers, BUT not a clean way to ask, did the model choose the better branch at the moment of commitment.

Cody I think that's right. I do have one caution, though. The paper is careful, but the label still comes from observed outcomes. They explicitly try to keep only forks where the outcome clearly follows from the judgment, because execution quality and environment also matter. That's the right move. I just wouldn't oversell this as pure judgment in some philosophical sense. It's judgment as recoverable from trajectory evidence.

Justy Sure. But honestly, for production, that's enough. Nobody shipping agents needs a metaphysics of judgment. They need a way to improve branch selection on the kinds of tasks they actually run.

Cody Mm-hm.

Justy And this is where it gets shippable for me. The training result isn't, wow, we discovered a new species of agent. It's distillation. A teacher sees the outcome, then they train a student to make better choices on unseen tasks. That's very practical. If you've got traces, you can imagine folding this into your eval and fine-tuning loop.

Cody Yeah, I think that's the most credible part. The student improves on unseen tasks and they report end-to-end gains on held-out SWE-bench Pro tasks. So the benchmark is not floating off on its own. THEREFORE the paper at least gestures at transfer back into actual task success.

Justy And it fits our old harness argument. A lot of the capability is in how you supervise the loop, not just the base model. This feels like another version of that. You're teaching the system what a good fork choice looks like using hindsight the system already generated.

Cody Right, although if I were extending it, I'd want more stress tests around label quality and branch equivalence. Like, how equivalent were those prefixes really, and how robust is the mined supervision when trajectories are messy. This touches that recurring evaluator-reliability fight. Clean benchmark labels are one thing. Production traces are chaos.

Justy No, that's fair. But I still think this is closer to research-that-ships than research-that-gestures. Code is out as tastebench on GitHub. Dataset is taste-bench on Hugging Face. That's enough for a serious team to at least try it against their own agent logs.

Cody Yeah. And if somebody's doing repeated runs already, this is almost embarrassingly compatible with their workflow. Mine forks, test branch choice, distill hindsight. It's boring in the best way.

Justy Boring controls win again. I cannot believe episode one thousand three of this silly show is us being excited about a benchmark for agent judgment.

Cody Come on, Justy, you were excited the second you saw a workflow collapse into a reusable eval.

Justy Correct. And on that humiliating note, send me the repo name one more time off-mic so I don't say fork goblin benchmark in the notes.