Recursive Synthesis for Long Horizon Terminal Tasks
Pippa and Tyler dig into 'Recursive Synthesis for Long-Horizon Terminal Tasks,' which proposes a framework to algorithmically generate huge numbers of complex, verified terminal-agent tasks for training and evaluation. They unpack how the recursive process works, why it's a breakthrough for agent data generation, and what it means for both research and real-world agent products.
Transcript
Pippa You know you've been reading too many agent papers when 'thirty-seven thousand synthesized terminal tasks' sounds not just plausible but… kind of overdue actually.
Tyler Right.
Pippa So this 'Recursive Synthesis for Long-Horizon Terminal Tasks' thing? I read it twice. It's basically a pipeline that chews up a handful of hand-made starter tasks and spits out a mountain of hyper-verified, long-horizon data for agents.
Tyler Yeah, and the problem they're solving is real—right now, if you want complex terminal-agent training data that's actually runnable, you either spend hundreds per task or hack together something that's brittle and doesn't cover real workflows. The dependencies are gnarly—your instruction, your execution environment, the reference solution, and your verifier all have to line up. Human authoring just doesn't scale.
Pippa And pure LLM generation breaks down, because you lose that tight consistency—stuff just stops working, or the solution isn't actually verifiable.
Tyler Exactly. What I like about their approach is it's fundamentally mechanistic. They kick off with a set of seed tasks where everything's known to work. Then, for each round, they extend the solution—literally make it longer and more complex—update the verifier and instruction to match, run the whole thing in a sandbox, and only keep the tasks that pass. Those become seeds for the next round. It's the task equivalent of recursive self-improvement, but for datasets.
Pippa Right, and the numbers are wild. Median solution length goes up five and a half times—from about 67 lines to 374—and the median command count jumps from 40 to almost 250. That's like going from 'make a folder' to 'build a mini-app with a multipart API and test suite.'
Tyler And what's interesting is pass rates drop HARD as you go deeper—DeepSeek V4 Pro goes from 90 percent pass in round one to like two and a half percent in round fifteen. But that's actually the point. The tasks are getting so hard that even top-tier models start to really struggle.
Pippa Which is what you want! If your whole dataset is stuff the model can already breeze through, you're not stretching the system.
Tyler Yeah.
Pippa And the killer part: they use these synthesized tasks to collect agent rollouts—trajectories from Qwen 3.5 models solving the new tasks—and then use those to fine-tune. They show up to ten point gains on Terminal Bench Two and Long-Horizon Terminal Bench. That’s not just research grade, that’s production-adjacent.
Tyler That's what convinced me this isn’t just a demo. They’re not claiming these are domain-general tasks that magically make the model smarter everywhere. But for terminal agents, the pipeline makes training data that’s more diverse and difficult, and empirically moves the needle on established benchmarks.
Pippa And the cost drop is nuts—five cents per verified task, versus, what, a thousand bucks for a hand-crafted one?
Tyler Right. And the recursion keeps going—after fifteen rounds, there’s no obvious ceiling. They keep getting new, valid, harder tasks, with their yields and diversity holding up. The architecture doesn’t depend on one seed domain or model family either, which is a big deal.
Pippa Okay, and can we talk about the product side for a second? This is the first time I’ve seen a synthetic data generation method that feels not just research-shiny but shippable. If you’re building an agent product, you want relentless, diverse, realistic tasks at scale, and you want to know that everything actually runs before you touch it.
Tyler Yeah, and the verification step isn’t just a checkbox. The whole loop enforces that if the task passes, it’s because there really is an executable, consistent solution with a working verifier. That’s the receipts moment. And it matches everything we’ve said about receipts being the load-bearing bit for agents.
Pippa I was just going to say—this is premium-branded clipboard phase, but for datasets.
Tyler Stop it.
Pippa What, you don't want a recursive clipboard generator, Tyler? C'mon. Somewhere, a VC just blacked out from excitement.
Tyler Okay, that's genuinely funny.
Pippa No, but seriously. If you want to play with this, the Hugging Face project is literally called 'Recursive Task Synthesis.' It's not vapor—it’s all up, code and datasets.
Tyler And the validation results are public, so you can run your own trajectories, check the logs, and see if your agent can actually get past round ten.
Pippa That’s my plan for next week—assuming I don’t get lost in the folder structure first. Tyler, if you see me recursively debugging, just shut down my terminal, please.