Improving Agents is a Data Mining Problem

Laura Wait — 'improving agents is a data-mining problem'? That sounds insane on a podcast.

Harper Mm-hm. But then the post spends seven minutes arguing that continual learning, harness engineering, and post-training are all just curating data at scale from agent traces.

Laura So every continual learning company is secretly an observability company — got it.

Harper Right. The paper trail of traces is the currency: it’s how agents project their experience into data we can mine, then feed back into the system.

Harper They even coin a cute term — 'scaling dreaming' — for distilling long-horizon signal from millions of long traces.

Laura Okay, okay — I’m sold on the poetry. What’s the actual mechanism? Because I’m not moving a roadmap based on poetry.

Harper Start with traces, run evals, mine the traces to find what to improve. Then curate evals as training data and hill-climb. Rinse, repeat. That’s the recipe.

Laura So the agent’s trace is the test set, the eval is the train set, and I’m fine-tuning my judge model to judge the traces that judge the agent. Is that the loop?

Harper Exactly. They fine-tuned a small open judge to mine signals across their tracing project and say on narrow tasks it beats closed frontier models while being orders of magnitude cheaper.

Harper They then plug that signal into LangSmith Engine, which reads every trace, finds issues, creates code fixes, auto-commits useful context, and supposedly improves the agent over time.

Laura Wait — the engine is a product? The article is a thinly veiled product pitch wrapped in methodological language.

Harper Well, yeah. The paper trail of traces really is the currency, and if you’ve got the infra to mine them, it’s high-leverage. Terminal Bench 2.0 saw a thirteen-point-seven percent lift just from tweaking the harness and hill-climbing correctness metrics.

Harper So the practical takeaway is: get agents out early, mine the traces, auto-research the harness against your evals, and only fine-tune when the harness hits an intelligence ceiling or the inference bill explodes.

Laura Harness-first, fine-tune, harness-second — that funnel? Super. Until it collapses into 'scaling dreaming' as the new buzzword for 'we’ll figure it out later'.

Harper Yeah, no — that’s the spot where the argument strains. They drop the term like it’s a mechanism, not a placeholder for 'we hope the data’s useful'.

Laura I mean, it’s not wrong — agents are producing more data than humans have in history. But mining which signals matter in millions of long traces? That’s infra you have to build, not a slide title.

Harper Which LangSmith Engine is built to do. It’s a product, sure, but the underlying claim isn’t about LangChain — it’s that trace mining is the high-leverage muscle.

Laura Fine. Let’s say I buy it — who should actually care? Only teams already drowning in agent logs?

Harper Teams shipping agents into production, full stop. If your feature set depends on agents doing something repeatable, you’re already in the data-mining business whether you like it or not.

Laura So the practical win is faster iteration: collect traces, mine them, auto-patch the harness, and only then reach for the fine-tune sledgehammer. That’s it?

Harper That’s the funnel. Harness engineering gives high-bandwidth feedback, fine-tuning reshapes the model for your tasks, and second-pass harnessing checks generalization.

Laura It still reads like a LangChain shill piece disguised as methodological advice. But I’ll grant: the Terminal-Bench 2.0 harness work and the Circuit Breakers harness are legit repos to steal.

Harper LangSmith Engine’s code is open too if you want to fork it. And they’re using Qwen-3.5 and DeepSeek-V4 as the judge models — cheap, open, and fine-tunable.

Harper If you’re running agents at scale, mining traces is table stakes. You don’t have to adopt LangSmith; just build the infra.

Laura Yeah, no — I obviously didn’t sleep on this — literally. Go fork the harnesses, run the evals, and let me know if the traces actually tell you something useful.