Building a 100x Cheaper Trace Judge with Fireworks
LangChain Labs and Fireworks built a fine-tuned Qwen model to detect 'perceived error' in production agent traces—matching frontier performance at up to 100x lower cost. The model judges user-visible mistakes (corrections, rejections, repeats) from human-AI messages only, tested across two internal datasets (chat-langchain Docs Q&A and Fleet no-code agents).
Transcript
Justy Okay, LangChain just dropped a way to judge agent traces a hundred times cheaper using Fireworks.
Cody A hundred times…
Justy And it matches frontier models on this 'perceived error' thing.
Cody Perceived error?
Justy User thinks the AI messed up. Corrected it, rejected the answer, asked again, or the AI just admitted it was wrong.
Cody Right. So it’s not objective correctness. It’s 'did the human notice something off.'
Justy I had the worst flight yesterday—three-hour delay, then they lost my bag. Anyway —
Cody That explains the three coffees.
Justy This is why fleet traces. They trained it on their own Docs Q&A agent and Fleet.
Cody Hm. Two very different use cases, though. Docs is technical, Fleet is no-code everything.
Justy Exactly. That’s the generality test.
Cody Yeah. And they only used human and AI messages — stripped out all the tool calls.
Justy They think the signal’s in the conversation, not the plumbing.
Cody I mean, sure, a lot of it is. But tool calls can be where things go sideways. A failed database lookup might like a perceived error downstream.
Justy Fair. They even say that’s a lever they’ll pull later.
Cody But here’s the thing — they’re claiming frontier parity.
Justy Mm-hm.
Cody Against frontier model? On what eval? A hundred examples per dataset isn’t a lot to prove generality.
Justy Okay, but from a product angle, who CARES about perfect parity if it’s a hundred times cheaper?
Cody There it is. The classic Justy move: 'good enough and we can ship it.'
Justy I’m saying you can blanket-judge every trace now. Before, you sampled. Now, you can just run this thing on everything.
Cody Right. And the dataset prep is almost suspiciously simple — no trimming, no fancy prompting.
Justy That’s the point. It’s supposed to be plug-and-play.
Cody I’ll believe the cost when I see the exact numbers, but fine, the idea holds water.
Justy So teams with agents in production can finally see where users are actually hitting errors.
Cody Or where they they’re hitting errors. Still useful.
Justy Anyway. If you want in on the beta, they’ve got a sign-up.
Cody Of course they do.
Justy I’m grabbing one of your beers and calling this an Exploring Next win.
Cody That is such an Exploring Next take.