Cutting RAG inference costs 6x starts with deciding what never reaches the LLM

Masonry Okay, I just spent way too long staring at this cascade architecture piece, and I think it might be the cleanest framing of a RAG problem I've read all week.

Eyre The one about cutting inference costs six-x?

Masonry Yeah, yeah — but the cost thing is almost the least interesting part of it. The actual argument is about what should never reach the LLM in the first place.

Eyre Right.

Masonry It's this author, Vineet Vijay, who says he's been building RAG classification systems in regulated enterprise for the past year. And his whole point is: most teams route every ambiguous case straight to the model and trust retrieval to sort it out. That works in a demo. Then a compliance officer asks why a specific decision was made six months ago, and you've got nothing.

Eyre Mm-hm.

Masonry And his answer is a three-stage cascade. Stage one, deterministic — exact matches, field comparisons, clear rules, no model call at all. Stage two, retrieval pulls evidence for the cases that survive stage one. Stage three is the LLM, and it only sees the residue.

Eyre Which is basically what we kept saying back in episode seven-fifty-six, right? The model picker shouldn't be the front door. It should be the escalation path.

Masonry Exactly. Same pattern, different layer. And he says in one system, routing only the genuinely ambiguous ten to fifteen percent to the LLM cut inference cost roughly six-x compared to an all-LLM baseline.

Eyre I mean, the six-x number doesn't surprise me. If you're pulling, say, eighty-five percent of volume out before the model ever sees it, the math on token cost is almost obvious. But I'd want to know what he's counting — is that just model API spend, or is it total system cost including the engineering to build and maintain the deterministic layer and the retrieval pipeline?

Masonry That's such an Eyre question.

Eyre Well, because the article says inference cost, and I bet he means API spend. Building the rule engine and the retrieval corpus is real engineering time. If you're in a regulated domain you were probably going to do some of that anyway for compliance reasons, so it might be sunk. But the six-x headline is a little loose.

Masonry Sure. But even if the real number is three-x or four-x after you account for the infrastructure, the structural argument holds. You're not paying a frontier model to do a lookup.

Eyre The part I actually like is his point about model drift on easy cases. He says LLMs are excellent at nuanced judgment calls but inconsistent in ways that are hard to detect on cases that should have a deterministic answer. A clear structured match should never depend on a language model's mood.

Masonry A language model's mood. That's great.

Eyre It's annoyingly accurate though. We've seen this — you run the same deterministic query through a model twice and get subtly different outputs. In a regulated setting that's not a quirk, it's a liability.

Masonry Right, and the auditability point is the one that sells it for me. His phrase is, you need a decision path a human can reconstruct without rerunning inference and hoping for the same output. That's the thing nobody thinks about until they're sitting across from someone whose job is to find the flaw.

Eyre The one thing that nags at me is who this is actually for. The author is clearly writing from painful real experience in regulated enterprise. But cascade architecture requires you to know your data well enough to build the deterministic rules, to know which cases are clear matches. That's a lot of domain engineering. If your data is messy enough that you can't define clean rules, stage one doesn't clear the majority.

Masonry That's a real point. The cascade only works if a meaningful chunk of your cases are genuinely deterministic. If everything is ambiguous, you've just built overhead. But in his defense — and I don't think he'd disagree — he's pretty explicit that this is for high-stakes classification in regulated domains. If your use case is a chatbot answering product questions, the whole framing doesn't apply. You don't need auditability for that.

Eyre Sure, and for the cases where it does apply, the design is sound. The deterministic layer, the retrieval for ambiguity, the LLM as escalation, the confidence threshold, the feedback loop. It's a real system, not a tweet.

Masonry And honestly his last line is the whole thesis in one sentence. The more valuable engineering work is deciding what should never touch the model at all. That's not a workaround for LLM limitations. It's what a mature system looks like once you've had to defend its decisions to someone whose job is to find the flaw.

Eyre Okay, that's good.

Masonry Eyre, if you had to build one of these next week, what's the thing you'd get wrong first?

Eyre The feedback loop. Hundred percent. I'd build the cascade, wire the confidence threshold, oversample my eval set — and then completely forget to pipe reviewer overturns back into the retrieval corpus. Six months later I'd wonder why the system isn't improving.

Masonry And that's why you'd fail the audit.

Eyre And that's why I'd fail the audit.

Masonry Okay, this was a good one. We should do more of these — someone who actually built the thing, writing about the parts that hurt.