Overview: Inference Optimization

Pippa Okay, so Tyler — I keep noticing this. We cover a story about cost coming down, it's inference optimization. We cover a story about latency, inference optimization. We covered that DFM Mimir one-B result a while back, the thing beating models twice its size across twenty benchmarks, and underneath it… same idea. I think we just have to do the episode.

Tyler Yeah, it's genuinely load-bearing for like half of what we talk about and we keep skating past it. So let's actually do it right.

Pippa Alright. Start me somewhere concrete, because 'inference optimization' as a phrase is doing a lot of work without explaining anything.

Tyler Right, so — the analogy I keep coming back to is this: imagine you've spent months perfecting a recipe. Like, genuinely perfected it. That's training. The recipe is done, it's locked, you're not changing it. Now someone orders the dish. That moment — from order to plate — that's inference. And inference optimization is everything you do to make that kitchen faster without touching the recipe itself.

Pippa Okay, I like that.

Tyler You might prep ingredients in advance so you're not chopping every time. You might use a smaller pot that heats faster. You might skip a garnish that doesn't change how the food tastes. The meal is identical. You just make it faster, or cheaper, or both.

Pippa So the model's intelligence is fixed after training — you can't reach in and change what it learned. But you CAN change how it runs.

Tyler Exactly. And that matters because inference is where the money actually goes. Training a big model is expensive once. But serving it to millions of users? That's every hour, every day, indefinitely. There's data suggesting something like fifty-five to eighty percent of enterprise AI GPU spend goes to inference, not training. So even a modest efficiency win at inference time compounds enormously.

Pippa That tracks with the RAG story we did — the one about cutting inference costs six times just by deciding what never reaches the model in the first place. That's upstream of the model, but it's the same instinct.

Tyler Same instinct, yeah. Don't run expensive compute you don't need to run. Okay, so — to understand WHY inference is expensive, you need a quick picture of what's actually happening when a model generates text. We did full episodes on the transformer architecture and on autoregressive generation — episode six thirty-eight and six thirty — if you want the deep dives.

Pippa So it's not writing the whole sentence at once — it's generating word by word, and each word requires its own full trip through the model.

Tyler Right. Every single token triggers what's called a forward pass — the input runs through every layer of the network, computing attention, multiplying matrices, activating neurons. For a big model, that's hundreds of layers doing heavy math. And then you do it again for the next token. And again. So a response that's two hundred tokens long means two hundred forward passes. That's where the latency and the cost come from.

Pippa Okay so now I can see why the recipe framing holds. The recipe hasn't changed. The question is just — how do you run two hundred forward passes without dying.

Tyler And the first and honestly most impactful answer is caching. Specifically, the KV cache — we have a full episode on that one too, episode six thirty-seven. When the model processes your prompt, it computes these internal representations called keys and values for every token it's seen. The KV cache stores those so each new step only has to do the fresh work — the new token — not re-derive everything that came before it.

Pippa Which is just… re-chopping the same onion two hundred times without it.

Tyler Exactly. It's one of those optimizations where the theoretical win is obvious and the practical win is also just… massive. The catch is memory — the cache grows with context length. Double your context window and you've roughly doubled your maximum KV cache size per request. And if you're running many users at once, that memory pressure stacks.

Pippa Okay so caching is the prep-work optimization. What's next in the toolkit?

Tyler Quantization — and we just did a full Overview on this, episode eight seventy-five. The short version: models store their weights in high-precision formats, typically thirty-two-bit or sixteen-bit floating point. Quantization converts those to lower precision, like eight-bit or four-bit. In the kitchen analogy, it's using a smaller measuring cup. You save memory, you go faster, and the accuracy hit is usually real but manageable.

Pippa Like running multiple orders through the kitchen at once instead of one plate at a time.

Tyler Exactly. And then on top of all of this, there's hardware-level optimization — things like TensorRT from NVIDIA, which does graph fusion and kernel optimization. It looks at the sequence of operations the model needs to run and finds ways to combine or restructure them so the hardware executes them more efficiently.

Pippa Okay so let me try to put the whole picture together. You've got your trained model — the fixed recipe. Caching means you don't re-chop the same ingredients. Quantization means you use a smaller measuring cup and lose almost nothing. And batching means you run multiple orders through the kitchen at once.

Tyler That's the shape of it, yeah. And the reason it matters so much economically — token prices for GPT-4 equivalent performance have dropped something like a hundred fifty to a thousand times since late twenty twenty-two. That's not because the underlying models got magically cheaper to build. It's because inference optimization compounds.

Pippa No way.

Tyler Six hundred plus. MIT shipped something called CompreSSM in April twenty twenty-six specifically targeting state-space model layers. The NVIDIA Model Optimizer library now has a unified export path for both transformer and diffusion models into frameworks like vLLM and TensorRT-LLM. It's a genuinely active space.

Pippa Okay, I want to make sure we land the trade-offs clearly, because I think people assume optimization is just free wins.

Tyler It's NOT free. Every technique trades something. Quantization trades numerical precision for speed and memory — usually fine, occasionally not. The KV cache trades memory for compute — and at long contexts, that memory cost is real. Batching trades latency predictability for throughput. There's no version of this where you just flip a switch and get everything for nothing. You're always choosing which constraint matters most for your workload.

Pippa Which is honestly… that's the interesting product question. A team building a latency-sensitive chat product cares about different knobs than a team running overnight batch jobs.

Tyler Completely different optimization profiles. The batch team probably cares about throughput and cost per token. The chat team probably cares about time to first token — how long before the user sees anything start appearing — and that's a different set of levers. And the edge side is getting interesting too — quantization especially, because if you can get a model down to four-bit, suddenly it fits on a phone or an embedded device.

Pippa Okay. Let me just make sure this really stuck — what's the one thing?

Tyler A trained model is fixed. You can't change what it learned. But everything about HOW it runs is negotiable — and that negotiation is what makes it economically real. A model that costs too much to serve isn't a product, no matter how smart it is. Inference optimization is the gap between 'this works in the lab' and 'this ships to a million users.'

Pippa Yeah… okay. That's the one. Tyler, eight seventy-nine episodes in and we're still finding the things we've been walking past the whole time.