Overview: Knowledge Distillation

Justy Okay so — knowledge distillation. We have said those two words on this show probably a dozen times and every single time we go 'we should really do a proper episode on that.' Episode nine twenty and we're finally doing it.

Cody Yeah, no more punting. It keeps coming up because it's genuinely load-bearing for a lot of the stuff we cover — and I think we owe it a real explanation instead of just gesturing at it every time a paper uses it.

Justy So let's start with the actual problem it's solving, because that's the part that makes the whole thing feel obvious once you see it. You've trained a huge model. It's brilliant. It's also enormous and slow and expensive to run — you can't put it on a phone, you can't run it in real time, it costs a fortune per query.

Cody And the naive fix — just make a smaller model and train it on the same data — doesn't really work. The small model doesn't have the capacity to figure out all the patterns the big one figured out. It's starting from scratch. So the question is: can you somehow transfer what the big model learned into the small one? And the answer is yes, but the HOW is the interesting part. Here's the analogy I keep coming back to.

Justy Okay, I'm with you.

Cody You could just hand the junior person a textbook — that's training from raw data. Or you could have the senior sit next to them, explain their reasoning out loud, say 'this looks like case A, but watch out for this one detail that makes it actually case B.' That running commentary is knowledge distillation. The junior learns the expert's judgment, not just the right answers.

Justy That's the frame I want people to hold onto. The big model is the senior expert. The small model is the new hire. And the whole trick is in what exactly gets passed between them — because it's not just the final answer.

Cody Exactly. So — what does the big model actually produce that the small model is learning from? When a model classifies something — say, is this image a cat, a dog, or a bird — it doesn't just spit out 'cat.' It produces a probability for every possible answer. Maybe ninety-five percent cat, four percent dog, one percent bird. Those are called soft targets, or soft outputs.

Justy As opposed to hard labels, which are just — 'cat. done.'

Cody Right. A hard label throws away all that nuance. But the soft target tells you something the hard label doesn't: the teacher almost said dog. That four percent is information. It says 'this image has some dog-like quality to it, even though it's a cat.' A student learning from that is learning the teacher's sense of similarity and ambiguity, not just the final call.

Justy That's the thing that clicked for me when I first got this. The near-misses are the lesson. The teacher's hesitation is data.

Cody Yeah. And there's one more ingredient — temperature scaling. The teacher's outputs get run through a temperature parameter before being handed to the student. Higher temperature flattens the distribution — makes the probabilities more spread out, more informative. Lower temperature sharpens it, so it looks more like a hard label. If the teacher is ninety-nine point nine percent sure, there's almost nothing in the other answers — the student just sees 'cat' again.

Justy Okay. So mechanically — what actually happens during training?

Cody You have your big teacher model, already trained. You run it on a dataset and collect those soft outputs for every example. Then you train the student model — smaller, fewer parameters — to produce outputs that match the teacher's soft distributions. The loss function penalizes the student whenever its distribution diverges from the teacher's.

Justy Let's make this real with some actual systems. DeepSeek-R1 is probably the most striking recent example — the distilled versions of that model hit ninety-four point five on MATH-500. Which is wild, because that's a distilled model outperforming models that were trained directly on the same task.

Cody Yeah. And there's cross-modal distillation too — the weird one where a vision model teaches a text model, or vice versa. The idea is that sometimes a model trained on one type of data has learned representations that are useful for a different type. VideoChat3 was an example in that space — video understanding, trying to get a smaller model to handle multi-modal reasoning efficiently by leaning on a stronger teacher.

Justy Okay so now I want to push on the limits. Where does it break?

Cody Capacity gap is the big one. The student has to be large enough to actually absorb what the teacher is teaching. If you try to distill a massive frontier model into something tiny, the student just doesn't have the parameters to represent what it's being shown. You get some improvement over training from scratch, but you're leaving a lot on the table. Edge cases are another one — the teacher is good at edge cases because it has a ton of capacity devoted to them.

Justy Which matters A LOT if you're in healthcare or autonomous vehicles — the weird cases are exactly the ones you care most about.

Cody Exactly. And then there's the upfront cost — you have to run the teacher on your entire training dataset to generate those soft targets. If the teacher is a huge model and the dataset is large, that's expensive. The economics only make sense if the student gets deployed a lot. One-time expensive teacher run, then you serve the cheap student forever. The math works out well for production systems. Less obvious for research or one-off use cases.

Justy Also — fine-tuning is adjacent here and I want to make sure people don't conflate them. Fine-tuning takes a pre-trained model and trains it further on new data to adapt its behavior. Distillation is specifically about transferring one model's knowledge into a different, usually smaller, model. Different goals, different mechanism. And they can be combined — but they're not the same thing, and I've seen people use them interchangeably in the wild.

Cody Which… is wrong.

Justy Gently wrong.

Cody Sure, gently. And to answer where this stands right now — still completely live. This is not a historical technique. It's the standard approach for getting capable models onto edge devices, phones, real-time inference systems — anywhere you need speed and low power without tanking accuracy. The DeepSeek-R1 story is recent, DART-SD is recent, Seed is recent. It's actively being used and actively being pushed further.

Justy And from a product standpoint — this is the thing that makes a lot of on-device AI actually possible. Every time someone tells you there's a capable model running on your phone, distillation is usually somewhere in the story. Quantization handles the precision side, making the numbers smaller. Distillation handles the knowledge side, making the model smarter than its size suggests. They're often used together.

Cody Which is a real reason teams reach for it beyond just the compute savings. The energy angle too — a smaller model running in production uses less power. At scale that's not nothing.

Justy Okay. Let me try to land this. The ONE thing I want someone to walk away with — if you forget everything else — is this: the student isn't learning from raw data. It's learning from the teacher's judgment. The soft targets carry the teacher's sense of what's similar, what's ambiguous, where the close calls are. That's what makes the student smarter than its size.

Cody The near-misses are the lesson. You said that earlier and I think that's actually the best one-line version of this. The teacher's hesitation is the signal. Hard labels throw that away. Soft targets keep it.

Justy Cody, I cannot believe it took us nine hundred and twenty episodes to finally sit down and just… explain this one properly. Next time we punt on a concept, someone remind us of this moment.