Overview: Sampling and Temperature

Cooper This keeps coming up, and I swear we keep skating past it like everybody already knows what it means. So, okay, we finally just do the thing, episode seven seventy-four and all that.

Miles Yeah. And honestly, this one is sneaky because people hear "temperature" and think it’s some mystical creativity setting. It’s not mystical. It’s just the knob that changes how willing the model is to take the less obvious next word.

Cooper I had one of those weeks where I kept seeing it in tool settings and thinking, yep, sure, everybody nods and nobody explains it. Which is very Exploring Next of us, apparently.

Miles Exactly. The clean mental model is: the model is always choosing the next token from a bunch of possibilities, and sampling is the way it picks instead of always grabbing the top one. Temperature just changes how spread out that choice is.

Cooper So the same prompt can give you different answers because it’s not reading off a script. It’s making a choice at each step, and that choice is probabilistic.

Miles Right. We did a whole episode on conditional probability, episode six seventy-one, but the short version is just this: the model assigns different chances to different next tokens. Then it uses those chances to decide what comes next.

Cooper Okay, that part matters. Because if you think it’s just "the answer" sitting there, then temperature sounds like some weird seasoning sprinkle, and it’s not.

Miles No. And the other prerequisite here is autoregressive generation, episode six thirty. That just means the model makes output one token at a time, each new token depending on the ones before it. So every step is a fresh next-token decision.

Cooper Mm-hm.

Miles If you want the plain-language version of decoding strategy, it’s just the rule for how the model turns those probabilities into an actual token. Greedy decoding always takes the most likely one. Sampling says, more or less, let’s draw from the distribution instead.

Cooper And that’s the first big intuition, right? Greedy is the model being a tiny little hall monitor. Sampling is the model saying, fine, I’ll consider the second-best answer sometimes.

Miles Yeah, and that difference is huge in practice. If you always pick the most likely token, you often get text that’s safe but weirdly repetitive. It can get stuck in grooves because it keeps taking the locally best move over and over.

Cooper That’s the part people underestimate. The top choice at each step is not always the best path across many steps. It can look fine token by token and still sound like a robot that forgot how sentences work.

Miles Exactly. So sampling exists to keep the model from collapsing into that one-track behavior. It still stays inside the model’s learned probabilities, but it doesn’t pretend there’s only one possible next word.

Cooper Right, so let’s do the mechanics without turning this into a math sermon. What is the model actually outputting before it samples?

Miles It outputs logits, which are just raw scores for each possible next token. They are not probabilities yet. You feed those through softmax, which turns the scores into probabilities that add up to one.

Cooper Okay, good, because I knew softmax was lurking in here like a raccoon in the cabinets.

Miles Ha! Yeah, exactly. And temperature gets applied before softmax. You divide the logits by the temperature value, then softmax turns the adjusted scores into the final distribution.

Cooper So if temperature is one, nothing changes.

Miles Right. One is the neutral point. Below one, the distribution sharpens. The high-scoring tokens pull farther ahead, and the low-scoring ones fade out more. Above one, the distribution flattens, so less likely tokens become relatively more plausible.

Cooper So low temperature is basically the model saying, I’m pretty sure this is the word. High temperature is, I’m not sure, let’s keep the door open.

Miles That’s the right shape of it. And then after that adjustment, the model samples from the distribution. Weighted dice, basically. A token with forty percent probability gets picked more often than one with twenty five percent, but the lesser one still has a shot.

Cooper I like that because it makes the difference between randomness and chaos clearer. It’s not just random. It’s random inside a preference structure.

Miles Exactly. That’s the useful part. People sometimes hear "sampling" and think it means the model is wandering around blindly. It isn’t. It’s still constrained by the probabilities it learned from training.

Cooper Okay, but why does flattening or sharpening those probabilities help? Like, why not just leave it alone and use the model’s natural distribution?

Miles Exactly. And this is where your through-line metaphor actually holds up. If the model is a receptionist sorting incoming requests, temperature changes how decisive the receptionist is. Low temperature is, I know the usual answer. High temperature is, let me check a few more doors before I decide.

Cooper That’s good. And the receptionist is still following the building’s rules, not making stuff up out of nowhere. It’s just how wide they cast the net.

Miles Yes. And since you brought up the show being weirdly obsessed with this, let’s connect it to the things we’ve already looked at. In that concurrent image understanding and generation piece, the whole point was that the system had to keep outputting plausible next steps while staying coherent across modalities. Sampling is part of why that kind of system can stay flexible instead of locking into one canned path.

Cooper Mm-hm.

Miles And when we went through the self-correcting coupled Markov jump processes thing, the interesting bit was that the system was juggling transitions, not just picking one final answer. That’s the same intuition here: the path matters, so the way you choose each next move matters.

Cooper I like that because it keeps this from feeling like a toy setting. This is a control surface on how the model walks through its own output.

Miles Yeah, and in real systems, that control surface is often exposed because different users want different trade-offs. A code assistant might default low. A brainstorming tool might default higher. A chat product may let the user nudge it, because one person wants crisp and another wants playful.

Cooper Oh interesting.

Miles The catch is that higher temperature buys variety by spending some reliability. You get more surprising completions, but also more errors, more drift, more weird turns of phrase. That trade-off is not a bug. It’s the whole bargain.

Cooper So if somebody says, can we just turn temperature up and get better ideas, the answer is, maybe better ideas, and also more junk.

Miles Exactly. And that’s why the shape of the task matters. If you’re generating a list of possible names, junk is tolerable. If you’re generating code, junk is expensive. If you’re generating a careful answer, junk is embarrassing.

Cooper Ha! yeah, okay.

Miles One more practical piece: reproducibility. If you set a fixed random seed, then the same prompt at the same temperature can produce the same output again. Without a seed, you can still get different samples from the same setup because the draw is stochastic.

Cooper Right, so the randomness is controlled, but it’s still randomness. That’s actually a nice mental model for why debugging this stuff can be annoying.

Miles It is. And it’s also why people sometimes think a model is being inconsistent when really they changed the decoding setup. Same prompt, different temperature, different sample. That’s expected behavior, not a betrayal.

Cooper Which, honestly, is very on brand for models. They’re not being dramatic, they’re just following the math you handed them.

Miles Exactly. And where this stands now is simple: sampling and temperature are still core, everyday decoding controls. Nothing has replaced them in general use. They sit underneath most chat, writing, and code generation setups as a basic way to tune behavior.

Cooper Yeah, no, that tracks. The field has added fancier sampler variants and all kinds of parameter soup, but the basic idea is still the thing people are actually turning.

Miles Right. Top-p, top-k, repetition penalties, all that stuff can matter, but temperature is still the first knob most people understand. And for a newcomer, that’s the one worth getting right before you chase the rest of the zoo.

Cooper Okay, land it for me like I’m going to forget it again in ten minutes, which is not impossible.

Miles Sampling is the model choosing from a probability distribution instead of always taking the top answer. Temperature changes how sharp or flat that distribution is, which changes how predictable or adventurous the output feels.

Cooper Good. So the whole trick is that the model’s not just thinking harder or softer. It’s being asked to choose differently, and that choice changes the whole vibe.

Miles Exactly.

Cooper There we go. I cannot believe this is how we’re spending a Wednesday, but honestly, episode seven seventy-four finally earned its keep.