Overview: Autoregressive Generation

Justy Okay, we have to stop pretending we’ve explained this enough. Autoregressive generation is one of those things we keep saying like everyone already has it in their head, and I’m pretty sure that’s rude.

Cody Yeah, fair. It’s one of those terms that sounds heavier than it is. The basic picture is just: the model writes one token, looks at what it wrote, then writes the next token from that new context.

Justy So it’s basically a very anxious writer. It keeps rereading the last sentence before it dares to add the next word.

Cody Exactly. And that’s why people call it autoregressive generation, or autoregressive decoding. Same thing, just different emphasis on whether you’re talking about the process or the output.

Justy Right, and I think that naming trips people up because it sounds like some separate fancy algorithm. It’s really just the loop. The model makes a guess, appends it, then uses the longer text as the new starting point.

Cody Mm-hm. And that loop is doing something important: it turns generation into a sequence of small decisions instead of one giant impossible one. If a model had to output an entire essay in a single shot, that would be a nightmare.

Justy Because the output length isn’t fixed.

Cody Right. A neural network is usually a fixed-input, fixed-output machine in the abstract. A neural network is just a stack of learned math that maps inputs to outputs. Autoregressive generation is the trick that makes that fixed machinery behave like a variable-length writer.

Justy Okay, that’s already cleaner than how we usually say it. And just to keep us honest, a language model is basically a neural network trained to work with text, right?

Cody Yeah. More specifically, it learns to assign probabilities to text continuations. And probability here just means a model’s guess about what’s likely next, given what came before. Conditional probability means the chance of something happening given some context already exists.

Justy So when the model sees "the cat sat on the," it’s not magically knowing English. It’s estimating what token is most likely after that prefix.

Cody Exactly. And token is the chunk the model actually handles. Sometimes that’s a word, sometimes part of a word, sometimes punctuation. We did a whole episode on tokenization, episode six hundred one, but the only thing you need right now is that the model isn’t literally thinking in human words.

Justy Right, it’s thinking in those little pieces. Which is already kind of funny, because the whole elegant story is built on a very unromantic little loop.

Cody Very unromantic, yeah. But here’s the key mechanism: during training, the model gets lots of examples where the answer is already there. It sees a prefix and learns to predict the next token from that prefix.

Justy So training is like giving it the worksheet with the answers hidden one step to the right.

Cody Yeah, that’s a decent way to put it. The model sees the full sequence during training, but it only gets punished or rewarded for how well it predicts each next token. That’s how it learns the distribution over continuations.

Justy Wait, but then at runtime it doesn’t get the answer sheet, right? It has to use its own last guess.

Cody Exactly. That gap matters a lot. Training lets it look at the real future tokens while learning, but inference only has its own past outputs. So at generation time, it predicts one token, appends it, then predicts again from the updated text.

Justy Okay, and that’s the part people hand-wave over when they say, like, "the model just writes." It’s not writing all at once. It’s repeatedly choosing the next thing under uncertainty.

Cody Right, and because it only ever conditions on the left side, it can’t cheat by looking ahead. That left-to-right rule is the whole architecture of the behavior.

Justy How does that connect to sequence modeling? That phrase always sounds like a grad-school warning label.

Cody Sequence modeling just means the task is about ordered things, where order matters. Text is one sequence, but so are time steps, code tokens, music notes, all of that. Autoregressive generation is one way to model a sequence by predicting the next element from the previous ones.

Justy Okay, so sequence modeling is the broad category, and autoregressive generation is the left-to-right version of it.

Cody Yep. And the reason it works so well is that language itself often behaves like a sequence where earlier context shapes later choices. That doesn’t mean language is simple. It just means this one-step setup matches the problem surprisingly well.

Justy That’s the part I think people miss. It sounds almost too basic to be the thing under the hood of these giant systems.

Cody Mm-hm. But the simplicity is the point. The model doesn’t need to invent a whole sentence structure in one go. It only needs to get the next token right, over and over, while the growing context carries the rest.

Justy And because the context grows, each new guess gets more information to work with. That’s the whole self-feeding bit.

Cody Exactly. Auto means self. Regressive means looking back at prior values. In plain language, it’s self-conditioned generation. The model uses its own earlier output as part of the input for the next step.

Justy Mm, that lands. And I want to pause on the training part for a second because that seems like the sneaky bit. The model learns from full examples, but it has to live with partial examples when it’s actually generating.

Cody Right, and that mismatch is one of the main catches. During training, the model gets the true previous tokens. At inference, it gets its own previous guesses, which may be wrong. That’s why small mistakes can snowball.

Justy So if it goes a little off the rails early, the rest of the generation is now based on a slightly bad history.

Cody Exactly. That’s error accumulation. It’s not that the model suddenly forgets how to talk. It’s that each next decision is conditioned on whatever it already produced, good or bad.

Justy Okay, that explains why long outputs can get weird. I always thought people were just being dramatic when they complained about drift.

Cody No, it’s real. And it’s also why decoding strategy matters. If you always take the single highest-probability next token, that’s greedy decoding. It’s deterministic, but it can get stuck repeating itself or taking the blandest possible path.

Justy And if you sample instead, you get more variety, but you can also get nonsense.

Cody Yep. Sampling means you pick from the probability distribution instead of always taking the top choice. Then there are knobs like top-k, where you only consider the top few candidates, and top-p, where you keep the smallest set of tokens whose combined probability reaches a threshold. Temperature adjusts how flat or sharp the distribution feels.

Justy That is such an Exploring Next sentence. It sounds like we’re tuning a spaceship, and really we’re just deciding how adventurous the next word should be.

Cody Pretty much. But those knobs matter because the core loop is simple and unforgiving. If you want more creativity, you pay for it with more risk. If you want more determinism, you can get stiffness or repetition.

Justy Can we anchor this in actual systems? Because this is where it stops being abstract for me.

Cody Yeah. Think about the GPT-5.6 thing we looked at, or the Claude model selection flow in Claude Code. Those systems are all using the same basic left-to-right generation behavior. You prompt them, and they keep extending the sequence one token at a time based on what’s already there.

Justy And the Nemotron Two Tower model we looked at too, right? Different architecture in the broader sense, but the text generation part still has to unfold token by token.

Cody Right. Even when the surrounding system is doing something fancier, the text output itself is still usually autoregressive. That’s why the interface feels interactive. It’s not precomputed all at once; it’s being assembled in order.

Justy Mm-hm.

Cody And that’s also why latency exists in the first place. If you need one thousand tokens, you’re not doing one forward pass. You’re doing something like one pass per token, which is why generation can feel slower than people expect.

Justy That’s the part product people feel immediately. A model can sound amazing in a demo and then you hit the real wait time and suddenly the user experience is just... a spinner with opinions.

Cody Exactly. And that’s why production teams care so much about decoding efficiency. The model may be capable of good text, but the user only experiences it if the loop is fast enough to feel responsive.

Justy Okay, let me make sure I’ve got the mental model straight. The model is basically a very expensive autocomplete, but autocomplete that keeps consulting its own previous output instead of a human typing.

Cody Yeah, that’s close enough for a first pass. I’d just be careful not to make it sound trivial. The model isn’t just finishing words. It’s using a learned distribution over possible continuations, and that distribution carries a lot of structure.

Cody That is such an Exploring Next take.

Justy I know. But come on, it’s a good one. Alright, I’m done being smug about it for the rest of the day.