Overview: Error Accumulation in Generation

Laura Okay so this one keeps coming up — we name-drop it, we gesture at it, and then we move on. And honestly a bunch of people have said just… slow down and explain it. So today we actually do.

Harper Yeah. And it fits naturally inside the autoregressive generation territory — we did a whole Overview on that back in episode six thirty — but error accumulation specifically is the piece we've never sat with long enough. Today's the day.

Laura Alright, so set it up. Give me the mental picture before any of the terminology.

Harper Okay. Imagine you're writing on a whiteboard, but there's a rule: you can never erase. Every word you write is permanent, and every new word you write, you're looking at all the previous ones to decide what comes next. That's basically what a language model does. It generates text one chunk at a time — one token at a time — and each new token is predicted based on everything already on the whiteboard.

Laura Right.

Harper So the moment something wrong goes on that whiteboard — a small factual slip, a slightly off detail — it's there forever. And the model's next prediction looks at that wrong thing and treats it as ground truth. It doesn't know it made a mistake. It just… keeps writing from where it is.

Laura So the error doesn't just sit there. It actively steers what comes next.

Harper Exactly. That's the core of it. The mistake becomes context, and context shapes predictions. So you get this compounding effect where one slip early on pulls the whole trajectory a little sideways, and then each step after that is navigating from that shifted position.

Laura Okay and this is why we keep calling it accumulation specifically, not just… errors.

Harper Right, because a single wrong token would be fine in isolation. The problem is the chain. A quick bit of math that I find genuinely clarifying: if each token has a ninety-nine percent chance of being correct — which sounds really good — a hundred-token output has roughly a thirty-seven percent chance of being entirely clean. Zero errors.

Laura Wait — thirty-seven percent? That's… not reassuring.

Harper Yeah. And a hundred tokens is not a long output. That's maybe two or three sentences. For something like a five-hundred-word essay, or a function with logic branches, or a multi-step reasoning chain — you're talking about far more tokens, and the probability of getting through completely clean drops hard. The math isn't the scary part though. The scary part is what the model does with the error once it's there.

Laura Okay so walk me through the mechanism. How does the model actually get into this situation?

Harper So the short version of autoregressive generation: the model predicts the next token given all the previous tokens, one at a time, and the growing sequence is what carries the work forward. That's the basic loop. Now here's the trap. During training, the model is always shown the correct previous tokens — real text written by humans. But during generation, it's predicting from its OWN previous outputs. Which may already be wrong.

Laura So it trained in one world and it's operating in a slightly different one.

Harper That's exactly it. The technical name for this gap is exposure bias, or sometimes distribution shift. The model was never fully trained for the regime it actually runs in — where the context it's reading was written by itself, mistakes included. A small error shifts the probability distribution over what comes next, and the model just… follows that shifted distribution. It doesn't have a mechanism to step back and notice something went wrong.

Laura And this is why you get those outputs that sound totally confident and are quietly completely wrong.

Harper Yes. Hallucination is largely this. The model isn't lying — it's following the statistical path laid by its own earlier tokens. Once it's written a plausible-sounding wrong claim, the next tokens are predicted in a world where that claim is true. So you get a cascade of related false claims, all internally consistent with each other, all wrong.

Laura Okay, that's good. That's the part I don't think people have internalized — it's not randomness, it's coherent wrongness. The errors cluster.

Harper They cluster, yeah. And they reinforce. A factual mistake early on actively pulls the probability distribution toward related false claims. So paragraph four is building on paragraph two's mistake, and by the end you have something that hangs together narratively but is quietly off the rails. And that's a particularly brutal environment in long-horizon agent tasks — there's no natural reset point. The agent keeps building on its own prior outputs across many steps.

Laura So which tasks are most exposed to this?

Harper Code generation, multi-step reasoning, narrative writing — anything where early decisions constrain everything downstream. The mitigations are all about breaking the chain. Task decomposition so each agent gets a shorter window with handoff points where something can check the work before it becomes context for the next step. Intermediate verification gates — synchronous checks that run between an output and any downstream action.

Laura That one is still so good. Or so bad. Both.

Harper Both. And then constrained decoding — restricting the output space so it can only generate tokens that conform to some schema — limits how far off the rails any single step can go. And reranking: generate multiple candidates and filter before committing. We did a whole Overview on reranking last episode, nine seventy-two. Key point here is it operates on finished candidates — it's a filter, not a fix. None of these fully solve the exposure bias problem.

Laura And scaling? Like — bigger model, better per-token predictions, does that eventually make this go away?

Harper It helps and it doesn't. A bigger model does make better individual token predictions — the ninety-nine percent in my earlier example goes up. But the compounding math still applies. You're starting from a higher baseline, which means you survive more steps before an error is likely, but you're not immune. The structure of the problem is architectural — it's in how autoregressive generation works — not a capability gap that scale closes.

Laura So you get a better whiteboard writer, but they still can't erase.

Harper They still can't erase. And this is actually what I find useful about thinking of it as an architectural property rather than a model quality problem — it reframes what you're looking for. You're not asking 'is this model smart enough' in the abstract.

Laura Okay. Where does this sit right now, in terms of how the field is actually dealing with it?

Harper It's still the central unsolved problem in long-horizon generation. Not solved, not going away — the field is building AROUND it. Verification gates, shorter windows, agentic retry loops with structured traces — those are the current live responses. The exposure bias problem at the training level is still open.

Laura And this is why, when we talk about multi-agent coordination being load-bearing, it's not just an architecture preference. It's a direct response to this problem.

Harper Directly. Break the chain, add a gate, verify before you propagate. That's the practical answer in two thousand and twenty-six. Not elegant, but it works.

Laura Okay. So if someone's walking away from this with one thing — the whiteboard you can never erase. Every token the model writes becomes the ground it stands on for the next one. Get something wrong and you're not just wrong in one spot, you're navigating from a wrong position for everything that follows.

Harper That's it. And the corollary: when you see a model confidently generating something that's coherently, structurally wrong — not random noise, but wrong in a way that hangs together — that's usually this. Not deception, not randomness. Just the model following its own earlier tokens into a corner it can't see out of.

Laura Harper, I genuinely think this is one of those things that once you see it you can't unsee it. Every long output I look at now I'm thinking about where the whiteboard went wrong.

Harper Yeah. That's the curse of actually understanding the mechanism. You start reading outputs differently.

Laura Alright. Episode nine seventy-three of whatever this show is. Go read something long and see how far it gets before it starts navigating from somewhere wrong.