Topic
Sequence Modeling
5 episodes
-
Attention Is All You Need
Onyx and Echo dig into the original Transformer paper — what problem it solved, how scaled dot-product attention and multi-head attention actually work, why dropping recurrence was the real lever, and whether the architectural choices have held up across nearly a decade of subsequent work.
-
Overview: Fine tuning on Execution Traces
We keep throwing around fine-tuning on execution traces, so we finally define it from the ground up. We’re talking about training a model on the steps, not just the answer, and why that changes what it learns.
-
Overview: Sequence Modeling
We slow down and finally define sequence modeling, the idea underneath next-token prediction, language models, and a surprising amount of modern AI. We keep it grounded in one picture: covering the next word and training a model to guess what belongs there.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
Overview: Autoregressive Generation
We finally slow down and make autoregressive generation click: the whole thing is just a model writing one token, then using what it wrote to choose the next one. We keep the focus on the loop, the trade-offs, and why that one-step-at-a-time setup is still the backbone of modern language models.