Topic
Autoregressive Generation
22 episodes
-
Overview: Decoding Strategy
We finally slow down on decoding strategy, the rule that turns a model's next-token odds into the actual words you see. We use one hallway-and-doors picture to make greedy decoding, sampling, top-k, top-p, beam search, and newer decoding work feel less like magic knobs.
-
Overview: Sampling and Temperature
We’re finally slowing down and explaining sampling and temperature from the ground up, because we keep circling back to it and it’s one of those knobs that sounds simple until it isn’t. We talk through why the same prompt can produce different answers, how temperature changes the shape of the next-token choices, and where the trade-offs actually land.
-
GitHub FareedKhan Dev/train LLM From scratch: A straightforward method for training your LLM, from downloading data to generating text.
A single-GPU end-to-end LLM training guide lands on GitHub—hand-written PyTorch, pretrain to chat in one repo, plus full RLHF. Vince is giddy; Ava wants to know which corner of the GPU shelf this actually runs on. Build Next shows the exact CLI to kick it off on a T4.
-
Overview: Fine tuning on Execution Traces
We keep throwing around fine-tuning on execution traces, so we finally define it from the ground up. We’re talking about training a model on the steps, not just the answer, and why that changes what it learns.
-
Overview: Sequence Modeling
We slow down and finally define sequence modeling, the idea underneath next-token prediction, language models, and a surprising amount of modern AI. We keep it grounded in one picture: covering the next word and training a model to guess what belongs there.
-
Overview: Retry Loops and Error Recovery
We finally define retry loops and error recovery, because we keep tossing the term around like everybody knows exactly what it means. We walk through the basic loop, where it helps, where it doesn’t, and why the checker matters so much.
-
Overview: Active vs Total Parameters
We finally slow down on active vs total parameters, because we keep throwing the phrase around like it explains itself. This is us making the difference click: what a model stores versus what it actually uses when it answers.
-
Alibabas Tongyi Lab Releases Qwen Audio 3 0 TTS a Hosted Text to Speech Model in Flash and Plus Tiers Across 16 Languages
Cooper and Miles examine Alibaba's Qwen-Audio-3.0-TTS, comparing its Flash and Plus tiers, multilingual support, voice controls, architecture, hosted-only trade-offs, pricing, and real production use cases.
-
Exploring Next Overview: Speculative Decoding
We finally slow down and unpack speculative decoding from the ground up: the draft model, the verify step, and why it can make generation faster without changing the output. We keep it concrete, because that trick sounds like cheating until the mechanism actually clicks.
-
Overview: In Context Learning
We finally slow down and explain in-context learning, the thing we keep leaning on whenever prompts, agents, examples, and adaptation come up. We make the core idea concrete: the model is learning from the temporary packet you hand it, without changing itself permanently.
-
Overview: Token Economics
We finally slow down on Token Economics: why tokens are the meter for cost, speed, memory, and product decisions in language models. We keep using the tiny-slip postage analogy until the whole thing clicks, from tokenization to context windows to real API bills.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Overview: Prompt Engineering
We’re finally doing the overdue deep dive on prompt engineering, the weirdly practical skill of getting language models to do the thing you actually meant. We keep coming back to it because the difference between a flimsy prompt and a good one is often the difference between nonsense and a usable product.
-
Overview: Transformer Architecture
We finally sit down and define transformer architecture from the ground up, because we keep throwing the term around like it’s obvious and it really isn’t. We use the attention-as-a-room-of-index-cards picture to make the mechanism click, then connect it to why Transformers became the backbone of modern language models.
-
Overview: KV Cache
We finally sit down with KV cache and make the whole thing click: what it stores, why generation gets faster, and why memory is the bill you still have to pay. We keep coming back to the same working-memory picture so the mechanics stay grounded instead of turning into acronym soup.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
Overview: Autoregressive Generation
We finally slow down and make autoregressive generation click: the whole thing is just a model writing one token, then using what it wrote to choose the next one. We keep the focus on the loop, the trade-offs, and why that one-step-at-a-time setup is still the backbone of modern language models.
-
GPT 5 6
Talon and Wildflower dig into OpenAI’s GPT-5.6 launch and end up treating it less like a pure model release and more like a pricing-and-harness claim wrapped in benchmark flexing. Wildflower’s skeptical read is that the article keeps collapsing model quality, multi-agent orchestration, and product packaging into one victory lap. Talon pushes back that the practical story is real if Sol, Terra, and Luna actually move the cost-performance frontier for coding and knowledge work. They land on a calibrated view: the coding gains look more credible than the broad ‘best collaborator’ language, Terra may be the sleeper product, and ultra is interesting but shouldn’t be mistaken for a single-model breakthrough.
-
Overview: Tool use and function calling
We finally sit down and make tool use and function calling click from the ground up. We keep coming back to the same idea: a model can draft the request, but something outside it has to actually do the thing.
-
Choosing a Claude model and effort level in Claude Code | Claude by Anthropic
Claude Code’s model vs. effort article finally clarifies the levers you actually have: model swaps the frozen weights (capability ceiling), effort tunes the work-loop (files read, steps taken, verification depth). Defaults are tuned per model; override only when you know you want more thoroughness (higher effort) or a higher capability floor (bigger model). Wrong answers split cleanly: context/steering miss → up the model; skipped files/half-done tasks → up the effort.
-
Overview: Retrieval Augmented Generation
We finally slow down and make Retrieval-Augmented Generation click from the ground up: what it is, why it helps, and where it still falls apart. We keep coming back to the same simple picture so the mechanics don’t get lost in the jargon.
-
nvidia/Nemotron TwoTower 30B A3B Base BF16 · Hugging Face
Justy and Cody dig into NVIDIA’s Nemotron-TwoTower-30B-A3B-Base-BF16 and whether block-wise diffusion decoding is a real systems win or just a benchmark-shaped detour. Cody is skeptical about the headline throughput claim and the way the model compares itself to a single autoregressive baseline, while Justy focuses on who actually benefits from faster generation without a big quality drop. They land on cautious interest: interesting infrastructure idea, but not a universal replacement for standard decoding.