Topic
Speculative Decoding
6 episodes
-
NVIDIA Brings Simplified Local AI Support To NVIDIA GPUs Carrying 24+ GB VRAM While vLLM & & llama.cpp Optimizations Boost Compute By Up To 1
NVIDIA ships kernel optimizations for local AI inference on RTX and DGX platforms, delivering up to 1.9x performance gains through vLLM and llama.cpp, paired with one-click agent setup (Perplexity Portable Computer, Hermes Agent, OpenClaw) for GPUs with 24+ GB VRAM. The move collapses the friction between local-first and cloud-capable workflows.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Overview: Token Efficiency
We finally sit down and define something we've been throwing around since episode one: token efficiency. What it actually means, how it works on both the input and output side, and why it's quietly one of the most load-bearing ideas in how real AI systems get built and paid for.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.
-
Overview: Decoding Strategy
We finally slow down on decoding strategy, the rule that turns a model's next-token odds into the actual words you see. We use one hallway-and-doors picture to make greedy decoding, sampling, top-k, top-p, beam search, and newer decoding work feel less like magic knobs.
-
Exploring Next Overview: Speculative Decoding
We finally slow down and unpack speculative decoding from the ground up: the draft model, the verify step, and why it can make generation faster without changing the output. We keep it concrete, because that trick sounds like cheating until the mechanism actually clicks.