Topic
Inference Optimization
13 episodes
-
How Much Is a Token
Talon and Wildflower discuss the eroding utility of the 'token as a unit of economic value in AI, sparked by Anthropic's tokenizer changes and the rise of competitive open-weight inference providers.
-
NVIDIA Brings Simplified Local AI Support To NVIDIA GPUs Carrying 24+ GB VRAM While vLLM & & llama.cpp Optimizations Boost Compute By Up To 1
NVIDIA ships kernel optimizations for local AI inference on RTX and DGX platforms, delivering up to 1.9x performance gains through vLLM and llama.cpp, paired with one-click agent setup (Perplexity Portable Computer, Hermes Agent, OpenClaw) for GPUs with 24+ GB VRAM. The move collapses the friction between local-first and cloud-capable workflows.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
Accelerating GPT 5.6 Sol Ultrafast with OpenAI
Vince and Ava dig into Cerebras powering OpenAI's GPT-5.6 Sol Ultrafast mode. Ava leads skeptical on the benchmark framing and the article's leap from token speed to real-world inevitability, while Vince argues the product point is simpler: if frontier-quality answers arrive fast enough to stay on the critical path, new workflows open up. They land on a calibrated take that the mechanism is plausible and strategically important, but the evidence shown is narrower than the headline and pricing plus access will decide who actually cares.
-
I ditched Ollama as my default runtime, and the replacement starts models in a fraction of the time
Laura and Harper dig into an XDA piece arguing that BaseRT is a better default local runtime than Ollama on Apple Silicon because it reaches first token much faster, especially on long prompts. They agree the claim is real but narrow: BaseRT looks strong for Mac-first, performance-sensitive workflows, while Ollama still wins on breadth, compatibility, and convenience.
-
Overview: Inference Optimization
We finally give inference optimization its own episode — the idea that's quietly under half the stories we cover. We walk through what it actually means to make a trained model run faster and cheaper, from caching to quantization to batching, and why it matters more than almost anything else once a model ships.
-
1Aggregate results comparing DFM Mimir 1B against the HRM Text 1B, Qwen 3.5 2B and Gemma 4 E2B, displaying highly competitive performance across 20 benchmarks.
Pippa and Tyler examine DFM Mimir v1, a one-billion-parameter HRM trained from scratch with permissible post-training data, and ask whether its strong benchmark results make it genuinely useful for low-resource language work.
-
Overview: Quantization
We finally sit down with quantization and make it concrete: why shrinking a model’s numbers can speed inference up without wrecking the thing. We walk through the trade-offs, the common formats, and where quantization still matters in practice.
-
Cutting RAG inference costs 6x starts with deciding what never reaches the LLM
Masonry and Eyre dig into a cascade architecture for RAG in regulated settings: deterministic rules first, retrieval second, LLM only for the genuinely ambiguous residue. They connect it to their long-running infrastructure-over-capability thesis and debate whether the asymmetric prompt framing is a real engineering move or just prompt engineering with a budget.
-
Compute Forecast — AI 2027
Vince and Ava dig into Romeo Dean’s 2025 “Compute Forecast — AI 2027” and tease apart which parts of the compute story feel grounded (10x global AI-relevant compute, concentration in a few labs) versus which jumps (a million “superintelligent” research agents at 50x human speed, three and a half percent of U.S. power) feel more like scenario fiction. They map the technical assumptions behind H100-equivalent growth, utilization, and chip efficiency to actual product and research decisions, and argue that the real takeaway isn’t “AGI by 2027” but “whoever owns the scheduler and the power bill sets the rules.”
-
Model Behavior: Week of July 27, 2026
We're watching the frontier splinter into specialized tiers — raw capability matters less than matching the right model to the task's actual constraints. Opus 5 proved it Friday at half the cost of the frontier, and this week's open-weight and Flash-tier releases confirm the pattern: the market isn't consolidating around one best model, it's fragmenting into capability-per-dollar buckets.
-
Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.
-
A Scorecard for the AI Age
OpenAI’s scorecard argues AI value must be measured in useful work per dollar, not just token cost. Cooper sees a practical product story; Miles pokes at the metrics and pushes for mechanistic honesty. The two hash out whether the framework holds up and what it changes day-to-day.