Topic
KV Cache
12 episodes
-
Introducing Cursor Router · Cursor
Cursor Router is Cursor's new Teams and Enterprise model-routing layer, using a classifier trained on more than six hundred thousand live requests to select models by task, context, complexity, and domain. Jessica sees a clean adoption story for teams stuck paying frontier rates for routine coding work; Cathy likes the production-oriented evaluation and cache-aware accounting, while keeping an eye on how much trust enterprises place in Cursor's routing judgment.
-
Overview: Context Window Management
We finally slow down and explain Context Window Management from the ground up, because we keep hand-waving it whenever agents, memory, cost, and long tasks come up. The whole thing is the fixed-desk problem: what stays on the desk, what gets compressed, and what falls off.
-
Kimi K3 Kimi API Platform
Two friends unpack the Kimi K3 API docs, debating its 1M‑token claim, hybrid attention, and tool dynamics, and weigh who should pay the price for the hype.
-
Model Behavior: Week of July 13, 2026
We read this week as the moment the race got less obsessed with tallest-model bragging and more obsessed with who gives builders the best menu. The funny part is that the open-weight crowd is making the incumbents act practical faster than they probably wanted.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Long Horizon Terminal Bench: Testing the Limits of Agents on Long Horizon Terminal Tasks with Dense Reward Based Grading
Laura and Harper dig into Long-Horizon-Terminal-Bench, a new benchmark exposing the gap between short-task agent demos and real multi-hour workflows. They break down the dense-reward grading system, the staggering token costs (9.9M per task), and why current models are failing at sustained execution despite high step-level competence.
-
Overview: KV Cache
We finally sit down with KV cache and make the whole thing click: what it stores, why generation gets faster, and why memory is the bill you still have to pay. We keep coming back to the same working-memory picture so the mechanics stay grounded instead of turning into acronym soup.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
OpenAI Releases GPT 5.6 (Sol, Terra, Luna): A Three Tier Model Family With Programmatic Tool Calling in the Responses API
OpenAI's GPT-5.6 family — Sol, Terra, and Luna — introduces three permanent capability tiers with distinct cost profiles and a new cache billing model, plus a multi-agent Ultra mode that runs four agents in parallel by default.
-
Hot French startup ZML releases free product to speed inference across lots of AI chips | TechCrunch
Laura and Harper dig into ZML's new free inference server and the bigger claim underneath it: that the real leverage now is software that decouples models from chip vendors. Harper likes the direction but doubts the article proves the hard part, while Laura thinks the product story is strong even if the benchmarks are still missing.
-
Viability of local models for coding
Birgitta Böckeler tests local LLM viability for coding after a year away from the space. She maps a complex web of factors—RAM, model size, quantization, tool calling, context windows, reasoning modes—that determine whether small models actually work for agentic coding on consumer hardware (M3 Max / M5 Pro). Her core finding: locals are runneable and faster than a year ago, but tool calling is still shaky, reasoning can backfire, and quality is hit-or-miss. She's not claiming local models are ready to replace cloud APIs; she's charting what actually works and what doesn't on real machines.
-
nvidia/Nemotron TwoTower 30B A3B Base BF16 · Hugging Face
Justy and Cody dig into NVIDIA’s Nemotron-TwoTower-30B-A3B-Base-BF16 and whether block-wise diffusion decoding is a real systems win or just a benchmark-shaped detour. Cody is skeptical about the headline throughput claim and the way the model compares itself to a single autoregressive baseline, while Justy focuses on who actually benefits from faster generation without a big quality drop. They land on cautious interest: interesting infrastructure idea, but not a universal replacement for standard decoding.