Topic
Classifier
7 episodes
-
Introducing Cursor Router · Cursor
Cursor Router is Cursor's new Teams and Enterprise model-routing layer, using a classifier trained on more than six hundred thousand live requests to select models by task, context, complexity, and domain. Jessica sees a clean adoption story for teams stuck paying frontier rates for routine coding work; Cathy likes the production-oriented evaluation and cache-aware accounting, while keeping an eye on how much trust enterprises place in Cursor's routing judgment.
-
Overview: Calibration
We finally slow down and make calibration click: what it means for a model’s confidence to match reality, how you measure that, and why it matters when you actually want to trust the thing. We keep it grounded in the systems we’ve been circling for ages, because calibration is everywhere once you start looking.
-
Overview: Model Routing
We finally pin down model routing, because we throw the term around all the time and somehow never actually define it. We walk through how a router sends each request to the model most likely to handle it well, and why that can save cost, latency, and a lot of dumb overgeneralization.
-
Overview: Classifier
We finally slow down and make classifier click from the ground up: what it is, how it learns, and why the boring details like labels, loss, and held-out tests matter. We keep it in first-person and keep it practical, because that’s the whole point of calling this a Classifier episode.
-
Overview: Conditional Probability
We keep running into conditional probability anywhere we try to reason from partial evidence, so we finally sat down and made it the whole point. We’re breaking down P(A|B), why the denominator matters, and why this little idea quietly sits under a ton of AI behavior.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Redeploying Claude Fable 5
Anthropic lifts export controls on Fable 5 after addressing an Amazon-reported jailbreak with a new classifier that blocks the bypass in over 99% of cases. The episode unpacks the technical move, the product impact, and whether the safeguard trade-off (more false positives) changes anything for users.