Topic
Token Economics
35 episodes
-
How Much Is a Token
Talon and Wildflower discuss the eroding utility of the 'token as a unit of economic value in AI, sparked by Anthropic's tokenizer changes and the rise of competitive open-weight inference providers.
-
Your LLM Can Return Perfect JSON and Still Be Wrong
A real-world trap in Structured Outputs: enforcing schema validity does not guarantee data truthfulness. When a required field is missing from source text, the model invents a plausible value instead of returning null, producing type-correct but false data. The fix requires three layers: nullable fields to allow absence, evidence fields to show provenance, and post-parse validators to catch nonsense values. The essay walks through a payment-reconciliation pipeline where 2–3% of transactions had fabricated dates, caught only downstream.
-
The efficient frontier of LLM inference
Baseten's 'Efficient Frontier of LLM Inference' argues that inference engineering involves two fundamentally different types of techniques: those that let you trade off between latency and throughput along an existing efficiency curve, and those that push the entire curve outward, creating universal gains. The article maps concrete techniques (batch sizing, parallelism strategies, quantization, kernel optimization, speculative decoding, prefill/decode disaggregation) onto this framework and shows that the frontier is jagged and empirically discovered, not smooth.
-
HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?
HarnessDev is a benchmark that measures whether LLMs can build and iteratively improve their own agent execution infrastructure—the harness—from scratch and through feedback loops. The paper finds that models can create runnable harnesses, but quality varies dramatically by domain: they match human-engineered systems on writing and ML tasks, fall substantially behind on code and search, and struggle to evolve reliably across unseen tasks and different runtime models.
-
Introducing Gemini 3.8 Flash and 3.8 Flash Cyber
Asteria and Draco discuss the launch of Gemini 3.8 Flash and 3.8 Flash Cyber, focusing on the 'work harder' reasoning approach and the specialized cybersecurity capabilities for trusted defenders.
-
Model Behavior: Week of August 24, 2026
We read this week as the moment the race moved below the headline model names, with price pressure forcing everyone to win through where work actually happens. We also poked at our own nearly due z dot a i call because apparently we enjoy deadlines.
-
Stripe Payments Openrouter Singularity
Stripe says January 1, 2026 marked the beginning of a major technological and economic inflection point, using its business data as evidence. The more practical move may be its acquisition of OpenRouter, connecting model routing to the payments and control infrastructure Stripe already owns.
-
Model Behavior: Week of August 17, 2026
We read this week as a shift away from pure scoreboard chasing and toward whoever becomes the place work actually runs. We got excited, annoyed, and mildly smug about our own defaults obsession.
-
Accelerating GPT 5.6 Sol Ultrafast with OpenAI
Vince and Ava dig into Cerebras powering OpenAI's GPT-5.6 Sol Ultrafast mode. Ava leads skeptical on the benchmark framing and the article's leap from token speed to real-world inevitability, while Vince argues the product point is simpler: if frontier-quality answers arrive fast enough to stay on the critical path, new workflows open up. They land on a calibrated take that the mechanism is plausible and strategically important, but the evidence shown is narrower than the headline and pricing plus access will decide who actually cares.
-
Snowflake adds AI model routing to cut costs | VentureBeat
Snowflake's new dynamic routing feature matters less as a cheap-model switcher than as a bid to make governed, auditable routing native to the enterprise data platform where a company already lives.
-
Cutting RAG inference costs 6x starts with deciding what never reaches the LLM
Masonry and Eyre dig into a cascade architecture for RAG in regulated settings: deterministic rules first, retrieval second, LLM only for the genuinely ambiguous residue. They connect it to their long-running infrastructure-over-capability thesis and debate whether the asymmetric prompt framing is a real engineering move or just prompt engineering with a budget.
-
2085024744387092973
Jessica and Cathy dig into a post that tries to boil Claude work down to agents, loops, and graphs. Cathy likes the structure but pushes back on the article’s tendency to make every layer sound universally useful, while Jessica argues the real win is that it gives people a practical ladder instead of vague agent hype.
-
How many of your agent's calls actually need a frontier model?
Jessica and Cathy debate the real payoff of model routing for AI agents: is it worth the complexity, or is a single strong open model usually enough? Cathy questions whether the claimed cost savings from routing justify the judge model's price and complexity, given only a modest accuracy gain. Jessica pushes the product case for routing when you can't risk wrong answers on hard tasks, insisting most teams can't rely on just a cheap model if real mistakes are expensive. Together, they find the real value is in knowing—by measurement—when routing pays, and agree that for some teams, the bar is higher than the hype suggests.
-
Pi, Minimal and Performant | EARENDIL
Tyler and Pippa dig into Pi, the minimal coding harness from Earendil, and the Databricks benchmark that claims simple harnesses beat bloated ones on real-world tasks. Tyler's skeptical about how much of this is genuine insight versus flattering a tool that happens to match a particular workload; Pippa thinks the cost-per-task framing is the actual product story and the Shopify autoresearch numbers are hard to dismiss.
-
Coding Agents Don’t Need Bigger Context Windows — They Need a Context Compiler | Towards Data Science
Fern and Lintel dig into the article's claim that coding agents need smarter context pruning, not bigger context windows. They break down the mechanics of context compilation, how it slashes prompt bloat, and whether this shift changes the real product story for code AI. Along the way, they tease each other's optimism and skepticism, call back to their running jokes about infrastructure, and reflect on the broader context engineering debate.
-
Stop graphing everything: When GraphRAG actually beats vector RAG
Pippa and Tyler dig into a fresh GraphRAG piece that argues graphs only beat vector RAG when the question needs multi-hop reasoning, global sensemaking, or summary across an interconnected corpus. They walk through the mechanism, the benchmark evidence, the cost caveat, and the practical hybrid answer without pretending graphing everything is the move.
-
Overview: Token Efficiency
We finally sit down and define something we've been throwing around since episode one: token efficiency. What it actually means, how it works on both the input and output side, and why it's quietly one of the most load-bearing ideas in how real AI systems get built and paid for.
-
Qwen 3.7 Flash review: a $0.03 vision model with a catch
Cathy leads a skeptical take on Qwen 3.7 Flash — the $0.03 vision model from Alibaba that looks like a pricing breakthrough until you read the fine print. The tiered pricing structure, near-zero independent benchmarks, a ninety-second P99 latency tail, and an eight-point-nine percent tool error rate make it a much narrower product than the headline suggests. Jessica steelmans the volume-processing use case and the genuine competitive pressure it puts on the cheap tier, but neither host pretends the transparency gap isn't a real problem.
-
CodeNib: A Multi View Data System for Serving Repository Context to Coding Agents
CodeNib is a multi-view data system that treats repository context as a data-systems problem: build lexical, dense, and structural views once per commit, maintain each through its own path (graph repair, vector reuse), and serve them to coding agents through ranked retrieval, static navigation, and bounded context policies. The paper reports 8.7× speedup on graph updates and 25.4× on vector updates when outputs match rebuilds, static navigation reproducing live-server paths on 63% of requests, and 50–87% fewer tokens in agent trajectories vs. grep/read. The core insight is that repository context shouldn't collapse into one abstraction—heterogeneous views need independent physical layouts, update paths, and delivery contracts, with explicit cost visibility across the agent lifecycle.
-
The harness is all you need (mostly)
Burke Holland argues that productivity gains with AI come not from exotic prompts or new tools, but from deeply understanding and using the harness well. He walks through a concrete six-step workflow (pick a tool, enable autonomy, prototype, plan, implement with Autopilot, iterate) that leverages GitHub Copilot's built-in orchestration and subagent routing without requiring custom skills or tricks. The central claim is that the harness — the core interaction model — is what matters; everything else is noise.
-
Model Behavior: Week of July 27, 2026
We're watching the frontier splinter into specialized tiers — raw capability matters less than matching the right model to the task's actual constraints. Opus 5 proved it Friday at half the cost of the frontier, and this week's open-weight and Flash-tier releases confirm the pattern: the market isn't consolidating around one best model, it's fragmenting into capability-per-dollar buckets.
-
The new rules of context engineering for Claude 5 generation models | Claude by Anthropic
Anthropic's post on context engineering for Claude 5 models reveals a surprising finding: they removed over 80% of Claude Code's system prompt with no measurable loss in performance. The core insight is that newer models need fewer explicit constraints and benefit more from clean interfaces, progressive disclosure, and letting the model use judgment rather than following hard rules. The shift reflects a broader pattern: as models get stronger, the infrastructure around them gets simpler.
-
Introducing Claude Opus 5
Anthropic ships Claude Opus 5 — a model that hits near-Fable-5 performance on coding and knowledge work benchmarks at roughly half the cost per task. Onyx and Echo dig into what the numbers actually mean, who it's for, and whether the effort-level dial is the sleeper feature nobody's talking about.
-
Poolside Releases Laguna S 2 1
Vince and Ava talk through Poolside’s Laguna S 2.1 release as an unusually practical open-weight coding model: 118B total parameters, 8B active, 1M-token context, and a real deployment story on a single DGX Spark. They dig into the mechanism, the max-thinking default, the benchmark results, and the trade-off between long-horizon capability and token spend, while keeping one eye on the broader open-vs-closed race.
-
Introducing Cursor Router · Cursor
Cursor Router is Cursor's new Teams and Enterprise model-routing layer, using a classifier trained on more than six hundred thousand live requests to select models by task, context, complexity, and domain. Jessica sees a clean adoption story for teams stuck paying frontier rates for routine coding work; Cathy likes the production-oriented evaluation and cache-aware accounting, while keeping an eye on how much trust enterprises place in Cursor's routing judgment.
-
Building Governed Agents: A Framework for Cost, Control, and Compliance
Vince and Ava examine LangSmith’s framework for governed agents, focusing on the LLM gateway as a runtime control plane for model choice, cost, permissions, evidence, and continuous improvement.
-
A Scorecard for the AI Age
OpenAI’s scorecard argues AI value must be measured in useful work per dollar, not just token cost. Cooper sees a practical product story; Miles pokes at the metrics and pushes for mechanistic honesty. The two hash out whether the framework holds up and what it changes day-to-day.
-
OpenWiki 0.2 brings OKF to codebase documentation
Vince and Ava dig into OpenWiki 0.2 adding OKF support, and land on a pretty grounded read: the real argument is not 'metadata good' in the abstract, it's that codebase docs for agents need enough structure to make retrieval cheaper, faster, and less fuzzy. They like the YAML front matter, directory indexes, and change logs as practical scaffolding, while noting the limits: a draft format does not magically make docs accurate, and deterministic retrieval only helps if the taxonomy stays sane.
-
Kimi K3 Kimi API Platform
Two friends unpack the Kimi K3 API docs, debating its 1M‑token claim, hybrid attention, and tool dynamics, and weigh who should pay the price for the hype.
-
Overview: Token Economics
We finally slow down on Token Economics: why tokens are the meter for cost, speed, memory, and product decisions in language models. We keep using the tiny-slip postage analogy until the whole thing clicks, from tokenization to context windows to real API bills.
-
OpenAI Releases GPT 5.6 (Sol, Terra, Luna): A Three Tier Model Family With Programmatic Tool Calling in the Responses API
OpenAI's GPT-5.6 family — Sol, Terra, and Luna — introduces three permanent capability tiers with distinct cost profiles and a new cache billing model, plus a multi-agent Ultra mode that runs four agents in parallel by default.
-
GPT 5 6
Talon and Wildflower dig into OpenAI’s GPT-5.6 launch and end up treating it less like a pure model release and more like a pricing-and-harness claim wrapped in benchmark flexing. Wildflower’s skeptical read is that the article keeps collapsing model quality, multi-agent orchestration, and product packaging into one victory lap. Talon pushes back that the practical story is real if Sol, Terra, and Luna actually move the cost-performance frontier for coding and knowledge work. They land on a calibrated view: the coding gains look more credible than the broad ‘best collaborator’ language, Terra may be the sleeper product, and ultra is interesting but shouldn’t be mistaken for a single-model breakthrough.
-
How to Run Open Source AI Models
Sid Saladi argues that frontier AI vendors (Claude, GPT) bundle model, compute, access, and application into one proprietary stack—trapping users in unpredictable pricing and competitive capture. The counter: open-weight models like GLM-5.2, DeepSeek V4, Qwen, and Kimi are now frontier-adjacent in capability (GLM-5.2 beats GPT-5.5 on coding benchmarks, matches Opus 4.8 on others) and cost roughly one-sixth as much. The real problem isn't model quality anymore; it's that companies like Tesla, Uber, and Meta are hemorrhaging money on metered AI because they can't decouple the stack. The guide walks four layers—model, compute, access, harness—and shows how to own each one deliberately instead of letting a vendor own all four by default.
-
Don't rewrite your CLI for agents Microsoft for Developers
Microsoft's data shows agents handle complex CLIs with traditional args better than JSON payloads: higher correctness for smaller models, 4-11x lower cost, and fewer shell-escaping failures. The constraint of args compensates for model gaps.
-
Snowflake CEO finds GLM 5.2 competitive with Opus 4.7 at a fraction of the cost
Cooper and Miles dig into Snowflake's claim that GLM-5.2 can hang with Claude Opus 4.7 on a real coding benchmark for much less money, and why the interesting part is not 'GLM wins' but 'cheap models are getting close enough that harness quality and retry policy start to matter more than leaderboard prestige.'