Ep 1123 Overview 8:09 w/ Onyx & Echo

Overview: Token Counting and Optimization

We finally sit down and explain token counting and optimization from the ground up — what tokens actually are, why they're the unit of everything that costs money or hits a limit, and the real techniques for using fewer of them without breaking your prompts.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1123"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1123 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script Sonnet 4.6 Voice Fish Audio S2.1 Pro

Transcript

Onyx Okay so someone wrote in and basically asked us to stop hand-waving — their words — about tokens. And I read it and thought, yeah, we kind of do just say 'tokens' and move on like everyone already knows what that means.

Echo We absolutely do. We've talked about token costs, token limits, token efficiency — and I think we've never actually stopped and said what a token IS from the ground up. So, let's do that.

Onyx And for anyone keeping score at home — we did a full Overview on token efficiency back in episode eight nineteen. Today we're zooming in on one specific piece of that: the counting, and the optimization. The 'how do you actually measure this thing and then spend less of it' part.

Echo Right. So, the intuition first, because the mechanism is actually pretty simple once you have the right picture. A language model does not read your text the way you do. It doesn't see words. It sees chunks — little pieces of text that might be a full word, might be part of a word, might just be a punctuation mark. Those chunks are tokens. And the model processes them one at a time.

Onyx So it's not reading sentences, it's reading… a ticker tape of pieces.

Echo Yeah, exactly — ticker tape is good. And here's why that matters immediately: every single thing about running a model is denominated in tokens, not words. The cost, the speed, the memory limit — all of it. So if you want to understand why something is expensive or slow or hitting a wall, tokens are where you look.

Onyx Before we go further — we have a full episode on context windows, that's episode six hundred, and one on token economics, episode six seventy-five, for the deep dives. But real quick: context window just means the total number of tokens a model can hold in its working memory at once. And token economics is basically how the pricing model works — tokens as the billing unit. Okay, keep going.

Echo So the analogy I keep coming back to is a taxi meter. You get in the cab, the meter starts running, and it doesn't care how far you THOUGHT you were going — it charges for the actual distance. Tokens are the meter. You write a prompt, the model slices it up, counts the slices, and that's your bill. The problem is the slicing is invisible. You don't see it happening.

Onyx And the meter runs differently in different cabs.

Echo Exactly — and that's the key thing. Different models use different tokenizers. The tokenizer is a small algorithm that runs BEFORE the model does any real work — it's just converting your text into a sequence of integer IDs. GPT models use one tokenizer, Claude uses another, open-weight models use others still. Same sentence, different tokenizer, different count, different cost. And we did a whole tokenization episode, episode six oh one, if you want the full picture.

Onyx Okay, that explains something I've noticed. If I write something with a lot of unusual words, it feels like it costs more.

Echo It does. And language is a big one here. English tokenizes really efficiently — the merge rules were learned mostly from English text, so common English words are dense. Chinese or Arabic or something with a script that doesn't map cleanly to the Latin alphabet… you can end up spending significantly more tokens per character. Same information, higher meter reading. And then there are the quirks that just bite you in practice.

Onyx Oh no.

Echo Yeah. A nicely formatted JSON blob can be genuinely token-heavy. And this is where the optimization story starts, because a lot of token waste is just… invisible formatting. Whitespace you added for readability. Repeated headers. Verbose field names. None of it is doing work for the model — it's just costing you.

Onyx So the first optimization move is just: look at what you're actually sending.

Echo Exactly. There are pre-flight token counting tools — Claude's platform docs have an endpoint where you can count tokens before you send a request, and there are browser tools that'll show you GPT versus Claude versus Gemini counts side by side. You don't have to guess. You can measure before you pay. And let me walk through the actual optimization toolkit, because there are a few different moves at different levels.

Onyx Over six hundred times, right? We saw that in the BenchLM data.

Echo Yeah. So if you're routing everything to the top-tier model by default, you're leaving a lot on the table. The smart move is routing discipline — figure out what actually needs the expensive model and what doesn't. And this connects to the inference optimization episode we did, episode eight seventy-nine. Once models hit capability parity on easy tasks, serving efficiency is the whole game. Token counting is how you play that game.

Onyx And there was that coding agent piece — the context compiler argument. Which was basically: don't give the agent a giant context window full of everything, build something that compresses and selects. The context window is your working memory budget. Token counting tells you how much of it you're using. The context compiler is a way to spend that budget wisely instead of just dumping everything in.

Echo That's a good way to put it. And the orchestration framework comparison — LangChain versus LlamaIndex versus raw API calls — that whole debate had token management running underneath it. Every framework makes different choices about how much context it carries between steps, how it handles tool call outputs, whether it prunes old history. Those choices are all token-counting decisions dressed up as architecture decisions.

Onyx Okay — where does this sit right now? Like, is token counting still the thing you have to think about, or has something come along that kind of… absorbs it?

Echo Still completely load-bearing. If anything, more so. As agent workloads get longer — multi-step reasoning chains, retrieval, tool calls, memory reads — the token spend per task compounds. A single chat message is maybe a few hundred tokens. An agent loop that runs for twenty steps, each with a context that's growing… you can hit tens of thousands of tokens per task easily. The meter is running the whole time.

Onyx And the tooling has caught up a bit.

Echo Yeah. Pre-flight counting is standard now — you can hit an endpoint and get an exact count before you send. API responses have always returned usage data, but more teams are actually reading it and building dashboards around it. There's a small ecosystem of cost management tools that sit in front of your API calls and track token spend the way you'd track cloud spend. It's becoming infrastructure, not an afterthought.

Onyx Which is probably the right place for it. You wouldn't run a cloud bill blind — same idea.

Echo Exactly. And the pricing spread I mentioned — that six-hundred-fold variation — that's not going away. The market has fragmented into tiers pretty deliberately. So the optimization question is now also a routing question: which tier, which model, for which task. You can't answer that without counting.

Onyx Okay. I think the thing I want to make sure landed — because we've been building to it the whole time — is that token counting isn't a billing chore you do after the fact. It's the feedback loop that tells you what your prompts are actually doing. You count, you see the waste, you trim the waste, you check that trimming didn't break anything, and you repeat.

Echo Yeah. And the one thing I'd add: not all tokens are waste. Some of them are load-bearing — they're buying you reliability, or clarity, or robustness. The optimization game is figuring out which ones are doing a job and which ones are just… whitespace and habit. Count everything, cut the ones that aren't earning their place.

Onyx The meter doesn't care which tokens are useful. You have to.

Echo That's the whole thing.

Onyx Echo, I feel like we owe that listener an apology for every time we just said 'tokens' and kept moving. Hopefully this squares the debt. We'll be back.