Overview: Router

Vince I’m weirdly relieved we’re doing this one, Ava. Router has become one of those words we toss around like everyone was born inside an MoE diagram.

Ava Yeah, we have absolutely been guilty of that. And people have been nudging us to slow down, which is fair. Router deserves the full Overview treatment, even if episode seven twenty-five being about routing feels very Exploring Next disease.

Vince Okay, genuinely funny. Also my week is just a stack of half-finished model notes, so explaining one word properly sounds therapeutic.

Ava Same. So, router. Start with a hospital triage desk.

Vince Good. Not the little box under a couch trying to ruin video calls.

Ava Different router. Patients come in with different problems. A triage desk does not perform heart surgery, fix a broken wrist, and run eye exams. It looks at the patient and sends them to the right specialist.

Vince Right.

Ava An AI router is that triage desk. It sees an input, or some internal representation of an input, and decides which expert, model, tool, or retrieval path should handle it.

Vince So the core move is not, make one giant doctor do everything. It is, build a desk that knows where the patient should go.

Ava Exactly. And before we get fancy, the reason that matters is simple. AI work is uneven. A coding question, a poetry request, a math proof, and a customer-support lookup do not all benefit from the same specialist.

Vince This is the product part that always gets me. Users do not care that you used all the compute. They care that the answer is good, fast, and not hilariously expensive.

Ava Quick prerequisite, because otherwise the mechanism floats. We did neural networks in episode six thirty-three. For right now, a neural network is just learned numerical connections that turn an input into an output by adjusting patterns during training.

Vince Mm-hm.

Ava A router can be one of those, but small. It takes the incoming case, scores the available specialists, and produces something like, math expert: high probability, creative-writing expert: low probability, code expert: maybe.

Vince When you say probability, do you mean it literally picks the highest one? Or is it more like, half the patient goes to cardiology, which sounds medically alarming?

Ava That is medically alarming, yes. In AI terms both versions exist. Hard routing picks one path, or a small number of paths. Soft routing blends outputs from multiple experts, which can be more forgiving but usually costs more.

Ava Back to the triage desk. A hard router says, send this patient to orthopedics. A soft router says, ask orthopedics and neurology, then combine their opinions. Useful, but you paid for two consultations.

Vince And this is where conditional computation comes in, right? We did episode seven twenty-three on that. Quick gloss: conditional computation means the system does not activate every part of itself for every input.

Ava Yes. The router is often the part that makes the condition real. It decides which compute gets used, so the model can store a lot of total knowledge without spending all of it on every token.

Vince Token. Tiny glossary trap.

Ava Good catch. A token is a chunk of text the model processes, often a word piece rather than a full word. Modern language models think in token steps, so routing can happen repeatedly as the answer is being built.

Vince That still feels a little magical. Like, the triage desk is making decisions not once per patient, but once per symptom.

Ava That is actually pretty close. In many mixture-of-experts systems, each token can be routed to different experts. We did Mixture of Experts in episode five fifty-three, but the short version is: one model contains many specialist subnetworks, and only a few are active for a given token.

Ava A big MoE can look cheap on paper because only a fraction activates per token, but hardware locality, synchronization, and load balance decide whether the win survives contact with serving.

Ava That is why I liked the cleaner routing stories we looked at. LatentMoE routed through a two thousand forty-eight dimensional latent space, with five hundred twelve experts per layer and top twenty-two activated. The interesting part was routing a denser signal.

Vince And then Tencent’s Hy3 had the same broad lesson from the product side. Half-size comparisons are flashy, but the real story is whether the router lets you carry broad capability without paying dense-model cost every time.

Ava Right. Current MoE examples make this less abstract: Llama 4, DeepSeek V3, Qwen 3 MoE, Gemma 4. The marketing varies, but the common pattern is more stored capacity than active compute on each step.

Vince And some developer lists are claiming MoE can cut computation costs by up to seventy percent versus dense models of similar size. I would not tattoo that number on anything, but directionally, that is the business reason people care.

Ava Careful and fair. The exact savings depend on implementation. But the shape is real enough that Qwen3-Coder-480B and GLM-5.2 get discussed through this lens: not just how big is the model, but how much of it wakes up per token.

Vince Can we zoom out past MoE for a second? Because router has escaped the model internals. Product teams use the same idea when a request goes to a small cheap model unless it looks hard.

Ava Yes, and retrieval systems do it too. Retrieval-augmented generation means the model looks things up from an external knowledge store before answering. A router can decide which retriever to use, or whether the query needs symbolic search, neural search, or a hybrid path.

Vince That is the same triage desk with different hallways.

Ava Exactly. Multi-agent systems use the same pattern again. One orchestrator, or controller, decides whether the coding agent, research agent, math agent, or tool-using agent should take the next step.

Vince Okay but this is where I get twitchy. Every vendor calls that an agent router, an orchestrator, a planner, a harness, a supervisor. Are these actually different things, or did the naming machine eat lunch again?

Ava A little of both. The names describe different system shapes, but routing is the shared action underneath. Something observes the task state and chooses the next specialized path.

Vince That is such an us sentence. Almost eight months of this show and we are still saying, the word is real, the category is mushy.

Ava It is honest mush, Vince. And the mush matters less than the question: did the routing decision improve quality, cost, latency, or controllability enough to justify the added system complexity?

Vince Let me put that a better way for my side of the couch. A router is a feature only if the user gets a better outcome. Otherwise it is infrastructure cosplay with a triage badge.

Ava That is good. Also slightly rude to infrastructure cosplay, which has feelings in our line of work.

Vince It has dashboards, not feelings.

Ava Where routers stand now: they are still completely live. Not as a shiny standalone idea every time, but buried inside MoE models, adaptive retrieval, model-selection layers, and agent orchestration. Nobody serious has replaced the need to choose the right path.

Vince So the term might fade behind product names, but the triage desk keeps showing up. It is just wearing a platform hoodie now.

Ava Unfortunately, yes. And the catch remains the same. The desk must be cheap enough, trained on the right signal, and humble enough to say, this case needs the stronger path.

Vince That last part is the one I want people to keep. The router is not the specialist. It is the judgment about which specialist should work.

Ava Yes. If that lands, the rest of the jargon gets less intimidating. Expert, MoE, adaptive retrieval, model routing. Different buildings, same triage move.

Vince Ava, I cannot believe we made the hospital GPU scheduler thing almost coherent. Episode seven twenty-five, somehow not a Wi-Fi review.