vLLM x AgentX: Optimizing for Real World Agentic Serving
Vince and Ava unpack vLLM’s AgentX post on agentic serving, debating whether its full-stack KV-cache and P/D disaggregation story is a genuine product win or benchmark engineering dressed as insight.
Transcript
Vince Okay, so vLLM is basically saying agentic serving isn’t just a bigger context problem, it’s a whole stack problem. Ava, you’re gonna hate that.
Ava I don’t hate it, I just think they’re dressing up KV cache plumbing as a product insight. AgentX is real traces, fine, but they’re benchmarking their own optimizations against Opus 5 API pricing. That’s a weird comparison.
Vince Come on, the pricing comparison is the point for users. If you can get 14.6 to 106 times cheaper on DeepSeek V4 Pro, MiniMax M3, Kimi K3, that’s not plumbing, that’s a budget.
Ava Sure, but TCO against a closed API with unknown routing isn’t a controlled experiment. It’s marketing math. The interesting part is the workload they’re describing.
Vince That’s exactly what I mean. Median 43 turns per session, 142K input tokens, 444 output, prefix hit above 96%. That’s not a toy workload, that’s Codex-level enterprise traffic.
Ava Right.
Vince And 44% of sessions have subagents with a median of four rollouts. You can’t treat that like a chat request. You need cache warm and close to compute.
Ava I’ll give them that. Prefix cache pressure is real. The question is whether their fixes generalize beyond the three models they show.
Vince They’re not claiming universal. They’re saying hybrid KV cache management with one uniform page and a shared block pool is the foundation. That lets them reallocate between full attention, sliding window and linear attention on the fly.
Ava Mm-hm.
Vince DeepSeek V4 originally fragmented into 92 tensors. They packed it into one contiguous backing allocation per block. That cuts descriptor and P/D transfer overhead and saves roughly 10% memory with the FP4 indexer.
Ava Okay, that’s a real efficiency win. I’m less sold on the distributed offloading story. Mooncake Store as a hierarchical pool with standalone-store mode sounds great until you’re the one running the CPU and disk tier.
Vince You’re being Ava about it. They’re shipping session-aware retention too — interval-based prompt-end snapshots plus Marconi-style selective retention for reuse boundaries.
Ava Which is sensible for hybrid models where you need linear state at reuse boundaries. I’m just flagging the operational tax. Asynchronous lookups, parallel send/receive, PR 46188, 45444 — that’s a lot of moving parts.
Vince No argument. But the bitter lessons are the part I like. Pipeline parallelism does not fit warm agentic turns. Decode context parallelism does not transfer cleanly to DeepSeek V4. Load balance does not guarantee better performance.
Ava Exactly. That’s the honest bit. They’re basically saying the control layer is the product, not the model weights. I’ve been saying that since ep 924.
Vince You have. And here it’s showing up as 130K total tokens per GPU-second on DeepSeek V4 Pro and up to 376 tokens per second interactivity on MiniMax M3 on AgentX.
Ava Those numbers are verifiable on a public benchmark. I’ll concede that. My pushback is scope. This is vLLM showing vLLM wins on vLLM-optimized models.
Vince Fair. But for teams actually running agents at scale, the cost delta versus Opus 5 is the difference between shipping and not shipping. That’s Justy’s question.
Ava Who should care? Anyone with long-running multi-turn sessions and prefix reuse. If you’re doing 1-shot chat, ignore it. If you’re doing code agents with subagents, you should care.
Vince And the routing integration with Dynamo and llm-d means cache hits can land on any instance. That’s the part I want product people to hear.
Ava Yeah, I’ll buy that the data plane plus execution plane plus control plane framing is useful. It’s just not a new paradigm, it’s harness engineering.
Vince We agree then. The verdict is: the plumbing matters, and vLLM is making it less terrible. Build Next is basically try it.
Ava If you’re already on vLLM, pull the AgentX traces SemiAnalysis released and test your own P/D ratio. And if you’re using Mooncake Store, try standalone-store mode for CPU disk tiers.
Vince And stop pretending load balance is enough. The loop is the product. Okay, that’s our Wednesday. I’m out.