Kimi K3 Is Here: Efficient Day 0 Support on vLLM
Vince and Ava unpack Moonshot AI's Kimi K3, a 2.8‑trillion‑parameter multimodal MoE, and its day‑zero support in vLLM. They walk through the model’s hybrid attention, the engineering tricks that make a 1 M‑token context feasible, the practical deployment recipe, and how it stacks up against other frontier models.
Transcript
Vince Whoa, I just saw the vLLM release notes for Kimi K3 and thought, if this lands today, everyone's dev pipelines are about to get a serious upgrade.
Ava Yeah, the model is a 2.8‑trillion‑parameter Mixture‑of‑Experts with a one‑million token context, and vLLM actually ships day‑zero support—no extra glue code, just the Docker command they published.
Vince Right.
Vince Exactly, the moment you pull the weights you can spin up an eight‑GPU node and start serving, which is huge for anyone who’s been waiting for a truly open‑weight frontier model.
Ava By the way, how's your week been, Vince? I swear the only thing keeping me sane is the fact that my GPU farm finally got the power budget upgrade.
Ava Kimi K3's core novelty is Kimi Delta Attention, a hybrid that mixes a recurrent linear state with periodic full‑attention layers. That lets the model keep a fixed‑size state instead of a growing KV cache, which is why a one‑million token window is affordable.
Vince Yeah.
Vince From a product standpoint, the target user is anyone running multi‑tenant AI services—think SaaS platforms or internal toolchains that need long context and multimodal inputs. The Docker recipe spins up eight NVIDIA B300 GPUs (or the AMD MI355X equivalent) and the vLLM flags enable prefix caching and speculative DSpark decoding, which together push throughput from about 118 tokens per second up to roughly 370 tokens per second per user.
Ava Sure.
Ava vLLM had to rewrite its KV manager to hold two parallel caches—paged blocks for the full‑attention layers and compact recurrent blocks for the KDA layers. They also introduced a hybrid prefix‑caching system that snapshots the recurrent state before extending it, so long prompts can reuse both caches.
Vince Okay.
Vince I love the speed boost, but I'm still uneasy about the AMD support claim. The docs say MI355X works, yet the pre‑built images still reference a FlashInfer build that's only verified on NVIDIA. If you run into missing kernels you'll hit a wall, so early adopters should probably stick with the B300 stack until the AMD path is fully tested.
Ava Mmh.
Ava You know, KDA sounds like a new dance craze—'Delta Shuffle' maybe? Imagine agents doing a little two‑step every token. No wonder the specifiers call it 'speculative decoding', it's basically the model trying to dance ahead of the music.
Vince Okay, that's genuinely funny.
Ava In the broader race, DeepSeek V4 Flash just dropped a 3‑trillion‑parameter open model last month, and OpenAI’s GPT‑5.6 tier still anchors the closed side with its Luna pricing. Kimi K3’s 1 M token window and cheap speculative path give the open‑weight camp a tangible advantage in long‑context workloads, even if the raw FLOPs aren’t quite the highest. It’s a clear sign the frontier is fragmenting between massive closed models and these highly engineered open alternatives.
Vince Right.
Vince If you want to give it a spin, the exact command is: vllm serve moonshotai/Kimi-K3 \ --tensor‑parallel‑size 8 \ --trust‑remote‑code \ --load‑format fastsafetensors \ --enable‑prefix‑caching \ --enable‑auto‑tool‑choice \ --tool‑call‑parser kimi_k3 \ --reasoning‑parser kimi_k3.
Ava Exactly.
Ava Give it a try on a single node and watch the latency numbers climb.
Vince All right, Ava, that's a lot to chew on—let's see if the real world lives up to the hype. Catch you later.