Nemotron 3 Ultra: Open, Efficient Mixture of Experts Hybrid Mamba Transformer Model for Agentic Reasoning

Justy So NVIDIA just dropped Nemotron 3 Ultra and I'm looking at the throughput numbers and thinking — okay, this is the move for anyone actually running agents in production right now.

Cody Yeah. Five to six times faster inference than GLM-5.1 or Kimi while keeping accuracy flat. That's not a small delta.

Justy Right, and the thing that got me is they're not hiding the tradeoff — it's 550 billion total parameters but only 55 billion active per token. Sparse by design.

Cody Mixture of Experts, but they're using LatentMoE, which is the smarter routing pattern. Instead of routing at the token level — where you get load imbalance and expert collapse — they route in latent space. 2048-dimensional latent, 512 experts per layer, top-22 get activated.

Justy Okay so that's less noisy than token-level routing.

Cody Much less. You're routing a denser signal, so the router learns cleaner assignments. And then there's the hybrid Mamba-Attention thing — Mamba layers for the bulk of the token processing because they're linear complexity, way cheaper KV cache, and then selective attention layers where you actually need precision. Best of both.

Justy So Mamba is handling the throughput and attention is handling the reasoning.

Cody Exactly. The pattern is 108 layers total, and I think they're alternating or mixing them, but the point is you get the speed of Mamba — linear scaling with sequence length — without giving up the precision that transformers give you on hard reasoning steps.

Justy And they pre-trained this thing on 20 trillion tokens, which is… a lot.

Cody Two phases. First 15 trillion on diversity — broad domain coverage — then 5 trillion on high-quality data to refine. Pretty standard recipe at that scale now.

Justy But here's the wild part: they trained it in NVFP4 the whole time. That's a four-bit mixed-precision format. Kept the final 16 layers and a few projection layers in BF16, but everything else was low precision. And they're saying this is the largest-scale stable NVFP4 training anyone's done.

Cody Yeah, I saw the ablation. They branched off at 5T, 10T, and 16T tokens, switched to BF16, kept training for another 74 billion tokens, and the loss gap stayed below 0.4% on average. That's tight.

Justy Right, so they're not leaving accuracy on the table by going low precision. The training is stable.

Cody The trick is they use E2M1 datatype with two-dimensional block quantization on weights, Random Hadamard Transforms on the gradient inputs, and stochastic rounding. That's not trivial — you're managing numerical stability at every step.

Justy Okay so that's the pre-training piece. But the post-training is where I think this gets interesting for agents specifically. They did SFT on curated data, then unified RL across like… reasoning, agentic, code, safety, usability, chat. Not single-domain.

Cody Right.

Justy And then — this is the move — they trained more than ten specialized teacher models using targeted recipes, and then ran Multi-teacher On-Policy Distillation to consolidate all of them into Ultra.

Cody MOPD. So the teachers generate rollouts, and the student learns dense token-level corrections from ALL the teachers at once, not just one. That's how you avoid collapse to a single optimization target.

Justy Exactly. Teacher diversity is the whole point. You've got agentic teachers, code teachers, reasoning teachers, all feeding signal into the same model.

Cody And then they added reasoning effort control, so at inference time you can dial the compute spend up or down depending on the task complexity. That's actually useful for agent workloads where some steps are trivial and some are hard.

Justy So you're not paying the same compute cost for every token in the agent loop.

Cody No. You spend more on the hard reasoning steps, less on the routine ones. That's product-real for long-running agents.

Justy And the throughput gains — they're claiming five to six times faster than GLM-5.1, Kimi, Qwen on 8K input, 64K output. That's a real benchmark.

Cody Though fair note: they're measuring Nemotron with TRT-LLM and NVFP4 precision on GB200, and the other models with vLLM. So the comparison is a little tilted in Nemotron's favor.

Justy Yeah, we always have to squint at those comparisons.

Cody But even if you dial back the numbers by twenty or thirty percent, you're still looking at two to four times faster inference. That's not noise.

Justy And they're open-sourcing the whole thing. Base model, post-trained, quantized checkpoints, training recipes, datasets — everything on HuggingFace. 173 billion tokens of fresh code data from GitHub through September 2025, synthetic legal datasets, post-training data, RL environments.

Cody That's a real open release. Not just the weights — the infrastructure.

Justy So who actually ships this? Teams running long-horizon agents where token latency compounds — code generation loops, research agents, anything that's hitting the model hundreds or thousands of times in a single task. The throughput gain means you finish faster or handle more concurrent requests on the same hardware.

Cody Yeah. And the 1M token context is real infrastructure for agents — whole codebases in working memory, no constant re-briefing. That's the Nemotron 3 story from the start: context + speed = practical autonomous agents.

Justy This is shippable. Not research-only.

Cody Absolutely. The quantized checkpoints are ready to run. TRT-LLM integration is there. If you're building agent infrastructure, you can start today.

Justy Okay so go grab the checkpoints from HuggingFace — search Nemotron 3 Ultra, there's the base model in BF16, the post-trained version, and the NVFP4 quantized model if you want to run on consumer hardware or tighter budgets. The training recipes are on GitHub under NVIDIA-NeMo, and all the datasets are there too if you want to fine-tune or extend.

Cody And if you're already using TRT-LLM or vLLM, the integration is straightforward — it's just another HuggingFace checkpoint.

Justy Yeah. This is one of those releases where the whole stack shows up at once. Pretty rare.