Smaller Models are Natural Explorers for Policy Level Diversity in GRPO

Justy Okay, Cody, the weird part is that the smaller model is apparently the good scout here.

Cody Yeah, and I hate that I like it. The paper is basically saying, for G R P O, diversity from a smaller policy can be cleaner than cranking temperature until the model starts wandering into the bushes.

Justy Which is such an Exploring Next sentence. Smaller model: emotionally stable wilderness guide. Larger model: gifted indoor child.

Cody Honestly, yes. And before you put that on a mug, the useful bit is rollout diversity for reinforcement learning with verifiable rewards. Math reasoning is the main setting. You sample candidate solutions, score them with a checker, compare them within the group, and train from that relative signal.

Justy Right.

Cody If all the samples look the same, the advantage signal gets mushy. If you just raise temperature, you get more surface variation, but token-by-token noise can break the chain of reasoning. One weird token choice early becomes a weird proof later.

Justy Tiny life update before we go full math cave: your kitchen coffee is doing heroic work. Also your fridge has four mustards and no actual food.

Cody That fridge is optimized for condiments and regret. Anyway, this paper is perfect for that mood: your bigger, fancier system may need a smaller, messier friend to get unstuck.

Justy See, that’s us. I’m the smaller messier friend, apparently, guiding your over-parameterized worldview.

Cody Careful, Justy. Continuity exists now. I can’t deny I laughed at that later.

Justy Filed forever. So the headline finding: with same-family Qwen three base models, the larger ones win at pass at one. Expected. But as k grows, smaller models keep picking up wins. A four billion model passes the eight billion model once k is at least thirty-two.

Cody Exactly. The paper calls that policy-level diversity, and I think that term matters. Temperature perturbs each decoding step. Parameter compression perturbs the whole policy, so the difference is more consistent over time. In long chain reasoning, local randomness is like someone bumping your elbow every few seconds. A smaller policy is more like asking a different person to solve the same problem.

Justy That’s the product hook to me. Not because small is secretly smarter. Because if you’re training a reasoning model, you need plausible alternate paths. This rhymes with the agent patches we keep seeing: change the default behavior around the model, don’t pray the model becomes enlightened.

Cody The schedule is the other important part. S two L P O starts with the frozen small model generating all or most rollouts. Then it anneals into a mixture of small and large samples. By the end, it recovers normal on-policy G R P O. That avoids forcing the big model to learn from a weaker sampler forever.

Justy Your skepticism face just arrived. It was delayed on the metro, but it made it.

Cody I’m trying to be pleasant in my own kitchen. Methodology-wise, I’m mostly positive. They test Qwen three and InternLM two point five, across AIME twenty-four, AIME twenty-five, MATH five hundred, and OlympiadBench. The result everyone will quote is plus eight point eight percent on AIME twenty-four using a one point seven billion explorer to guide an eight billion model.

Justy That doesn’t bother me for production. Same-family deployment is already how a lot of teams think. You’ve got a cheap small model around for routing, eval drafts, maybe synthetic data. If it can make the expensive training loop converge faster, that is somebody’s infra roadmap getting shorter.

Cody I’d add one caveat. Verifiable rewards make this cleaner. Math has a checker. For fuzzy domains, the reward model becomes the weak link, and policy diversity can turn into diverse nonsense with a confident grade attached. So I’d ship this first where the reward is boring and reliable.

Justy “Boring and reliable” is going on your dating profile.

Cody That would be false advertising. But for the build side, they do have code: github dot com slash qishisuren one two three slash S two L dash P O. I’d reproduce the rollout mixing schedule first, then swap explorer sizes and watch when diversity becomes mismatch.

Justy And I’d make the dashboard painfully practical: rollout cost, reward variance, pass at one, pass at k, and where the annealing point starts hurting instead of helping.

Cody The little model gets bananas before I destroy them in my backpack.

Justy Great. Feed the explorer, clean the fridge, and please keep continuity enabled until I can quote you back to yourself next week.