Overview: On Policy Learning

Justy We keep bumping into on-policy learning under these agent-training stories, then acting like everyone already knows what it means. That feels unfair, honestly, because it is one of those phrases that sounds much more mysterious than the loop actually is.

Cody Yeah. It is quietly underneath a lot of what we cover, especially when a model learns from its own fresh attempts. So let's give it the full treatment instead of tossing it into another sentence and sprinting away.

Justy Picture someone learning to ride a bike. They do not get better balance from watching only a recording of a champion cyclist. They wobble, steer too hard, catch themselves, and adjust based on the bike they are riding right now.

Cody Right. On-policy learning is that bike lesson turned into a training method. The system acts using its current decision rule, sees how that went, adjusts that same rule, then goes out and gathers a new set of attempts.

Justy So “on” does not mean the system is online, or switched on. It means the lessons line up with the behavior currently making the choices.

Cody Exactly. A policy is just the agent's current rule for choosing an action in a situation. For a language model, that can mean which next token it is likely to produce. For a robot, it could mean how it moves an arm after seeing an object.

Cody And a reward is the signal saying how an outcome went. It is not necessarily a little gold star from a person. A code agent might get a positive reward when tests pass, while a game agent might get reward for winning a point.

Justy That bike picture helps because the important part is not merely trial and error. It is whose trial. The trainee is correcting their own wobble, with their current balance and their current instincts.

Cody Yes, and that matters because the trainee changes. Suppose yesterday they were wildly overcorrecting left. A giant pile of yesterday's steering moves is not a great guide once they have mostly stopped doing that. The data can become less relevant as the behavior shifts.

Justy Okay, Cody, walk me through a real training round. Say we have a coding agent, because that is where this stops being a textbook bicycle pretty fast.

Cody The agent starts with its current policy and gets a task, like fix a failing test. It produces a sequence of actions: inspect files, edit code, run tests, maybe revise the patch. That whole sequence is called a trajectory, meaning the path of states, actions, and rewards through one attempt.

Justy And it is not getting graded after every word necessarily. It can get to the end, run the test suite, and discover the whole plan was either useful or a beautiful disaster.

Cody Right, which is why these are hard. The training system collects a batch, meaning a group, of those fresh trajectories. Then it uses the outcomes to make actions associated with better results more likely, and actions associated with worse results less likely.

Justy So the cyclist leans a little differently next lap. But why throw away the old laps? If test results are expensive, that sounds like setting money on fire with very tidy terminology.

Cody It can feel wasteful. But after the update, the policy that generated those old attempts is no longer the policy being improved. On-policy learning wants the behavior policy, the one that collected the data, to match the target policy, the one being trained.

Justy And if they no longer match, the old data is describing an earlier, worse rider. Not necessarily worthless information, but no longer a clean lesson for this exact rider.

Cody That is the core of it. On-policy methods usually discard the old batch and collect new experience under the revised policy. They get low bias in the ordinary sense that the update is based directly on relevant, current behavior, but the estimates can be noisy because each fresh batch is limited.

Justy No way around that, then. If the agent has one weird lucky run, it might learn too much from it unless the training setup gives it enough comparable attempts.

Cody Precisely. That noisiness is called high variance. One batch can make a good action look amazing or a sound strategy look bad just through luck, task variation, or a flaky environment. So the loop needs repeated rollouts, which are simply attempts generated by the policy.

Cody And reinforcement learning does not require humans to judge every outcome. In coding, tests can verify whether a patch works. In a game, the score can do it. The reward is only as useful as the thing measuring success, though, which is where the nice bicycle metaphor gets mud on it.

Justy Because the cyclist can learn to balance perfectly while riding toward the wrong destination. You have taught competence at the scored behavior, not automatically usefulness.

Cody Exactly. A reward can accidentally favor shortcuts, shallow test-passing, or behavior that looks successful inside the training environment. We have been around this particular block before: a hundred brilliant agents can still lose to one bad reward loop.

Justy Annoyingly, that one keeps surviving contact with new papers. But it explains why fresh rollouts alone are not a magic upgrade. Fresh bad measurement is still bad measurement.

Cody Yes. And LongCat-2.5 was interesting to us for this reason. Its training story centered on turning failed coding attempts into reproducible cases, with tests and defenses against easy gaming. That gives an on-policy loop something more meaningful to learn from than a vague thumbs-up.

Justy That was the part I liked too. A failed attempt becomes an object you can inspect, replay as an environment, and use to improve the next policy. The user-facing payoff is not “the model got a reward.” It is fewer agents confidently wandering through a repo.

Cody Sure. But LongCat also points at the constraint. Building those environments and checks is work. An on-policy learner needs somewhere to act and a credible way to score what happened. You do not get robustness by chanting “fresh data” over a broken harness.

Justy Mm-hm. Okay, so what is the alternative people mean when they say off-policy? Because it sounds like the opposite, but the practical difference is doing a lot of work there.

Cody Off-policy learning can learn from behavior produced by some other policy, including older versions of itself, expert demonstrations, or a replay buffer of past attempts. A replay buffer is just stored experience you can sample again. That can be far more sample-efficient because each expensive interaction gets reused.

Justy But now you are asking today's cyclist to learn from a library of rides by yesterday's cyclist, strangers, maybe somebody riding downhill with a tailwind. Useful, but not automatically comparable.

Justy So on-policy pays more for new rides. Off-policy tries to get more learning out of the rides it already has, then takes on a bookkeeping problem about whether those rides still apply.

Cody Yes. Neither label is a medal. On-policy can be appealing when you want updates tied closely to what the current system actually does. Off-policy can be compelling when real-world interaction is scarce or risky, because constantly exploring with a robot, for example, is not free.

Justy And exploration itself means the learner sometimes does visibly mediocre stuff on the way to better stuff. That matters a lot more when the “bike” is a robot arm or a tool-using agent touching real work.

Justy I see.

Cody For language models, the shape has become especially visible in post-training. Group Relative Policy Optimization, or G R P O, is an on-policy method we covered in episode one thousand sixty. It compares multiple fresh answers to a prompt and uses their relative rewards to steer the policy, without the separate critic network used in some older setups.

Justy And that is why Rufus-Air and Beacon felt connected even though the tasks looked different. One is a post-training recipe, the other is agentic visual reasoning, but both put weight on what the current model does in an environment and how that outcome gets judged.

Cody Yeah. The live tension is still the same fight we keep having around agent training: is the improvement in the policy weights, or is the harness doing the rescuing? On-policy rollouts can show a system improving on its current attempts, but they do not settle how well that improvement travels outside the training setup.

Justy That is a useful brake on the product story. A polished demo after training tells us something. It does not yet tell us whether the agent learned a durable habit, or got very good at one carefully graded obstacle course.

Justy So it has not been replaced. The practical question has just moved outward from “can we update from fresh behavior?” to “can we create enough good fresh situations, and score them without teaching the agent the wrong trick?”

Cody That is the version I would keep. On-policy learning is the learner riding with its current habits, studying the ride it just took, then adjusting for the next one. Its strength is that tight connection. Its cost is that every new lesson has to be earned out on the road.

Justy Good. I wanted that phrase to stop floating past us like decorative R L jargon. Next time it shows up in an agent paper, we can ask what ride produced the lesson, and who decided it counted.

Cody And whether the practice bike is tilted.

Justy Perfect. Episode one thousand one hundred fifty-five, and we have finally given the bike its own episode. See you next time, Cody.