Agentic Rl

Justy Okay, so Cameron Wolfe just dropped this piece on Agentic RL and the central claim is wild — we've been training models to answer questions, but now we have to train them to live in a loop.

Cody Right.

Justy Like, the agent isn't just the LLM backbone anymore. It's the whole harness — the tools, the environment, the instructions all running in this multi-turn trajectory where the model has to recover from its own mistakes.

Cody Yeah, that's the key shift. He's saying single-turn completions are basically dead for real work. If your agent can't handle state changes across ten tool calls, it's not an agent, it's just a chatty API wrapper.

Justy Exactly. And the way they're doing it now is encoding tool calls directly in the token stream. Qwen3 uses these XML-style delimiters — <tool_call> tags to trigger a stop, parse the call, execute it, then feed the result back as <tool_response>.

Cody Mm-hm.

Justy So the environment isn't some external black box anymore. The tool definitions ARE the environment dynamics. You're literally teaching the model the rules of the world via token delimiters.

Cody Wait — are we actually still not over the harness thing from ep five fourteen? Because this feels like Akshay's thesis but now with RL math strapped to it.

Justy Oh my god, you're right. 'Product is the harness' but now the harness is part of the loss function. That is such an Exploring Next take.

Cody I cannot believe we're making this connection on a Monday morning. But yeah, technically it holds up — the problem is long-horizon tasks produce these super-long execution traces that break standard optimizers.

Justy How so?

Cody So most post-training uses GRPO, which does group-wise optimization. But when you compact a long trajectory into sub-traces, different rollouts under the same prompt give you wildly different numbers of trainable steps. You can't compare groups anymore.

Justy Right.

Cody That's why GLM five point two switched to PPO. They needed a critic to estimate token-level advantages instead of relying on group-relative comparisons. It's more stable when the agent has to reason over hours of tool calls.

Justy Okay, but who actually cares about this? Like, is this for every team slapping an agent on their SaaS product?

Cody No way. This is for the folks building agents that run autonomously for days — think coding agents that debug across multiple files, or robots planning cross-day tasks like that WorldLines benchmark we covered.

Justy Right, right. If your agent only does one-and-done tasks, you don't need agentic RL. But if you're building something that has to maintain state, recover from errors, and coordinate across sessions? This is the infrastructure.

Cody Exactly. And the memory piece is crucial here — he mentions memory becomes part of the environment, accessible via tool calls. So persistence isn't just a UX feature anymore, it's an RL constraint.

Justy That's the product reframe. You're not just selling 'smarter models,' you're selling a harness that can survive its own mistakes over long horizons. The model is just the brain stem; the product is the loop.

Cody I hate when you're right about the product angle. But yeah, the technical bottleneck is real — if your rollout infrastructure can't handle variable-length traces with stateful feedback, your agent will collapse.

Justy So no generic 'add agentic RL' button then. This is deep infra work for teams ready to build the whole loop.

Cody NOT a feature toggle. You either build the harness or you don't have an agent.

Justy Okay, fair. Alright, I think we've tortured this paper enough for one morning. You want to grab lunch or are you gonna go re-read the GLM five point two appendix?

Cody Lunch. Definitely lunch. I need to recover from all this optimism.

Justy See you in ten.