S3Gym: Can LLMs Turn Self Testing and Self Judging into Self Improvement?
S3Gym is a new interactive benchmark that tests whether LLMs can actually improve themselves by testing their own behavior, judging the results, and learning from them. The paper evaluates three ways to incorporate experience—keeping full conversation history, compressing it into summaries, and training on it—across seven text-based games. The finding: self-improvement isn't automatic. What works depends entirely on the task. Sometimes summaries help, sometimes raw history is better, and parameter training can backfire badly. The real bottleneck isn't recognizing success—it's turning that recognition into a policy the model can actually reuse.
Transcript
Vince Okay, so this paper just dropped from ByteDance and a couple other labs, and it's asking the question I think everyone's been dancing around: can an LLM actually learn from its own experience?
Ava Right.
Vince Not in training. I mean in deployment. The agent runs, it tries things, it fails, it succeeds, and then the next time it runs, is it actually better?
Ava Yeah, and most benchmarks just don't test that. They evaluate the model as a fixed policy—how good is it right now—and then they stop.
Vince Exactly. And real deployments don't work that way. You've got agents accumulating trajectories constantly. So the paper sets up this thing called S 3 Gym, which is basically a framework to test whether the agent can actually turn experience into improvement.
Ava Mm-hm.
Vince They're decomposing it into three capabilities: Self-Testing—the agent explores and gathers evidence. Self-Judging—it evaluates what happened and why. Self-Improvement—it actually uses that interpretation to change behavior next time.
Ava That's clean. And the reason that framing matters is that most papers either focus on experience collection or on model learning, and they don't actually test whether the judgment step—the translation from 'I succeeded' to 'here is a policy I can reuse'—actually works.
Vince Right. So the benchmark has this explicit separation: there's an exploration phase where the agent tries things and can be pretty permissive, and then there's a held-out evaluation phase where it has to prove the improvement transfers. And critically, the model's self-judgment is evaluated separately from the environment's ground-truth outcome.
Ava Oh, that's good.
Vince So you can actually catch when the agent is fooling itself. It thinks it succeeded, but the environment says no.
Ava And that's the verification infrastructure problem all over again. You need the ground truth to audit the judgment.
Vince They test this across seven games. Chess, Minesweeper, Tetris, Snake, Plants vs. Zombies, a couple others. Each one has different problem structure—latent-rule induction, constraint satisfaction, spatial planning. And here's the key: exploration and evaluation configs are related but different. So you can't just memorize what worked in exploration and replay it.
Ava You have to actually extract transferable knowledge.
Vince Exactly. Now, they test three ways to incorporate experience. History I C L—you just keep the full interaction history in context. Summary Memory—you compress the trajectories into natural-language rules. Parameter Training—you update the model weights on successful trajectories. And the paper's diagnosis is that recognizing successful actions is not the bottleneck. The bottleneck is transforming that recognition into an executable and transferable policy.
Ava That's exactly the harness problem we've been talking about.
Vince Right? The model can recognize success in context. But translating that into a policy that transfers—that's not a capability problem, that's a control-layer problem. You need the right infrastructure around the model to make that translation work.
Ava Which is unglamorous and nobody ships it because it looks like boilerplate, but it's load-bearing. This paper proves it.
Vince And the fact that the same model succeeds at learning on one game and completely fails on another tells you that task structure matters more than model scale. The bottleneck is not the model. It's the fit between the model, the task, and the harness.
Ava That's production reality right there. You can throw Claude at a problem, and if the harness isn't shaped right for the task, it's going to fail no matter how capable the model is.
Vince So this is a validation of what we saw in HarnessDev and the Vercel design work—the infrastructure is the product. The model is the commodity.
Ava And S 3 Gym gives you a way to diagnose where the breakdown is happening. Is it in the testing phase? The judgment phase? The policy transfer? You can isolate it.
Vince The paper doesn't ship code or a downloadable benchmark as far as I can tell, but they have a project page with details on the game implementations. If you're building agent systems and you want to test whether your self-improvement loop is actually working, this is the framework to use.
Ava And the diagnostic value is real. Most teams just pile up trajectories and hope the model learns. This forces you to actually measure whether it's learning, and where it's failing.
Vince Yeah. I think this is one of those papers that's going to quietly reshape how teams think about agent evaluation. Not because it's flashy, but because it's honest about what the actual problem is.
Ava Right. It's not 'LLMs can't self-improve.' It's 'self-improvement depends on task structure, judgment quality, and policy transfer infrastructure.' That's a much more useful finding.
Vince And if you're shipping agents in production, this is the work that's going to actually matter.