How to Make Your First World Model from Scratch

Onyx Okay, so there's a CartPole tutorial making the rounds, and I think you have a problem with it — which surprised me because honestly I found it kind of great.

Echo I don't have a problem with the tutorial. The tutorial is fine. My problem is with what it's claiming to do versus what it actually does. The headline is 'build your first world model from scratch.' What it actually delivers — at least in the part I can see — is a really clean diagnosis of why DQN fails. Which is good! But those are different things.

Onyx Right.

Echo The core demo is genuinely well-constructed. You train a DQN on CartPole — four-number state, two actions, one point per step survived — it works. Then you ask the same agent to park the cart at a specific position, and it just… doesn't. Not because it breaks. It keeps balancing perfectly. It just has no ear for the new request. And the author's line for this is sharp: the goal is not something the agent has, it's something the agent IS. That's the baked-cake thing.

Onyx And that image earns its keep. Because the evidence is right there — five seeds, identical code, parking reward, five completely different outcomes. One run parks and holds it for all five hundred steps. Two sail off the rail. Two fall over at the target. Same reward function, same architecture.

Echo Yeah, and that variance is the honest part. The article doesn't hide it. The spread is the point — DQN learned which move pays off, not how the cart actually behaves. Those sound similar until you see run three nail it and run five sail into the wall.

Onyx Oh interesting.

Echo But here's where I get skeptical. The jump from 'DQN has no world model' to 'therefore, build one' — the article sets that up beautifully, and then the actual world model construction is where I'd want to see the numbers. The framing is: a world model lets the agent practice in imagination, so you don't need forty-six thousand real steps every time you change your mind. That is a real argument. Model-based RL has decades of serious work behind it.

Onyx Okay but — and I actually think this is the part you're underselling — the physical cost framing is doing real work here. The article explicitly puts step counts on it. Twenty-four thousand steps to park. Forty-six thousand to stay in the center zone. And then it says: put this on a real robot, and every one of those steps is a physical object really falling over. That's not a theoretical concern, Echo. That's the actual budget constraint for anyone doing robotics.

Echo No, you're right, that framing is honest.

Onyx And the reason that lands is because the tutorial earns it. You've watched the DQN flail through those seeds. You've seen the variance. So when it says 'this is the bill in a currency you cannot print' — you feel it. That's not hype, that's the actual argument for why model-based approaches exist at all.

Echo Sure, and I'm not disputing the motivation. Model-based RL is real, the sample efficiency argument is real, Sutton and Barto put the CartPole problem on the map back in eighty-three for exactly this kind of reason — it's old enough that every method gets tested on it first. What I'm saying is: the article promises a world model from scratch, and what I actually see is a very clean on-ramp to understanding WHY you'd want one. Which is valuable!

Onyx Hm.

Echo And honestly, the subtitle is what gets me. 'Letting it daydream its way through CartPole, and accurately measuring when the illusion collapses.' That second clause is doing a lot. Measuring when the illusion collapses is a HARD problem. Learned world models compound errors — you roll out imagined trajectories and the small mistakes stack. The article names this, which I respect. But naming it and solving it are different.

Onyx Okay, that's fair. Though — this connects to something we've been circling since basically episode five-fifty-eight, right? What even counts as a world model? We keep bumping into this. The field hasn't settled on whether it's environment simulation, forward reasoning, or both. And this article is using the term in the 'next-state prediction' sense — which is mechanistically honest. It's not claiming the agent has a rich internal physics engine.

Echo Right, and that's actually the thing I'd give it credit for. The framing isn't 'we gave the agent a world model and now it generalizes.' It's: here's what the agent can't do without one, here's why, here's the cost curve. That's a much more useful starting point than most tutorials that just hand you the working code and skip the diagnosis entirely.

Onyx Stop it — are you coming around on this?

Echo I'm saying the diagnosis is good. I'm still waiting on the payoff. Those are compatible positions, Onyx.

Onyx Okay, okay. So who actually should care about this? Because I think the audience is pretty specific.

Echo Someone who has shipped a standard RL loop — or read about one — and hit the 'I need to change the goal' wall. Or anyone doing robotics who's felt that sample cost in real hardware time. For that person, this is a genuinely useful conceptual unlock. The baked-cake framing alone is worth the read. For someone who already knows model-based RL, there's probably not a lot of new ground.

Onyx Yeah. And I'd add: anyone who's been handed the phrase 'world model' in a paper or a pitch deck and nodded along without quite knowing what the mechanism is. This gives you the mechanism at the level of 'here is what the agent actually has and doesn't have.' That's not nothing.

Echo It's not nothing. I just want part two to exist before I call the promise fulfilled. The subtitle says 'accurately measuring when the illusion collapses' — I want to see that measurement. That's the hard part, and the article sets it up beautifully without yet delivering it.

Onyx Which, honestly, is a very Echo thing to say about a tutorial called 'your first world model.' You want the fifth world model.

Echo I want the second one. The first one that actually daydreams and then we check whether the dream was right. That's not a high bar.

Onyx Yeah… okay, that's fair. Where we land: the diagnosis is real and well-evidenced. The five-seed variance plot is the article's best asset — it shows mechanistically that DQN learned a policy, not physics. The motivation for world models is honest and grounded in actual cost curves. The payoff is still TBD, which the article might be upfront about — this might genuinely be part one of a series.

Echo If it is, I'll read part two. That's not a low bar from me.

Onyx There it is. The repo's linked in the show notes if you want to run the seeds yourself — the author flags every number came from a real script on one computer, written to a file, which is the kind of receipts I like. Go check whether run three parks and run five sails. See you next one.