Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning

Pippa Okay, so this one is—I've been sitting with it since yesterday and it keeps landing differently every time I read it. Code-as-World. They're representing physical worlds as executable code.

Tyler Yeah.

Pippa Not embeddings, not latent features, not 'the model understands gravity.' Actual code. Objects with state, dynamics, rendering. You run it, you get pixels, you check if those pixels match the video.

Tyler Right, so the inversion is: instead of trying to extract the mechanism from pixels, you propose mechanisms forward and verify they produce the pixels you actually see.

Pippa Exactly. And then—this is the part that got me—they use those verified worlds as training data for a vision-language model to do quantitative physical reasoning.

Tyler Mm-hm.

Pippa So the model learns to infer measurable things from video: size, velocity, displacement. Not 'this ball is rolling' but 'this ball is moving at 3.2 meters per second.'

Tyler Okay, and that distinction matters because semantic understanding—'rolling'—doesn't require you to ground anything in metric space. But if you want to predict how the world responds to an intervention, you need the numbers.

Pippa Right. So the mechanism-as-code is doing two things at once: it's a compact representation—objects, composition, dynamics, all explicit—and it's a verification target. If your world hypothesis produces pixels that don't match the video, you know the hypothesis is wrong.

Tyler Mm.

Pippa The agent loops: propose a world in code, instantiate it, run it in a simulator, render it, compare to the evidence. If it doesn't match, the agent refines the hypothesis. Propose-execute-render-verify.

Tyler That's abductive reasoning. You're searching for the simplest mechanism that explains the observations. Newton's laws, heliocentric theory—they cite this pattern explicitly.

Pippa Yes. And the reason code is the right substrate is that it's executable. The agent doesn't have to guess whether the hypothesis is coherent; it just runs it.

Tyler Okay, but I want to poke at the discovery loop itself. How does the agent actually propose refinements? If the rendered world doesn't match the video, how does it know which parameter to tweak?

Pippa That's the question I had too. They use a language model to interpret the discrepancy and suggest the next hypothesis. So the agent reads the video, proposes an initial world, executes it, compares the render to the video, and if there's a mismatch, the LLM looks at both and suggests what to change.

Tyler So it's not a gradient-based search through parameter space. It's LLM-guided discrete proposals.

Pippa Right. Which means the bottleneck is the LLM's ability to reason about what went wrong. If the LLM misdiagnoses the discrepancy, you're going down a wrong branch. But that's the product insight, right? You're not trying to solve the inverse problem end-to-end. You solve it once offline—discover the verified worlds—and then use those as ground truth for supervised learning.

Tyler Which is scalable as long as the discovery loop converges reliably and the scenarios don't get too complex.

Pippa Yeah, that's the frontier. The paper tests on relatively constrained domains—simple rigid-body dynamics, known object types. What happens when you move to deformable objects, occlusion, multi-agent interactions? The discovery loop probably explodes.

Tyler More degrees of freedom, more ways for the LLM to misdiagnose.

Pippa They acknowledge this as a limitation. But what I'm struck by is the framing: they're not trying to build a general physical simulator. They're building a representation that's compact, compositional, and verifiable. Code is doing all three.

Tyler And the benchmarks—QuantiPhy is real, and outperforming Gemini-3.1-Flash on a specific physics reasoning task is non-trivial. But it's not clear how far this generalizes beyond measurement inference.

Pippa Right. This is a deep-dive episode, not a breakthrough signal. The mechanism is sound, the results on the chosen benchmark are strong, but the scope is narrow and the brittleness of the discovery loop is real.

Tyler Mm-hm.

Pippa That said, if you're building a physical reasoning system and you care about grounding in metric space, this is a concrete approach. You don't have to solve inverse physics online. You solve it offline, verify, and then supervise.

Tyler It's a good inversion of the usual pipeline. Most work assumes the model has to learn physics from pixels directly. This says: no, build the physics forward, verify it matches the pixels, use that as training signal.

Pippa And code—actual, executable code—turns out to be the right substrate for that verification because the agent can instantiate and run it. Not just reason about it abstractly.

Tyler Yeah, that's the elegant part. The executable nature of code enables the feedback loop.

Pippa Okay. So if you want to get hands-on with this: they've open-sourced the code. GitHub repo is called code-as-world, it's at mirros-lab. The project page has the datasets, the executable world examples, and the QuantiPhy evaluation setup.

Tyler And the blog post walks through the loop with concrete examples?

Pippa Yeah, the blog is the accessible entry point. The paper is dense but worth reading if you care about the discovery loop mechanics or the ablations.

Tyler Alright. This one lands as: elegant inversion, real results on a narrow benchmark, genuine brittleness on generalization. Worth implementing if your domain is constrained and you need metric grounding.

Pippa And if you're thinking about how to supervise physical reasoning models at scale, this is a concrete answer to that question.