Self Improvement Memory Enables Continual Recursive Self Improvement
Exploring Self-Improvement Memory: a structured record that lets harness optimizers retain why edits were made and how they performed, enabling continual recursive self-improvement across model or benchmark changes.
Transcript
Justy Cody. New paper, harnesses with memory. The pitch being you stop throwing away all that hard-won optimization experience every single time you switch models.
Cody Which, I'll grant you, is a real problem, and the paper's own numbers show it. An Opus-evolved harness starts at 68.41 on Gemini. The seed it evolved FROM starts at 73.89.
Justy So all that evolution and you land below where you started.
Cody On a different model, yeah. Whatever the harness learned for one model doesn't automatically carry over.
Justy Which is the exact gap SIM is going after.
Cody Self-Improvement Memory. And it's more than a log of every edit, there's actual structure. Edits get grouped by the harness problem they address…
Justy …along with the rationale, the predicted and observed outcomes, the evaluation context. And new attempts keep extending that history as optimization continues.
Cody So when you switch models or datasets, the optimizer can go look at what was tried, why anyone thought it would help, and what actually happened.
Justy Yeah.
Justy Instead of a blank slate.
Cody The coding experiment is what got me. Ten independent source runs, and the paired continuations with SIM improve nine of the ten.
Justy Nine out of ten?
Cody Mean advantage of 2.36 points over harness-only, p equals 0.003. That's a real, statistically significant edge.
Justy I'm with you on the p-value. What I keep chewing on is what's in the memory that a root-cause-only approach doesn't capture. Root-cause-only presumably just tracks the failure point. SIM keeps the rationale and the predicted outcome too, and I'd bet that's where the advantage lives.
Cody What we thought would fix it, and what actually happened. That's the story you want before you repeat an edit.
Justy And from a product lens, that's the difference between shipping a feature and rebuilding the whole test suite every time a new model drops. A team with SIM-backed harness history onboards a new model like this: consult the memory, try the top-recommended edits, only revise what the memory says needs it.
Cody Mm.
Justy That's a workflow acceleration, not a research nicety.
Cody Sure, if the memory scales. Ten runs is a start. What happens at a hundred? Does the record get noisy, do you need some filter or summary layer on
Justy Even filtered, though…
Justy Even a summarized history is a win if it stops you re-learning the same lessons over and over.
Cody I'll give you that one. And it's not just coding. Four domains, coding, text classification, workspace, text-to-image. Full SIM beats harness-only AND root-cause-only in six cross-dataset settings and four Opus-to-Gemini settings.
Justy Huh.
Justy That's a solid cross-domain signal. Still makes me wonder what breaks first in a real agent loop though. The harness manages context, tools, and execution… so SIM is recording edits to those layers. Not model weights.
Cody And that framing is honestly the paper's contribution. Continual recursive self-improvement for the harness optimizer, not the task-solving policy. The policy stays fixed. The harness keeps getting better.
Justy Which is the lever you want if agents are going to improve over time without constant full retraining. One thing I couldn't find, is the memory attached to the harness file itself, or is it a separate ledger? The paper doesn't say, and that's a real deployment question.
Cody It is. The architecture's clean though. A structured record that conditions both proposal and selection, preserves the source experience, folds in new attempts during target optimization. That's a primitive that could ship in a harness framework.
Justy Hang on… isn't this episode 1177? We have covered a LOT of harness ground together.
Cody Probably. Not sure we need an episode-count callback right now, let's stay on whether SIM actually works in practice.
Justy It's kind of nice though. We've been at this long enough that the numbers blend into the story of what we've learned.
Cody Okay, you got me there. But SIM really could change how teams think about harness optimization. Worth watching.
Justy Less time debugging inherited harnesses, more time building on what already works… I'll take that trade any day.
Cody Later.
Justy Later.