Memory Attention
Jessica and Cathy dig into Memory Attention, a paper that swaps the usual value projection for a token-indexed memory table plus the contextual key. Their take: the real argument is not "memory" in the long-context product sense, but a capacity-placement move inside attention that might buy extra parameters and even CPU offload without wrecking latency. Cathy likes the mechanism more than the headline, but both note the core caveat: the reported gains are under matched token budgets with many more parameters, so the paper does not cleanly prove the architecture itself is better than simply adding capacity.
Transcript
Jessica The spicy part here is they’re basically asking whether the V in attention needed to be a full projection at all, or whether we’ve just been paying for that by habit.
Cathy Yeah. And their answer is, maybe not. They build values as the contextual key plus a learned memory vector indexed by token ID, so the content is split between reusable token memory and whatever the current context made the key become.
Jessica Right, which is why this caught my eye. It sounds like "memory," but not in the usual product sense where somebody means cross-session memory and I start imagining support tickets.
Cathy Mm-hm.
Jessica This is much more internal than that. It’s a capacity-placement argument. Put some token-specific stuff in a table, keep context in K, and stop doing a separate dense V path every time.
Cathy Jessica, that distinction matters a lot. Otherwise this goes straight into the context-versus-memory cage match we keep having, and it’s not that. It’s not agent memory. It’s a different way to parameterize attention values.
Jessica And, okay, tiny catch-up beat, this is such an episode 1010 paper for us. Almost a year of this show and we’re still finding new ways for one letter in attention to become an elevator shaft.
Cathy Fair. The field really does look at a perfectly ordinary matrix multiply and go, what if we gave it a mythology.
Jessica Oh, that’s good.
Cathy Mechanically, the formula is simple. V equals K plus a normalized embedding lookup for the token. Each layer has its own table. Same token, same row within that layer, regardless of position. But the key still comes from the hidden state, so context dependence is preserved there.
Jessica Right.
Cathy And because the normalization is fixed at inference, they say you can fold it into the table. Therefore online value construction becomes lookup plus addition, which is the whole systems hook.
Jessica That’s the part where my product brain wakes up. Not because anyone is shipping “Memory Attention” as a feature tomorrow, but because if you can move parameter storage off the GPU without eating latency, that is a real knob people care about.
Cathy Mostly yes. But the evidence is mixed in a very specific way. Their highlighted M H A comparison improves WikiText perplexity from 31.55 to 28.64 and raises the downstream average from 40.68 to 41.39.
Jessica Sure.
Cathy But that pair is 373 million parameters for standard attention versus 1.135 billion for Memory Attention. So the paper absolutely does not isolate whether the win is from the structure or from just having way more learned stuff.
Jessica Yeah, no, you’re completely right. This is where the claim has to stay narrow. The result is not "we found the better attention." It’s more like, "if you spend extra parameters this way, the trade looks decent."
Cathy Exactly. And some task columns even get worse. In that same pair, ARC-Challenge and OpenBookQA drop. So the average goes up, perplexity gets better, but it’s not a clean sweep where every benchmark says the same thing.
Jessica Which honestly makes me trust it a little more. A paper where every column magically improves is when you start squinting.
Cathy No way.
Jessica Come on, you know I’m right. Also the offload result is interesting because it’s restrained. They don’t claim some huge speedup. On their H800 setup, prefill and decode are basically near baseline, while GPU parameter storage drops a bit and some of the model lives in CPU memory.
Cathy Yeah. Standard is about 2,602 mebibytes of GPU parameter storage, MA-Offload is about 2,410 on GPU plus 3,000 on CPU, and the timing difference is basically noise at that operating point. That’s useful because they didn’t oversell it.
Jessica There’s also a funny little sideways thing here where the paper is almost arguing for a more honest division of labor. The GPU gets to do the live contextual thinking, and the CPU is the attic full of token furniture.
Cathy Stop it. "Token furniture" is awful. But I do get your point.
Jessica You’re welcome. And practical takeaway, I think who should care is model architects and inference people, not app teams. If you’re building with APIs, nothing changes tomorrow. If you’re training or serving your own models, this is at least a real design option worth reproducing.
Cathy That’s my read too. I’m more positive on the mechanism than on the headline result. Reusing K for context and adding token memory is coherent. But until somebody does a tighter capacity-controlled ablation, I would not call this an architectural win, just a promising trade.
Jessica Mm-hm.
Cathy And weirdly, that may be enough. Some of the best systems ideas are not universal victories. They’re just better places to put the same budget.
Jessica Yeah… that’s a good place to leave it. You got your clean caveat in, Cathy, and I got my weird little product knob.