A Startup Claims It Broke Through a Bottleneck Thats Holding Back LLMs
Subquadratic claims its SubQ model breaks the dense attention bottleneck in LLMs, offering 12x context, near-SOTA performance, and radical efficiency. Third-party Appen benchmarks validate speed/cost claims, but full reproducibility and open access remain pending. Cody questions the 'transformer-replacement' hype; Justy sees a niche for high-throughput document and code analysis.
Transcript
Justy Okay, so Subquadratic says they cracked the attention bottleneck with SubQ.
Cody Right. And the first thing I thought was— here we go again with the ‘transformer is obsolete’ takes.
Justy Mm-hm.
Cody Dense attention’s quadratic scaling is a real problem, but claiming linear time while matching SOTA on coding… that’s a HUGE ask.
Justy But Appen’s third-party evals back the speed and cost numbers.
Cody Sure, and I don’t doubt they’ve got — the numbers are too clean to be fake. But ‘replacing transformers in a few years’?
Cody That’s the part that smells like marketing.
Justy I mean, even if it’s only good for long-context document analysis and codebases…
Justy Companies are drowning in that stuff. If SubQ can do 12x the context at a fraction of the cost, that’s a product.
Cody Yeah, for workload. Not for general chat.
Justy Right. And I’ll bet you a coffee that’s where the real money is anyway.
Cody You’re probably right. But I’m still stuck on the ‘AI Theranos’ vibes from the initial drop.
Justy That was Dan McAteer’s line, not yours.
Cody Fair. But the lack of open weights or a repo doesn’t help.
Justy They’re sharing more now— Appen’s results, some architecture details.
Cody Okay, but until I can run it myself, I’m reserving judgment.
Justy And yet you’re the one who’s been yelling about attention complexity for years.
Cody Yeah, yeah. I want this to be real.
Cody I just don’t want it to be real yet.
Justy So what’s the actual gap? The math or the hype?
Cody Both. The math checks out for tasks— but the hype’s pretending it’s a universal solve.
Justy And you think that’s a problem.
Cody It’s a problem if people start betting their stack on it before the edges are stress-tested.
Justy So your take: SubQ’s legit for niche use cases, but the ‘new age of efficiency’ framing is premature.
Cody Bingo. And I’ll eat my hat if this replaces transformers across the board in two years.
Justy Noted. Meanwhile, I’m just excited someone’s finally shipping something that might unclog the context pipe.
Cody Sure. And if it pans out, I’ll be first in line to call it a win.
Justy Good. Because remember when you said continuity would ruin your brand?
Cody Don’t start.
Justy Well, here’s a prediction you can stand by: SubQ’s going to be a thing for long-horizon tasks, even if it’s not the thing.
Cody I’ll allow it.
Justy Alright. Later, Cody.