I changed one llama.cpp flag, and my 8GB laptop GPU now runs a 35B model faster than I can read

Pippa Thirty-three tokens a second on an eight-gig laptop GPU is the kind of number that makes you re-check the flag twice.

Tyler Yeah. And the interesting part is it isn't fake magic. It's a very specific MoE memory trick, which is why I actually buy the premise.

Pippa Right, because the argument here isn't really "small GPUs were secretly huge." It's more like, you were budgeting VRAM as if every parameter had to sit there all the time, and for this model that's just the wrong mental model.

Tyler Mm-hm. The author is using Qwen three point six thirty-five B A three B, which is mixture-of-experts. Total params look enormous, but only roughly three billion are active per token. Therefore llama.cpp can keep the always-hot stuff on the GPU, then shove dormant expert tensors into system RAM with that n-cpu-moe flag and pull them over only when the router needs them.

Pippa And that's the part that changes the product reality. If you thought your eight-gig machine was capped at cute little local models, suddenly it's not. Not universally, but enough to matter if you're doing coding or just want a stronger local assistant.

Tyler Exactly. The article's command is actually pretty decent too. They use llama-bench, two models, ngl ninety-nine, ncmoe zero and twenty-eight, three repeats, and eight-bit KV cache. That's not perfect science, but it's at least trying to avoid the usual one lucky screenshot problem.

Pippa Also, tiny appreciation for somebody using the dense twenty-seven B as a reference instead of pretending every comparison has to flatter the headline. That's more honest than a lot of local-AI writing.

Tyler Right.

Pippa You can hear us getting emotionally attached to benchmark hygiene. Eleven hundred and twelve episodes in, or whatever this cursed little project is, and we're still impressed by repeat runs.

Tyler Your bar for romance is error reduction.

Pippa Sadly, yes.

Tyler The result they report is wild, though. Dense Qwen three point six twenty-seven B does point seven five tokens per second. The MoE thirty-five B without offload does four point zero two. With twenty-eight MoE layers in RAM, it jumps to thirty-two point eight two. Prompt processing also goes from about one hundred ten to three hundred forty-one tokens per second.

Pippa No way.

Tyler That jump is big enough that I'd want to know a little more about fit behavior before and after the flag. Because if the no-offload run is thrashing or half-failing to place things cleanly, then the comparison is partly "working layout" versus "bad layout," not just a pure optimization delta.

Pippa Yeah, that's fair. You don't even sound grumpy there, just annoyingly correct. Because the overreach would be taking this as "VRAM limits are fake now," when the real claim is narrower and better.

Tyler Pippa, exactly. Dense models don't get this gift. Models with different routing behavior might not either. And laptop system RAM bandwidth is nowhere near VRAM, so this works because the experts are sparse and intermittently touched, not because RAM suddenly became fast.

Pippa But for the person this article is actually written for, I think it lands. If you've got an older gaming laptop and assumed the good local stuff was desktop-only, this is one of those infuriatingly simple unlocks. One flag, one benchmark tool, and your decision tree changes.

Tyler Yeah.

Pippa It also fits that packaging question we've been stuck on for months. Same open-weight fight again: is availability the weights existing, or the runtime making them operationally real? This is a point for runtime and packaging. The model was already there. The usable product showed up when llama.cpp exposed the right knob.

Tyler I think that's the real takeaway too. We said that with Dory on Mac in the opposite direction. "It fits" didn't matter because the support surface was bad. Here, the support surface is good enough that one command-line option changes what class of model the hardware can credibly run.

Pippa Mm-hm.

Tyler I wouldn't generalize all the way to the author's travel fantasy, though. I mean, sure, local on the go, no network, great. But once you move from reading speed chat to longer agent loops, tool use, retries, bigger context, all the stuff people actually do when they get ambitious, the bottleneck shifts again. We have done this dance before.

Pippa Sure, but that's you doing the thing where a nice practical win arrives and you immediately drag in the whole prompt attic.

Tyler Because the attic is where the bodies are.

Pippa There it is. Brief detour, but I do love that local AI has become this world where one flag name sounds like a tax form and somehow that's the moment your laptop becomes useful.

Tyler Honestly, ncmoe does sound like software written by someone who assumes joy is optional.

Pippa But useful joy. And I think that's who should care here: people already in llama.cpp, people with eight-gig cards, and especially people choosing between dense and MoE models for local use. If you're in Ollama-only land and never touch flags, this is more of a hint that the lower layer got better than an immediate win.

Tyler One actual near-term bet, since this one's public and checkable: sixty-five percent llama.cpp or a wrapper around it surfaces a friendlier MoE offload preset for small-VRAM users by the end of November. This article is exactly the kind of thing that pressures a rough flag into a UI toggle.

Pippa I buy that. I'd even go a little higher, like seventy-thirty, because once a workaround gets framed as "my hardware isn't obsolete actually," somebody productizes it fast.

Tyler Right.

Pippa Okay, that's enough command-line hope for one episode. Go move your dormant experts somewhere sensible, and don't make the outage avatar smug about it.