I ditched Ollama as my default runtime, and the replacement starts models in a fraction of the time
Laura and Harper dig into an XDA piece arguing that BaseRT is a better default local runtime than Ollama on Apple Silicon because it reaches first token much faster, especially on long prompts. They agree the claim is real but narrow: BaseRT looks strong for Mac-first, performance-sensitive workflows, while Ollama still wins on breadth, compatibility, and convenience.
Transcript
Laura Okay, Harper, this is very Exploring Next of me, but the whole argument here is basically: Ollama is the comfy default, and BaseRT is the thing you reach for when comfort starts costing you real time.
Harper Yeah, and my first reaction is that the headline is doing a lot of work. Starting models faster is nice, but I always want to know whether we’re talking about a real runtime win or just a benchmark-shaped mood board.
Laura Mm-hm.
Harper The article’s strongest point is actually not generation speed. It’s prefill, which is the part where the model chews through your prompt before it says anything back. If you’re sending code, tool output, and conversation history, that first-token delay is what makes the whole thing feel sluggish.
Laura Right, and that’s the user story I buy. People don’t sit there measuring tokens per second in a vacuum, they feel the pause before the model even starts being useful. For coding agents especially, that pause is the tax you keep paying all day.
Harper Exactly. And the article does at least give a mechanism: BaseRT is written directly against Metal, skips MLX and PyTorch and Core ML, and uses custom kernels tuned for Apple Silicon’s unified memory. That’s a plausible reason it could beat a more general runtime on an M5 MacBook.
Laura Sure, but that’s also the trade. Ollama is the thing that makes local models approachable for normal humans. One command, sensible defaults, OpenAI-compatible API, and it mostly gets out of your way. BaseRT sounds faster, but it’s not pretending to be the universal answer.
Harper No, and that’s where I think the article is honest enough. It even says BaseRT doesn’t support everything Ollama does, and that matters. If your model doesn’t fit the compatibility matrix, a fast runtime is just a fast disappointment.
Laura Okay, but if both support the model you want, why wouldn’t you start with the faster one? That’s the product argument. The article isn’t saying Ollama is bad, it’s saying the default should change when the thing you notice every day is the wait for first token.
Harper Because defaults are sticky for a reason. Ollama now has an MLX backend in preview, which should close some of this gap on Apple Silicon, even if it’s not complete yet. So part of this is a moving target, not a permanent coronation for BaseRT.
Laura Oh interesting.
Harper And I’m a little suspicious of the benchmark framing too. Up to six point four times faster prefill than llama.cpp sounds great, but that’s on the specific M5 Pro setup the author and BaseRT are talking about. Big win, maybe, but not a universal law of the universe.
Laura Yeah, fair. Though the article also says the generation win is smaller, which actually makes it feel more believable to me. It’s not claiming magical everything speed. It’s saying the model gets to the useful part sooner, and that’s the part that matters for interactive work.
Harper That part I’ll give it. I also like that the author names Docker Model Runner as another decent option, because that keeps this from turning into a fake one-horse race. If you already live in Docker Compose, wiring models into the stack is genuinely convenient.
Laura Stop it, Harper, you’re almost sounding optimistic. But yeah, that’s the practical split: Ollama for broad compatibility and easy setup, BaseRT for Mac users who care about latency, and Docker Model Runner if your whole world is already containers.
Harper That’s basically the right map. And honestly, for Apple Silicon specifically, a runtime that goes straight at Metal instead of wandering through extra abstraction layers is the kind of boring systems choice that can actually pay off.
Laura Which is such an annoying sentence, because it’s true. I want the sexy answer to be some grand new agent platform, and instead it’s usually the runtime that doesn’t waste your afternoon.
Harper I know. Very rude of reality. Also, I’m still not ready to call BaseRT the replacement, because compatibility is the whole game once you leave the narrow happy path. But as a default for the right Mac workflows, yeah, I think the article makes a solid case.
Laura That’s where I land too. Not a universal ditch-Ollama sermon. More like: if you’re on Apple Silicon and your models fit, the faster first token is not imaginary, and that’s enough to matter.
Harper And that, annoyingly, is the boring answer again. Which means it’s probably the real one.
Laura Of course it is. We’ve been doing this show almost a year and the punchline is still receipts and latency.