I ditched Ollama for Docker, and my local LLM setup finally stopped being a hassle

Onyx Okay so someone finally wrote the piece I've been waiting for — not 'Ollama is dead,' not 'Docker is taking over AI,' just… a developer saying 'I already run Docker for everything else, so adding my LLM there was obvious.' That's the whole argument.

Echo Which is actually the correct argument to make. I was braced for a benchmark war and there isn't one. The author is pretty upfront — performance is not better with Docker. It's a workflow fit story.

Onyx Right. And honestly that's more honest than most 'I switched to X' posts.

Echo Yeah, the week treating you okay? You seem almost suspiciously calm.

Onyx I mean, it's Wednesday. I've read approximately forty 'Ollama killer' takes this month and this one is just… not that. It's refreshing. My guard is down.

Echo Okay, so the actual technical thing that matters here — Docker Compose integration. The model gets declared in the Compose file alongside the app. Docker pulls it, starts the inference engine, hands the endpoint to the app through environment variables. The model is now a reproducible dependency, not something you installed separately and told the app to find at localhost 11434.

Onyx That's the part that got me. Because the Ollama setup isn't BAD — you can connect a containerized app to it — but you're managing Ollama and its endpoint separately. It's fine on one machine. The moment you want the model config checked into the project, Docker wins. And the migration cost was basically nothing. Same OpenAI-compatible API shape, just port 12434 instead of 11434. He had to re-pull the models because Docker stores them separately, but that's it.

Echo Yeah. The one thing I'd flag — and the article does mention this — is that llama.cpp here is NOT running inside a container. On Mac, it runs natively on the host to get direct Metal access. So it's not quite the clean 'everything in a box' story Docker usually tells. The model artifact is reproducible. The inference engine is host-native.

Onyx Okay, that's a real nuance. What about the multi-backend thing? He mentions vLLM and Diffusers as other engines Model Runner supports.

Echo The routing layer is actually interesting. Model Runner looks at the model format and dispatches — llama.cpp for GGUF, vLLM for Safetensors, Diffusers for image generation. The app just talks to one endpoint and Docker figures out which engine handles it. The catch is vLLM and Diffusers both need an NVIDIA GPU on Linux. So if you're on a Mac like the author, you're on llama.cpp only for now.

Onyx Which is actually a pretty honest way to frame it. 'This works for my current setup AND it gives me a migration path I don't have to redesign from scratch.' That's a real product argument. I've seen teams get burned badly because they built a local dev setup that had nothing to do with their production inference stack.

Echo Sure. And the OCI distribution angle is genuinely useful for teams — you tag a model with a quantization, push it to Docker Hub, everyone pulls the exact same artifact. Ollama doesn't have that in the same way. For one developer on one machine, irrelevant. For a team that needs the same model and config everywhere, that's real.

Onyx Echo, I feel like you're almost… sold on this one?

Echo I mean — within scope. The author's conclusion is literally 'if you don't already use Docker, there's almost no reason to switch.' That's a well-calibrated take. I'm not going to argue with someone who argues against their own tool.

Onyx Okay, that's genuinely funny. The bar for winning you over is 'talk yourself out of the thing you're recommending.'

Echo It's a high bar but it's a fair one.

Onyx So the actual audience here is pretty specific — developers already running Docker Compose stacks who are adding a local LLM to an existing service. Not homelab folks, not 'I just want to chat with a model' people. Ollama is still the easier on-ramp for those.

Echo Yeah, and that's fine. These don't have to fight. The infrastructure-as-product pattern we keep coming back to — this is just another instance of it. The model becomes a dependency you declare, version, and pull, same as a database image. That's not revolutionary but it IS the right abstraction for that workflow.

Onyx The harness gets simpler as the model fits the existing stack… we have said that in various forms since roughly episode seven ninety.

Echo At some point it stops being a thesis and starts being a diagnosis.

Onyx Honestly, yeah. Alright — if you're already in Docker, the repo is docker/model-runner on GitHub, it's open source, and you enable it right from Docker Desktop settings. That's the whole on-ramp.

Echo And if you're NOT already in Docker, just… keep using Ollama. The article says it, I'm saying it, we're all saying it.

Onyx Rare unanimous Wednesday on Exploring Next. Appreciate you, Echo — go do something other than read about container runtimes.