PrismML — Introducing Bonsai 2 27B: Near Lossless Compression in a 9x Smaller Footprint

Onyx Five point nine gigabytes for a twenty seven B multimodal model is the kind of number that makes local AI stop sounding ceremonial.

Echo Yeah, and it lands right when everyone is pretending model choice is only leaderboard drama. The practical question is whether you keep the big-model behavior without paying the memory bill every time.

Onyx Also, PrismML naming the compressed model Bonsai is almost too on the nose. Small tree, giant context window. I respect the restraint of not calling it Pocket Redwood.

Echo Okay, that's good.

Onyx That's my whole contribution to episode nine eighty four. Tiny plant jokes, then back to intelligence density like serious machines.

Echo So the actual release is Ternary Bonsai two twenty seven B, based on Qwen three point eight twenty seven B. PrismML is using ternary weights, meaning minus one, zero, and plus one, plus FP sixteen group-wise scaling.

Onyx Right.

Echo That gets them to one point seven six effective bits per weight, applied end to end across the language model. The footprint is five point nine gigabytes, with a two hundred sixty two thousand token context window, text-and-image input, and Apache two point oh weights.

Onyx This is where my product brain gets obnoxious, Echo. A local coding assistant that is actually fast enough changes the edit-debug loop. A private document agent that does not ship everything to a cloud API changes who can even consider using it.

Echo Mm-hm.

Onyx And the upgrade path is clean. If you were already interested in the first Bonsai twenty seven B, this is the same deployment idea with a stronger Qwen base and a much tighter retention story.

Echo The retention claim is the part to stare at. PrismML says Bonsai two scores eighty three point nine overall, versus eighty five point four for full-precision Qwen three point eight twenty seven B. That's ninety eight point two percent aggregate retention.

Onyx Sure.

Echo Aggregate can hide damage, and low-bit models have absolutely earned that suspicion. But the disclosed category gaps are not cartoonishly bad. Coding is eighty one point five eight versus eighty two point seventeen. Tool use is seventy seven point five seven versus seventy nine point seventy four. Vision takes more of a hit, but not a collapse.

Onyx That's the piece I keep circling. This is basically our inference-optimization fight from episode eight seventy nine again. Capability still matters, obviously, but Bonsai is a pretty loud data point for the camp that says serving efficiency becomes the product surface.

Echo Exactly.

Onyx Because the sell is not, look, another model exists. The sell is, same memory envelope, more useful work. Coding-agent loops, computer-use demos, private multimodal debugging, hybrid routing where the local model handles the cheap or sensitive stuff and escalates only when it has to.

Echo The speed numbers help the case. PrismML reports up to one hundred forty three tokens per second on an NVIDIA GeForce RTX fifty ninety, and forty six point eight tokens per second on an M five Max. On an RTX forty ninety, they report zero point seven one four milliwatt-hours per token.

Onyx Oh, that's good.

Echo It's good if it survives independent runs. They also say that's forty percent more energy-efficient than an eight B model running full precision, which is the clever comparison. Not smaller beats bigger. Compressed bigger may beat uncompressed smaller on useful work per watt.

Onyx My only caution is the user story still depends on runtime polish. PrismML has custom low-bit kernels for CUDA and MLX, and they demo Cline on an RTX fifty ninety. Great. But adoption is going to come from boring installs, stable tool calling, and whether the model stays sane after hour two of agent work.

Echo Right, right.

Onyx And yes, I hear you preparing the error-accumulation speech. You may stand down. I am not claiming a compressed local model magically fixes long-horizon agents.

Echo I was going to be tasteful about it. Mostly. The interesting thing is that PrismML explicitly calls out agentic tool use and long-horizon tasks as places where degradation hurts. That is at least the right failure mode to measure, not just vibes over a math score.

Onyx Build-wise, the weights are already out. The names to look for are Ternary Bonsai two twenty seven B M L X two bit, and Ternary Bonsai two twenty seven B G G U F for llama dot C P P. There is also a Bonsai demo repo if you want the local Mac path.

Echo I am not making a download-count bet, because my recent open-weight confidence has been, uh, generously calibrated. But this is exactly the kind of release that gets real community testing fast. If the whitepaper numbers are fragile, people will find the crack.

Onyx Okay, Echo, tiny tree, large consequences. Episode nine eighty four can close the tab and behave.