One weekend, one M4 Pro with 48GB, five 4-bit local models, eight benchmarks. I wanted to move repetitive work off the cloud, save tokens, and find out what "runs offline" actually costs in quality. Here's the full benchmark, the final stack, and the three speculative-decoding traps I hit along the way.
Why move models back on-device
Frontier cloud models earn their keep on writing and hard reasoning. But the real bill comes from repetitive, mechanical, privacy-sensitive grunt work: structured JSON extraction, named-entity recognition (NER), classification, sensitive-data de-identification, literature triage. None of it needs frontier brainpower. The volume burns money anyway, and PII-class data should never leave the machine in the first place.
So I set out to build an offline local stack of "one high-end model + two small models running concurrently", push all repetitive tasks local, and only send genuine reasoning and final polish back to the cloud.
Hardware: one M4 Pro, 48GB unified memory; inference served by oMLX on local :8090. The five models under test (all MLX-quantized):
- Qwen3.6-27B (4-bit, ~15GB) -- quality-ceiling candidate
- Qwen3.6-35B-A3B (4-bit MoE, ~20GB, ~3B active) -- speed candidate
- Qwen3.5-9B (4-bit, 5.82GB) -- extraction workhorse candidate
- gemma-4-12b-coder (8-bit) -- coding-specialized
- gemma-4-E4B (4-bit, ~5GB) -- lightweight laborer
The scorecard: eight benchmarks, cross-compared
Each benchmark sampled ~30 items, thinking disabled, for relative ordering only. Caveat up front: with a 30-item sample, score granularity is ~1/30 = 3.3%, so any gap within +/-10% is noise. This is a tiering, not a ranking.
| Benchmark | gemma-12b-coder | gemma-E4B | Qwen3.5-9B | Qwen3.6-27B | Qwen3.6-35B-A3B |
|---|---|---|---|---|---|
| arc_challenge | 0.867 | 0.800 | 0.867 | 0.900 | 0.867 |
| cmmlu (Chinese) | 0.433 | 0.533 | 0.633 | 0.800 | 0.733 |
| hellaswag | 0.800 | 0.667 | 0.867 | 0.833 | 0.833 |
| humaneval | 0.900 | 0.733 | 0.833 | 0.933 | 0.900 |
| livecodebench | 0.167 | 0.333 | 0.133 | 0.467 | 0.500 |
| mathqa | 0.200 | 0.167 | 0.433 | 0.567 | 0.633 |
| mbpp | 0.633 | 0.700 | 0.567 | 0.867 | 0.767 |
| mmlu | 0.667 | 0.567 | 0.733 | 0.700 | 0.633 |
| Composite | 0.583 | 0.563 | 0.633 | 0.758 | 0.733 |
| Total time (8 tasks) | 1257s | 1725s | 2277s | 3266s | 824s |
- Qwen3.6-27B is the quality ceiling: highest composite (0.758), tops in Chinese (0.80) and coding (mbpp 0.867 / humaneval 0.933). Also the slowest; livecodebench alone took 2138 seconds.
- Qwen3.6-35B-A3B is the speed/quality sweet spot: as a MoE (~3B active), its 0.733 composite closes on 27B, yet it's the fastest of the five (824s, nearly 4x faster than 27B) and beats 27B on hard coding (livecodebench 0.50) and math (0.633).
- The small models each know their lane: 9B has the steadiest formatting plus strong Chinese and mmlu; gemma-E4B is a lightweight fallback; gemma-12b-coder's humaneval is strong but collapses on hard problems (livecodebench 0.167).
So... which cloud generation is this?
Two disclaimers before calibration: (1) benchmark numbers are not felt experience; (2) static multiple-choice is where smaller models close the gap most easily, and the real long-context / agentic delta is wider. With that:
| Local model | Benchmark-paper equivalent | Felt equivalent (incl. reasoning/agentic) |
|---|---|---|
| Qwen3.6-35B-A3B (driver) | GPT-4o / Claude 3.5 Sonnet (2024) | Between GPT-4o-mini and Claude 3.5 Haiku; touches GPT-4o's lower edge on code |
| Qwen3.6-27B (fallback) | Slightly above 35B, = GPT-4o on paper | Same, marginally stronger but slower |
| Qwen3.5-9B (extraction) | GPT-4o-mini ~ early GPT-4-turbo on knowledge | Above a beefed-up GPT-3.5-turbo / Llama-3-8B; strong Chinese knowledge and stable JSON |
| gemma-4-E4B (laborer) | GPT-3.5-turbo / Claude 3 Haiku class | Same tier; value is throughput, not smarts |
The ceiling of this local stack matches 2024-era GPT-4o on paper, but once the task becomes long-form agentic work, it feels more like GPT-4o-mini -- still one to two generations behind the current frontier. That's fine. It lands in the "high-speed laborer under a privacy mandate" sweet spot, which is exactly where I need it.
Why the daily driver is 35B-A3B, not the top scorer
Interactive chat is ruled by latency. 35B-A3B trails 27B by just 0.025 composite yet runs nearly 4x faster, and 27B's only speed rescue, MTP (Multi-Token Prediction), is proven broken on this oMLX build (next section). Verdict: 35B-A3B is the daily driver; 27B is demoted to an on-demand fallback for "when time doesn't matter and you want peak Chinese/coding quality."
The two small models, loaded concurrently (~11GB combined), parallelize well: sequential 11.6s vs. concurrent 4.02s, a 2.89x speedup. Qwen3.5-9B is is_pinned so it hot-loads at boot with zero cold start on repeat tasks; gemma-E4B keeps the second concurrency slot warm. Extraction latency all but disappears.
Three speculative-decoding traps (to save you a weekend)
More speed means reaching for speculative decoding: "small model guesses, big model verifies." I wired up all three paths. All three failed. Logged here so you can skip them. This is oMLX 0.4.4.
Trap 1: native MTP is a no-op
Qwen3.6-27B-MLX-4bit's config declares mtp heads, but the 4-bit file ships no mtp.* weights, so the MTPModule is silently skipped and speedup is zero (measured +0.8%, pure noise).
Trap 2: VLM-MTP crashes outright
Switching to the draft path, 27B is classed as a vlm engine and hits an mlx_vlm bug:
AttributeError: 'tuple' object has no attribute 'hidden_states'
at mlx_vlm/speculative/mtp.py:170
A text-only LLM shoved into the VLM-flavored MTP path. Straight 500.
Trap 3: dflash loader missing args
After downloading a dedicated draft model and wiring up dflash, the log said it plainly:
DFlashDraftModelArgs.__init__() missing 12 required positional arguments
-> Falling back to vlm engine.
oMLX 0.4.4's DFlashDraftModelArgs loader never fills the required fields from the draft config. E4B crashes with a 500; 35B-A3B silently falls back, returning HTTP 200 with no speedup whatsoever. The most insidious failure mode: it looks like success.
Lesson 1: the log is the only truth. Don't trust HTTP 200. That 35B "200" was a silent dflash fallback; without the log you'd have thought acceleration was working.
specprefill: the same knob is a 2.4x boost or a 6x drag
specprefill is the one path that works. But it's not universal acceleration; it's a batch-long-document knob. This turned out to be the round's most useful finding. An ON/OFF A/B on 35B-A3B (long prompt):
| Scenario | OFF | ON | Verdict |
|---|---|---|---|
| cache-miss (a new long prompt each time, 16450 tok) | 27.57s | 11.68s | 2.36x faster |
| cache-hit (same long prompt repeated, 8310 tok) | 0.81s | 4.78s | ~6x slower |
Here's why: specprefill bypasses oMLX's prefix/session KV cache and recomputes prefill in full each time.
- New long input, no shared prefix (one-shot full-document analysis, batch digestion): ON, big win.
- Interactive agent (fixed system + tools prefix, session-accumulated context, mostly cache-hits): OFF. Turning it on drags each turn from 0.8s to 4.8s.
This overturned my earlier decision to wire specprefill into the driver. The interactive driver (Hermes) is set back to specprefill OFF; the knob is repositioned as "flip on only for batch long-document runs."
Lesson 2: speed knobs depend on workload shape, not one-switch-on. The same specprefill is a 2.4x boost on cache-miss and a 6x drag on cache-hit.
Final stack and routing
| Model | specprefill | Role |
|---|---|---|
| Qwen3.6-35B-A3B | OFF | Interactive driver, feeds on the prefix cache |
| Qwen3.5-9B | on (no-op on short input) | Extraction workhorse, pinned/hot-loaded |
| gemma-4-E4B | off | Second concurrency slot, high-throughput preprocessing |
| Qwen3.6-27B | on (turn off for interactive) | On-demand high-quality fallback |
Routing converges to: sensitive-data de-identification / structured extraction / literature triage goes to 9B (overflow to E4B); offline discussion and coding goes to 35B-A3B; peak Chinese/coding quality when time is no object goes to 27B. Genuine reasoning and final polish still go back to the cloud frontier.
One free bonus: pruning orphan models with no routing references (leftover benchmark artifacts, broken drafts) shrank the hub from 60GB to 33GB, reclaiming 27GB.
What I'd tell you over coffee
- The log is the only truth. Don't trust HTTP 200. Silent fallbacks masquerade as success.
- Speed knobs depend on workload shape. cache-miss vs. cache-hit can flip the same knob from savior to burden.
- Be willing to overturn yourself. Once the data showed specprefill was net-negative for the interactive driver, I pulled it. Evidence over ego.
If you want to run an offline local stack on one Mac, this transfers directly: pick one MoE as the driver (speed/quality sweet spot), one 9B-class model as the extraction laborer, one tiny model as the concurrency slot; and leave speculative decoding alone on this build until oMLX fixes the draft loader.