One Month of Local Model Benchmarking: I Overturned My Own Rankings

In July I ran six rounds of local model tests. By month's end, I had overturned the rankings I had set myself.

Not slapped by someone else: slapped by samples I filled in myself. One benchmark was ruled invalid; a model I had personally sentenced to death jumped from 5th to 2nd, and the original champion dropped to 3rd. There was something uglier in the middle: an "acceleration success" that did not accelerate at all, while the service still returned HTTP 200.

The most expensive lesson this month: when the measuring stick is broken, every ranking goes wrong with it. The six sections below follow the timeline. Each section covers what I did first, then what I learned; it closes on how pi and local models divide work now, and four accounts still unsettled.

If you also self-host local models and run your own scores, what you should really look at is not my rankings, but the things that make you rank wrong.

07-05｜Acceleration Knob Testing: dflash Broke, specprefill Only Helps Batch Work

Local inference acceleration knobs are not something you turn fully on with one click. The same specprefill gave 2.36x speedup on long batch documents, but was a 6x drag on interactive chat; and dflash never ran at all on this version of oMLX, yet the response was still 200.

What I did

Early in the month I hit the acceleration knobs first. dflash broke immediately: the log spit out DFlash start failed: DFlashDraftModelArgs.__init__() missing 12 required positional arguments. E4B crashed with 500 as a result; 35B-A3B was uglier, silently falling back to vlm, still returning HTTP 200, with zero actual acceleration: the service was alive, but acceleration never happened.

For specprefill on 35B-A3B I ran long-prompt A/B: cache-miss dropped from 27.57s to 11.68s (about 2.36x), but cache-hit worsened from 0.81s to 4.78s (about 6x slower). The mechanism bypasses prefix and session KV cache and recomputes the full prefill every time (for the full picture of this class of acceleration techniques, see Six Techniques for Speeding Up LLM Inference). The ruling flipped accordingly: specprefill for the interactive primary model was turned off, and only temporarily enabled for long batch documents. The original decision that "the primary model gets specprefill" was overturned by my own data.

The same day I also confirmed that native MTP on 27B is unavailable: the 4bit file declares mtp heads, but has no mtp.* weights. On cache slimming, hub went from 60G down to 33G, reclaiming 27G; deletion used mv to staging, with the final rm done by the principal by hand (local hooks hard-block agents from running rm).

What I learned

The log is the only source of truth; do not trust HTTP 200. Silent fallback is more dangerous than a clean crash, because everything looks fine on the surface while the actual gain is zero.

07-10｜supergemma4-26b-uncensored First Run: mathqa Below Chance

When a model's score falls below chance, suspect the measuring stick first; do not write the verdict first. That round I did both: I flagged the anomaly, and still ranked it 4th. The bill came due at month's end.

What I did

supergemma4-26b-uncensored entered for the first time, sample n=30. The verdict at the time: core-7 mean 0.633, ranked 4th; mmlu 0.600 and cmmlu 0.567 were weak; mathqa was only 0.100, below chance, already flagged as anomalous, and the report at the time already recommended a retest. The conclusion was not to put it on the daily routing path, only as an "uncensored" specialized fallback. Models of this kind are usually further fine-tuned on an existing base; I wrote about how fine-tuning changes how a model talks in 76 Samples, 33 Seconds of Training: How LoRA Fine-Tuning Makes AI Speak Your Language.

What I learned

If an anomalous score is closed as "this model is bad," you bury the measuring-stick problem with it. mathqa 0.100 was flagged at the time, but was not formally invalidated until month's end; the temporary ranking at n=30 also shifted in whole blocks once larger samples came later.

A below-chance score is a measuring-stick failure signal, not a legal input to rankings. Parse the per-item JSON first, then decide whether to fold that benchmark into the mean.

07-17｜Ternary-Bonsai-27B 2-bit: What Extreme Quantization Preserved

2-bit quantization preserved generative tasks, but collapsed on multiple-choice in ways that looked senseless. What collapsed was not ability; the model forced long reasoning chains into the token limit, and the answer was truncated before it could come out.

What I did

Bonsai 2-bit was strong on generative extraction: gsm8k 0.967, humaneval 0.854, mbpp 0.833 (mean 0.885), approaching 4-bit 27B under 2-bit. MCQ collapsed: mathqa 0.233 (10 empty predictions out of 30 items), cmmlu 0.433 (5 empty), mmlu 0.667 with thinking off (3 empty). The cause of empty predictions was not hard to find: the model forced long reasoning chains, hit max_tokens, got truncated, and the answer never came out.

The cost of the thinking switch was clear: mmlu 0.900 with it on, 0.667 with it off, a 23.3 percentage point gap; but seconds per item went from 6.3 to 61.9, about 10x. The real weakness was Chinese: cmmlu was still about 0.52 even after removing truncation, trailing the 27B series at 0.80 and 0.833; livecodebench 0.267, and 179 seconds per item.

What I learned

This is a systematic harness problem that replays across models, so the correct order is fix the scoring script first, then retest. I named it at the time, but did not fix it. Extreme quantization can preserve generative tasks, but cannot hide how format and truncation amplify effects on multiple-choice.

07-24｜Nine-Model Comparison and Six pi Probes: Cutting the Local Model Fleet

Benchmark scores do not decide who can touch sensitive data. The overall first-place model missed masking a physician name on the de-identification probe, while two lower-ranked models masked both names cleanly. This round reduced nine models to two.

What I did

The nine-model full comparison (n=30, mean across 7 shared benchmarks) produced a table that looked clean at the time:

Rank	Model	Mean
1	gpt-oss-20b	80.5%
2	OptiQ	77.1%
3	27B-MLX	74.8%
4	35B-A3B	71.9% (3.4 seconds per item, speed king)
5	supergemma4	63.3%
6	Qwen3.5-9B	60.0%
7	Bonsai-2bit	58.4%
8	gemma-12b-coder	55.2%
9	gemma-E4B	54.8%

The pi six probes are live runs, not Q&A scores:

• --schema stability: gpt-oss failed all three times (reasoning block blew past the token budget, JSON truncated); A3B perfect all three times; OptiQ perfect twice.
• Sensitive de-identification: gpt-oss missed masking the physician name "林XX", a personal-data leak; A3B and OptiQ masked both names cleanly.
• Gatekeeper (contradiction detection): gpt-oss's only win, it could detect and list contradictions; A3B blindly complied.
• Language: gpt-oss hit terms from the Traditional Chinese forbidden-word list; OptiQ strongest on Chinese (cmmlu 83.3).

Later the same day a fix clarified the root cause: the reasoning block was already separated into reasoning_content; omlx_ask was printing it too, which made json.loads fail. After fixing the wrapper (--schema always suppresses reasoning, plus defensive JSON extraction), gpt-oss could take --schema. A3B's sole exclusive speed advantage was then not enough to keep it; second cut.

Cut result: a two-model fleet of OptiQ and gpt-oss. Retired A3B (25G), Qwen3.5-9B (5.6G), gemma-4-E4B (6.4G); retired models totaling about 37G, and rewired about 14 interfaces.

Three new models in the same period: Bonsai stuck at runtime (Ollama's built-in llama.cpp cannot decode ternary Q2_0_g128; needs a PrismML fork); Unlimited-OCR failed security review (modeling_unlimitedocr.py has 7 eval() sites that consume model output, forming a remote code execution surface) and was deferred; Qwythos-9B had provenance red flags (the author could not be verified as a real entity) and no capability delta, so not brought online. Unverified weights and unverified packages are the same class of risk; I go into more detail in There May Be a Backdoor Hiding in Your AI Toolchain.

What I learned

"Ranked first" and "can enter the sensitive path" are not the same thing. Which cloud generation a local model roughly matches is covered in A Local LLM on One Mac: Roughly Which Cloud Generation Is It?. gpt-oss wins on benchmark mean and contradiction detection, yet misses masking names and hits forbidden words; wrapper-layer bugs can also misread model ability as format failure. Fleet cuts must therefore weigh capability, safety, and the number of maintainable interfaces at once.

07-25｜Windows Device Comparison: What an RTX 4060 Ti 16GB Can Actually Run

The 16GB VRAM ceiling decides this device can only be a resident service, not a heavy-work primary. On this card, gpt-oss:20b is the best balance of speed and capability; larger models either do not fit or need offload and get slowed.

What I did

Windows device measured tokens per second: gemma4 at 72 to 74 (speed king), gpt-oss:20b at 66.7, qwen3.5:9b at 48, qwythos-eval at 42.5, cellergy35:v2 at 30 (slowest overall, and failed to deliver on all five questions). gpt-oss:20b delivered all five, output bare and clean, Apache license clean; set as the Windows device primary; Hermes config.yaml switched over from cellergy35:v2.

The thinking model's "empty response" was confirmed as an API handling artifact: thinking exhausted the 450 budget, not a capability defect. A Windows hard pit also showed up: PowerShell 5.1 Get-Content default encoding misread Chinese config as cp950 and wrote it back; after config corruption Hermes silently fell back to defaults, so the fix is always to go through the .NET UTF-8 API. The 16GB VRAM ceiling conclusion is therefore clear: heavy work goes to the Mac M4 Pro 48GB; the Windows device is positioned as a resident service.

What I learned

Platform encoding and API budget can manufacture "false capability defects," so Windows device and Mac roles should be split; do not grade both on the same leaderboard with the same expectations.

"Can stay resident" on 16GB VRAM and "can carry heavy work" on 48GB were never supposed to fight for the same rank on the same board. Once platform roles are split, the speed-versus-capability tradeoff becomes readable.

07-28｜Large-Sample Retest: Where the n=30 Ranking Went Wrong

With only 30 items per benchmark, confidence intervals were as wide as plus or minus 13 to 17 percentage points, enough for two models' ranks to swap by luck. After topping up to 1000 items, one cell shifted by 20.5 percentage points, outside the old interval.

What I did

The mathqa benchmark is voided. supergemma4's mathqa at n=300 was only 8.7% (random guess on five choices is 20%). After unpacking per-item JSON: blank predictions on 159 items (53.0% of 300), raw_response hard ceiling of 603 characters, incomplete final sentences on 80.7%; of the 26 correct items, the typical response was a single letter "A".

The cross-model dividing line is clear: models that emit a single letter (median length 1) scored 43 to 93; models that emit a solution process (median length 350 and up) scored 8.7 to 23.3. The counter-evidence: the same supergemma4 scored 90.0% on gsm8k at n=100. The scale is broken, not the capability.

supergemma4 was underestimated by n=30: mmlu rose from 60.0 to 80.5 (n=1000, up 20.5 percentage points, outside the old 95% confidence interval); arc from 86.7 to 95.0 (n=300, up 8.3 percentage points, also outside the interval); cmmlu from 56.7 to 69.7; truthfulqa from 86.7 down to 81.3; humaneval from 96.7 down to 94.5; gsm8k held at 90.0.

By contrast gpt-oss moved almost nothing on three cells (mmlu 83.3 to 82.9, hellaswag 83.3 to 83.0, gsm8k 96.7 to 95.0, error under 2 percentage points), but that was luck: n=30 confidence intervals are equally wide at plus or minus 13 to 17 percentage points and cannot argue that n=30 is enough. Matched-item McNemar (identical items) results: mmlu p of 0.086, hellaswag p of 0.216, gsm8k p of 0.074; all three not significant. gpt-oss paid for this with 6.95 seconds per mmlu item versus supergemma4's 0.92 seconds per item, about 7.5x inference time (hellaswag is 11.2x).

Biomedical subset (13 subjects, 179 shared items): gpt-oss got 144 correct (80.4%), supergemma4 got 141 correct (78.8%), only 3 items apart, treatable as equivalent.

After dropping mathqa, the shared 6-benchmark ranking reshuffled: OptiQ 81.1 (1st), supergemma4 79.2 (2nd, was 5th, up 15.9), gpt-oss 78.2 (3rd, was 1st). But the new table is still not final: except for supergemma4 and gpt-oss, the other seven remain at n=30; 1st-place OptiQ was not retested on a single cell.

cmmlu subject-level fracture (supergemma4) is more critical: modern_chinese at 0.00, elementary_chinese at 0.29, chinese_literature at 0.40, but chinese_food_culture, driving_rule, foreign_policy all at 1.00. It can answer facts about the Chinese-speaking world; it cannot handle Chinese as a language. Traditional Chinese writing and polish should never use it. Likewise, if OptiQ's cmmlu 83.3 is to substantiate "Traditional Chinese primary," subject-level scores are required, and it only has n=30, which is effectively no data.

Recommendation: pause retiring supergemma4; but it is an uncensored fine-tune with safety alignment weakened, and that is the real reason it cannot be an unconditional primary, unrelated to capability.

What I learned

Small-sample rankings can reverse wholesale. Format compliance can masquerade as capability failure; means can hide subject-level fracture. An unretested 1st place does not necessarily hold thicker evidence than an underestimated 5th.

pi and Local LLMs Today: Who May Touch Sensitive Data

When personal data and medical records are involved, the local channel is the only legitimate channel, and not just any local model qualifies. The one that fails to mask names, even with higher benchmark scores, is not allowed on this path.

pi's role was reordered by the principal on 2026-07-16: dedicated local handling of sensitive and personal data, plus offline fallback; it sits last in the outsourcing order (grok and codex, then agy, then subagent, and only then pi). Once sensitive data is involved, it is the only legitimate channel.

pi's local model is Qwen3.6-27B-OptiQ, via the oMLX local port 127.0.0.1:8090. Sensitive calls must be written explicitly:

pi --provider omlx --model qwen36-27b-optiq

Bare pi inside the workspace routes to local gpt-oss-20b, and gpt-oss fails to mask names. Measured latency is roughly: tools and thinking off, about 16 seconds; thinking only off, about 24 seconds; plus workspace read-only (which loads AGENTS.md context), about 56 seconds.

pi's four red lines:

1. No code implementation under contract constraints (measured: it stealthily adds imports).
2. Must not serve as gatekeeper (no contradiction-detection capability).
3. Structured output must always use --schema or --json-object.
4. Must not take literature retrieval or citation generation: pi's web_search returns "includeContent": false; it is a summarizer, not a retriever; source content is never fetched; the model assigns one generic paragraph line by line across each URL; literature identifiers are all real, content all fake. This is structural failure; swapping models does not fix it.

High-risk medical-record de-identification follows a three-stage iron rule: first-round mask by OptiQ, then rule-based regular-expression scan for personal data (deterministic backup), finally residual review by OptiQ. Single model, single pass is unreliable: OptiQ still had a non-deterministic miss-mask once even at temperature 0; as for gpt-oss, it is strictly forbidden on the sensitive path.

Contract compliance is probabilistic, not an inherent model property, so pre-deployment contract scanning can never be skipped. That is the same conclusion that holds again under a different scene in The Model Is Strong Enough, So Why Are Local AI Agents Still Unstable?.

pi cannot replace hermes (settled after three rounds of research on 2026-07-25): they sit at different layers. pi is a coding-agent command-line interface, deliberately not a resident multi-channel service; hermes is a resident gateway shell. pi has no first-class webhooks; exhaustive search found no ready-made LINE extension. The real lever for reducing harness sprawl is not swapping technology, but cutting dead hermes clones.

Three Generalizable Lessons

1. The Yardstick Comes Before the Model

mathqa at n=300 is only 8.7%, below five-way random guessing; blank predictions over half, hard ceiling at 603 characters, correct items mostly a single letter. Bonsai's blank MCQ predictions, thinking hitting max_tokens, and the 07-05 HTTP 200 zero-acceleration case all point to the same thing: suspect the harness and observation layer first, then the model. An unfixed yardstick will misjudge the whole row of models together.

2. Small-Sample Rankings Are Provisional Labels

n=30 confidence intervals can be as wide as plus or minus 13 to 17 percentage points; supergemma4's mmlu moved from 60.0 to 80.5, a shift outside the old interval. After dropping mathqa, the ranking went from "gpt-oss first, supergemma4 fifth" to "OptiQ, supergemma4, gpt-oss". McNemar on the same items was again not significant, showing that speed gaps (7.5x, 11.2x) are the daily ledger that matters more. Any "Nth place" before larger samples are filled in is only a working hypothesis, not a conclusion.

3. Average Ability Cannot Substitute for Path Specialization

gpt-oss can be the Windows device workhorse and the one win on contradiction detection, but cannot enter sensitive de-identification; OptiQ is more stable on Chinese and masking, yet still needs regex backup and residual review; supergemma4, even after ability was underrated, still cannot be an unconditional workhorse because it is uncensored; cmmlu subject breakdowns show that "knows Chinese-world facts" is not "can write Traditional Chinese". Routing must be split by task risk; one mean-score table cannot decide every path.

The Open Ledger

This month the rankings were torn down and rebuilt, but the books are not closed:

1. mathqa harness still unfixed (raise max token, or switch to extracting the last option letter that appears); historical misjudgments on 4 gemma-family models still uncorrected.
2. OptiQ is post-correction 1st place, yet the thinnest evidence in the field (all n=30), especially modern_chinese and elementary_chinese subjects, which essentially have no data.
3. livecodebench has no large-sample retest at all, and is the only benchmark where every model scores low (13 to 53), most likely hiding mathqa-type defects.
4. Systematic follow-up on contract compliance and --schema stability (07-24 old debt) still undone; the mathqa incident already proved that format compliance can masquerade as capability failure.

Rankings can be built first, then overturned. If open debts keep sliding, the next "settled" call still has a chance to be dismantled by our own data again; until the books are cleared, treat any new ranking as provisional.

---

How You Can Use This Piece

If you also have a machine running local models, I suggest three things first, and do not swap the order:

1. Open the per-item raw files from your benchmark runs, and look at blank-prediction rate and response-length distribution. Before the scores, confirm the items were actually finished.
2. Work out how wide a confidence interval your sample size can support. A ranking from 30 items cannot survive any retest.
3. Isolate the channel for sensitive tasks, use rule-based scanning as deterministic backup, and do not bet on a single model's single-pass output.

Free Download: Benchmark Self-Check Checklist and Blank-Prediction Script

I packed the three steps above into a checkable list, plus a small script that reads per-item JSON and reports blank-prediction rate and length distribution. Both sit on the resources page, no signup and no email required:

• Checklist: The Three-Step Local Model Benchmark Self-Check
• Script: blank_prediction_audit.py

python3 blank_prediction_audit.py /path/to/your_result.json

This piece is part of the Claude Code Masterclass series. For local vs cloud tier comparison, read A Local LLM on One Mac: Roughly Which Cloud Generation Is It?; for why systems still wobble after models get strong enough, read The Model Is Strong Enough, So Why Are Local AI Agents Still Unstable?.

Common questions are collected in faq_EN.md in the same folder. Next round, once I fix the mathqa yardstick, I will come back with a full-fleet retest.