Thirty questions cannot separate the front runners
Three benchmarks of 30 questions give a 95% interval of about ±8 points, so July’s Qwen3.6-27B OptiQ (83.3) is statistically tied with the September and October models.
Overview
Platform
Question subset
Every local model tested since July with per-question results, re-scored on the same questions so they sit on one scale.
The numbers reflect the hardware, settings and question samples used: a reference for choosing a model, not a general ranking.
What the results show
A hundred questions show the gaps, if conditions match
On v1, occamy-1.0 (91.0) and MiniCPM-V-4.7 (90.3) with thinking on lead Ridge and OrcaSAQ (84.7) with thinking off; part of the gap comes from thinking mode and host.
Same model, same questions, different score
On the same 30 MMLU questions, supergemma4 went from 18 to 22 and gpt-oss from 25 to 23; one 30-question run swings enough to flip a ranking.
Chinese knowledge is the weak spot of newer MoE models
Ornith-1.5-35B-A3B Vision MTPLX scored 36/100 on CMMLU; the 27B dense models of the same period scored 72–78.
One point per run
Sort by a column · use a checkbox to track a run
Each chart uses its own vertical scale
Logarithmic horizontal scale · upper left is better
Methods differ; use as a rough reference
How re-scoring on the same questions works
Our question samples are nested: every July n=30 question is inside the 100-question v1 set frozen in September, and the large July 28 re-test (n=100–1000) also contains those 100. Every run can therefore be re-scored from its per-question results on the same 30 or the same 100 questions. The Windows runs use the same frozen question set (matching hash), so their question ids overlap completely.
Three question scopes
- Same 30 questions: every run is comparable, but the 95% interval is wide (±13–17 points per benchmark). Use it for the long-term trend.
- v1 100 questions: only runs from September on and the large re-test; more precise. MMLU-Pro and TruthfulQA share questions only in this scope.
- As reported: each run’s own sample size, matching its report, but the questions differ between runs.
MathQA is left out by default
Checking the July 28 answers one by one showed that MathQA asks for the letter only, and models that write out their working get cut off at the 603-character output limit: at n=300, supergemma4 gave no extractable answer for 53% of questions and scored 8.7%, below the 20% of random guessing. This benchmark measured format compliance and output length, not maths. You can still tick it.
Thinking mode and host conditions
gpt-oss (some benchmarks), occamy, MiniCPM-V-4.7 and the Ternary-Bonsai-27B think variant ran with thinking on; the chart marks them with an outer ring. The three Windows runs used temperature=0, max_tokens=512 and thinking off; the Mac runs follow each test’s settings. These differences move scores, so the host and thinking-mode filters can be used separately.
What the July 28 large re-test taught us
supergemma4’s MMLU was 60.0 at n=30 and 80.5 at n=1000, outside the n=30 95% interval. Paired McNemar tests of gpt-oss against supergemma4 on the same questions (MMLU 1000, HellaSwag 200, GSM8K 100) all gave p above 0.05.
How to read this chart
9 images, 78 questions · both platforms listed
Only the two 27B models get more confident as the signal grows, and only past half at levels 4–5: leave signal detection and quantification to algorithms.
How to read this table
The seven right-hand columns are the mean probability that a model says a spot has signal, at each level of the synthetic dot blot from blank to strong. An ideal model stays near 0 on blank and rises steadily left to right; only the two 27B models roughly do, and only cross 0.5 at levels 4–5. d3-flash’s probability barely tracks signal strength (0.37 on blank, 0.52 on the weakest level), so the 4 extra questions d3-flash-q8 got right on Windows do not mean it reads images better. Blank questions ask which position has no spot; lenient scoring also accepts the weakest level, which the eye cannot tell from blank. Per-image latency depends on platform and engine and compares only within a platform. Our conclusion: leave signal detection and quantification to algorithms and use the models only to recognise obvious structure.
Measurement
Decision models only answer multiple-choice questions and produce no text, so they are kept out of the benchmark matrix in the inference tab. On the Mac, the 133 questions were run with the model loaded in-process, each question three times with original, reversed and shuffled option orders, the cache cleared after every call, and memory measured as the process’s real footprint (plain RSS misses the Metal cache). There was no HTTP in the loop, so latency can only be underestimated against real use.
Windows re-test (10-21)
The same 133 questions became 399 HTTP requests (three option orders) served by llama.cpp b11588 with a 4096 text context. Scoring reused the Mac round’s scoring program unchanged; re-scoring the three Mac models the same way reproduced their stored results exactly. VRAM is whole-GPU usage sampled every 2 s, including about 0.7 GB for desktop programs, shown here in GiB. Load time includes the wait for switching models. Co-residence with other models and concurrency were not tested.
What can be compared across platforms
Accuracy, calibration, abstention and safety metrics use the same questions and the same scoring program, so they compare directly across platforms. Latency gives direction only: the Mac is called in-process alongside another resident model, Windows over HTTP to local llama.cpp with CUDA, and HTTP can only push the Windows numbers up, so d3-flash’s 8.5 s on the Mac against 1.0 s on Windows is a reliable direction but not a reliable ratio. The memory columns measure different things and cannot be compared.