Local Model Benchmark Self-Check: A Three-Step Checklist

Step 1 | Confirm the questions were actually answered

Open the per-item raw files (JSON/JSONL). Do not only look at total scores.

• Blank prediction rate (predicted is empty, null, or missing key)
• Response length distribution (median, P90, whether hitting fixed character/token limits)
• Whether endings are truncated (half sentence, option letter not emitted, reasoning ate the budget)
• Whether correct items are mostly single letters (e.g. only "A"), while wrong items are long reasoning chains

Pass criteria: Blank prediction rate near 0; within the same benchmark, correct/incorrect response shapes should not systematically split into "short letter vs long truncation".

Script usage:

python3 blank_prediction_audit.py /path/to/result.json

Download blank_prediction_audit.py →

Step 2 | How wide a confidence interval the sample size supports

• Record each benchmark's n (item count)
• Rough binomial proportion 95% interval: about ±1.96 × √(p(1−p)/n)
• If the gap between two models is less than half the interval width, do not use this run to fix rankings
• For key benchmarks (especially cells that decide ranking), prepare a retest plan with n≥100

Experience anchors (from T20 runs, for comparison, not a universal formula):

Step 3 | Split sensitive-task channels

• Whether sensitive/PII/medical records only go through the local channel
• Whether there is a deterministic fallback (rule-based regex scan), not just betting on one model in one pass
• Whether de-identification is at least two rounds (first-round mask + residual recheck)
• Whether the benchmark leader has been through a sensitive probe; if it missed masking names, not allowed on the sensitive path

Two Final Gates Before Publishing

Hard stops: if any one holds, do not publish a ranking yet

1. A benchmark score is below random guessing, and per-item JSON was not opened.
2. Blank prediction rate > 10%, yet that benchmark's mean is still used for ranking.
3. Claiming "1st place / 5th place" as settled with only n=30.
4. The sensitive path has no local-only restriction and no rule-based fallback.

Three places to look first when fixing the yardstick

1. Whether max_tokens / response length caps truncate answers
2. Whether the extractor only accepts "the sole letter in the full text", and misses the option at the end of long reasoning
3. Whether the wrapper prints thinking/reasoning into the string meant for json.loads

Full write-up: One Month of Local Model Benchmarking, I Overturned My Own Rankings →