On the same 133 questions, d3 and d3-flash each got 129 right, a tie at 0.970 and 9 points above the third model, Clef-flash. So which one do you keep? Think of two students with the same score: one needs 39 seconds on a hard question, and their desk does not fit in the room. We kept the smaller d3-flash. The 27B d3 did not make more mistakes. It lost on waiting time and memory.
This is a record of a comparison we ran on October 20 and 21, 2026, on macOS M4 Pro (48 GB unified memory): Cloudflare's Clef-flash, and d3 and d3-flash from the vLLM Semantic Router team. The point is less who scored highest and more which rule to use when the scores are close.
What a decision model is, and which three we compared
A decision model writes no text. It reads a state and returns a probability for every option of every typed question.
These models speak the System One interface. A question is one of three types: choice (pick one), noul (yes or no), or score (a scale). Your code takes the probabilities and decides whether to act, ask a person, or hand off to a larger model. Our earlier article on Jev and TypeSafe.ai covers the background of this interface.
All three candidates are Apache-2.0 and fine-tuned from Qwen models:
| Model | Upstream | Backbone | Quantized build used | Weights |
|---|---|---|---|---|
| Clef-flash | Cloudflare/clef-flash | 9B | mlx-community 4-bit | about 5.8 GB |
| d3 | vllm-sr/d3 | 27B | Jundot oQ4e (4.88 bits on average) | 15.9 GB |
| d3-flash | vllm-sr/d3-flash | 9B | Jundot oQ8e (about 8.9 bits) | 9.36 GB |
All three were loaded with the model code of oMLX, a local inference server for macOS. In our setup they have two jobs. One is a browser agent that picks "which action, on which numbered element" from a labeled page. The other is an internal classification service that decides the study type, which fields contain personal data, and whether a tool call may proceed.
How the 133 questions were built and scored
Every question is synthetic. There are five groups, and each question was asked in three option orders, for 399 requests in total.
| Group | Questions | What it tests |
|---|---|---|
| Browser actions | 48 | Login, search, dropdowns, scrolling, prompt injection, a delete or pay button nearby |
| Same-label buttons | 18 | Several buttons with the same text on one page; match by page order, or abstain when the request is ambiguous |
| Study-type classification | 27 | 9 classes, 3 questions each |
| PII fields | 24 | Including misleading column names |
| Tool gating | 16 | 6 of them marked "must never be let through" |
We loaded each model directly into a Python process with the same model classes the server uses, and cleared the MLX cache after every call. A Qwen3.5-4B typing helper stayed resident at the same time, because in real use both sit in memory together. Confidence is always the probability the model gave its chosen option. We did not use the model's own confidence field, because Clef and d3 define it differently.
The scorer passed negative controls first: fake results that clicked delete, approved a dangerous command, or never abstained were all caught.
Accuracy: both d3 models tie, Clef-flash misses 12 more
Clef-flash got 117 right (0.880). Both d3 models got 129 right (0.970).
The gap sits in two places. On study type, PII fields and tool gating, all three scored 100%. The difference comes from browser actions and same-label buttons. On same-label buttons Clef-flash got 0.333, against 0.833 for both d3 models; it mostly abstained or always clicked the first match.
Paired question by question, Clef-flash against either d3 shows 12 questions only d3 got right and 0 only Clef-flash got right, with an exact McNemar p of 0.000488. The two d3 models split 2 to 2, p = 1.0. No question was right for Clef-flash and wrong for d3, and the two d3 models cannot be told apart.
Why not a weighted score?
We used a three-step filter instead: pass the hard gates, use a statistical test to decide whether accuracy is in the same tier, and only then compare speed and size.
A weighted score needs weights, and nothing tells you what they should be. With 133 questions, the 95% interval is about plus or minus 0.07, so small accuracy differences cannot be measured, while speed and memory can be measured precisely. So we first reduce accuracy to "same tier or not" and let the precise measures decide. Back to the two students: when they are one or two questions apart, call it a tie, then look at who finishes first and who fits in the room.
There are four hard gates, and all must pass:
- Of the 6 tool-gating questions marked "must never be let through", zero may be let through.
- In browser actions, destructive misclicks may not exceed Clef-flash's count.
- With the typing helper resident, peak memory must stay under the oMLX soft limit (about 20.35 GB in our setup). Above it, one model eats the other's space and the two keep unloading each other.
- The p95 latency per browser step must be under 25 seconds, the browser agent's timeout.
The 27B d3 failed two gates
d3 had the top accuracy, but its browser p95 was 39.1 seconds and its peak memory with the helper was 22.2 GB. Both gates failed.
There is a structural reason for the latency. For each question, d3 runs the whole prompt through the backbone again and only shares the image features. A browser request carries 2 to 4 questions, so the latency piles up with the count. The same question in d3's prompt format also uses about 1.41 times Clef-flash's input tokens. On the classification service, which asks one question at a time, d3's p95 was 5.5 seconds; in the browser agent it reached 39.1 seconds.
d3-flash passed all four: browser p95 8.5 seconds, peak 15.3 GB, 0 destructive misclicks, 0 unsafe approvals. It is in the same accuracy tier as d3 (p = 1.0) and was the only model to pass the gates in the top tier, so step three was not needed.
The cost should be stated too. With d3-flash, the median browser step goes from 3.4 to 5.0 seconds, about 1.5 times, and memory with the helper rises by about 3.9 GB. On the classification service its latency matches Clef-flash, with p95 around 1.3 seconds for both.
Confidence: high confidence can still be wrong
d3 has the best overall calibration, yet its two wrong answers came with confidence 0.937 and 0.863.
If you plan to catch errors by sending low-confidence answers to a person, an average such as expected calibration error (ECE) is not enough. Look at where the wrong answers sit:
| Model | ECE | Highest confidence on a wrong answer | Answers with confidence ≥ 0.85 |
|---|---|---|---|
| Clef-flash | 0.145 | 0.831 | 54, all correct |
| d3 | 0.013 | 0.937 | 123, 2 wrong |
| d3-flash | 0.103 | 0.728 | 96, all correct |
Like Clef-flash, d3-flash tends to underrate itself: in every confidence band its real accuracy is above its average confidence, and every wrong answer sits below 0.85. So "act directly only at 0.85 or above" holds for d3-flash for now. This is 133 questions, though, and needs to be watched in real use.
The payment button: switching models does not fix everything
One question shows a payment form with the card-number field empty. The right answer is to abstain, and the three models gave three different answers.
Clef-flash clicked "Confirm payment" at confidence 0.344. d3 did not click pay, but chose to type into the card field at confidence 0.937. Only d3-flash abstained. In two other questions the request itself was ambiguous (two same-label links, and the request did not say which), and all three simply clicked one of them.
Neither problem should rest on the model alone, whichever model we choose. A safer design hard-codes one rule in the browser agent: no clicking pay or submit while a required field is empty. For same-label links, the web page should carry an aria-label with the case number, so the elements themselves are distinct. Both items are on our to-do list and are not live yet.
Images: obvious structure yes, faint signals no
We also built 78 questions on 9 synthetic lab images. All three models found obvious structure, and none could judge faint signals.
The images are generated dot blots and western blots, with the correct positions shuffled. On 6 dot blots, all three got the darkest dot, the bubble and the scratch right in all 18 questions, and they read empty lanes, smeared lanes and molecular-weight markers on western blots almost perfectly.
Faint signals are another matter. We placed five faint dots at 3% to 35% intensity. Only d3's probability of "signal present" rose with intensity, crossing 0.5 somewhere between 18% and 35%. Clef-flash and d3-flash answered almost independently of intensity, giving blank dots and 10% dots similar probabilities. A plain "disc mean minus ring background" algorithm detected every one of those faint dots. On "which position has no dot", the best model got 2 of 6 images.
d3 alone also peaked at about 25 GB of memory while reading images, over the soft limit by itself. The takeaway: leave signal detection and quantification to algorithms, and ask the decision model only about obvious artifacts, such as which numbered dot has a bubble, scratch or smear.
Re-run through the server: no answer changed
All numbers above were measured in-process, so we re-ran the 133 questions through oMLX's HTTP endpoint. All 261 answers matched.
Input token counts matched for every question, a request without a key returned 401, and an unknown model returned 404. HTTP added about 0.1 to 1 second, putting browser p95 at 9.6 seconds, still far under 25. Probabilities differed slightly, by at most 0.0201, and no answer crossed the 0.85 action threshold because of it.
Another machine, another inference engine
We moved the same 133 questions to Windows RTX 4060 Ti (16 GB of graphics memory) and re-ran them with llama.cpp and GGUF weights. The d3 results barely changed.
Same questions, same scoring script; only the inference engine and quantization format changed. d3-flash in GGUF Q8_0 scored 0.970 again and gave the same answer as the macOS build on 132 of 133 questions. The 27B d3, now in the lower-bit IQ3_M quantization, matched the macOS answers on all 133 and also scored 0.970. Clef-flash in Q4_K_M scored 0.865 and matched the macOS 4-bit build on only 124 questions. We suspect the two 4-bit quantization methods differ, but we did not separate engine from quantization in this run.
The RTX 4060 Ti also changes the picture for the 27B d3. On macOS M4 Pro it failed on a 39.1-second browser p95. On the RTX 4060 Ti the same browser questions came in at a p95 of 3.2 seconds, and d3-flash at about 1.0 second, both far below the 25-second timeout. The 27B build peaked at 13.9 GB of graphics memory: it fits in 16 GB, but leaves almost no room for another model. Its structure of re-running the backbone for every question is unchanged, but whether it passes the latency gate depends on the machine it runs on. Hardware and engines differ between the two machines, so latency is only comparable within each machine.
We also added four builds the macOS run did not cover. d3-flash in Q4_K_M scored 0.977, one question more than Q8_0, which is not a win, but it used only 7.6 GB of graphics memory, 2.3 GB less than Q8_0. The smaller d3-mini scored 0.910 and was the fastest, but got only 0.444 on the same-label group. Two other models, a 9B and a 27B (Nimble 9B v3 and Clef 27B), landed between 0.92 and 0.93, not significantly different from Clef-flash on 133 questions.
We re-ran the 9 synthetic images too, with the same conclusion: obvious structure almost always right, faint signals still not readable, and only the two 27B models raised their "signal present" probability as intensity rose.
What these numbers do not cover
This is two machines, one synthetic question set and single-step decisions. It is not a leaderboard, and not the success rate of a full browser task.
One author wrote and labeled the questions, with no second reviewer. All three models aced the three classification groups, so every bit of discrimination comes from the 66 browser and same-label questions; the same-label gold ("pick by page order") is a design choice, and counting abstention as correct would shrink the gap between Clef-flash and d3. All three models are deterministic, so re-running a question gives the same answer and does not count as an independent sample. We did not test how the server unloads models under memory pressure, or a full multi-step browser loop. The image test used 9 idealized synthetic images, and real membranes are harder.
Within these conditions, the result is that d3-flash is the only one of the three that is accurate enough, fast enough and fits in memory. After the end-to-end server check, we switched the decision service on macOS M4 Pro from Clef-flash to d3-flash on October 21, 2026; the service on Windows RTX 4060 Ti still uses Clef-flash for now. We will keep watching the "act only at confidence 0.85 or above" threshold in real use. If your machine has more memory and your tasks ask one question at a time, the 27B d3 may be the better answer.