Running a Decision Model on a Mac: Question Design and Local Deployment

A decision model can run on your own machine when each question asks one thing, the options are mutually exclusive, and one option is an other class. On 2026-10-11, open-weight Clef-flash took about 1.8 seconds per paper to assign a study type on a Mac (MLX 4-bit) and a p50 of 266 milliseconds per paper on an NVIDIA workstation (GGUF Q4_K_M); across 77 PubMed abstracts the labels matched, and both accuracies were 92.2%, which is not a general accuracy. The input is a state, the text or JSON being judged, plus a typed question whose options are fixed in advance; one forward pass returns a probability for each option and does not generate free text.

Part 4 of the series | This part answers: How do you design the question and the options for a decision model on your own machine, and how fast does it run on a Mac and on NVIDIA?

Part 1 defines a decision model, part 2 covers how one is trained, and part 3 checks calibration on 77 PubMed papers. This part only covers how to write the question and how to run the model on your own machine. Part 5 is where an agent asks before it acts.

Ask one thing per question

Ask one thing per question: study type is a single choice, and a tool gate is two questions, action and high_risk, not one combined question.

Typed means the options and the shape of the answer are fixed before you send the question. The docs describe three kinds. A noul is yes-or-no. It returns the probability that the statement is true, a value from 0 to 1, and it has no confidence field. A choice picks one named option. The option count is 2 to 255. It returns a probability for each option plus a confidence field. A score is an ordered grade with 2 to 10 levels, and it also returns probabilities and confidence. One call can carry 1 to 64 questions. Clef and Clef-flash accept the same request shape as Jev at /v1/systemone and return the same response shape. Input may be text, JSON, or an image. Which of those a local path actually accepts is a deployment fact. The hardware section keeps those limits separate.

Our decision-model client ships two built-in profiles. study_type is one choice question for the study type. tool_gate splits the gate. action is a choice about which action to take, and high_risk is a noul about whether the case is high risk. The split exists because the two judgments can both be true. Forced into one choice, the model can leave only a single label, and the other judgment has nowhere to go; split, each probability means one thing and can get its own threshold. Those last two points are our inference. The profile already stores them as two questions.

The official docs state that System One models do not write replies, produce code, or generate explanations of their reasoning. You will not get a paragraph about why a class was chosen. The distinguishing condition has to sit inside the option, so the probability refers to a sentence you wrote beforehand.

Vendor-reported: several questions can be asked in parallel on the same state, the questions are isolated from each other, and adding a question barely adds latency. We did not remeasure that on the Mac or on the workstation. The timings later in this part are for the study-type question alone.

Some jobs do not belong here. Writing, summarizing, extraction, open-ended reasoning, multi-step planning, and explaining a reason all need free text, and this model does not generate free text. A question whose options cannot be listed in advance is a poor fit as well. A choice allows at most 255 options. Treating a longer list as a two-stage question is our reading. The documented fact is the cap of 255.

A paper form that leaves one blank on each row, and moves on only after that blank is filled, is the same shape of work. The output space of a question is closed before you ask, and the model only assigns probabilities inside that space.

Listing options: mutually exclusive, plus an other

Options have to be mutually exclusive, and one of them has to be an other class for cases that fit none of the rest. In the study-type question that class is original_research, which is also the label used when the call fails.

Mutually exclusive means that, under the definitions you wrote, a document should land in only one option. Each option carries one sentence a reader can apply on the spot. The profile calls that sentence the criteria. The sentence should state the split, not just the name. Two splits used in this list are explicit randomization, and no new wet-lab or clinical data. The first separates a randomized trial from a human study that was not randomized. The second separates a purely computational paper from one that includes new experimental or clinical data.

If two definitions can both be true, a choice still returns a probability for every option, but those probabilities no longer answer which single class applies. That is our inference. Before you ship the question, check whether any two criteria can be true together. If they can, rewrite them or split the question.

The last bubble on a multiple-choice sheet, the one that says none of the above, is the same device. When the earlier classes miss, the model has a defined place to go instead of being pushed into a merely nearby label. original_research is that bubble: other original research that belongs to none of the first eight classes.

What the docs mean by no hallucination is that the output space is constrained by a schema in advance, so a type error cannot occur. The returned option is one you listed. Whether it is the right option is still a probability, handled by calibration and a confidence threshold. Calibration means the reported probability sits close to the rate at which the label is actually correct. That second half is our reading, not an accuracy guarantee.

On a failed call the client does not invent a label; it uses the on_fail label written on the profile, and for study_type that is original_research. How to store the failure itself comes in the last section.

Writing the nine-class study-type question

Write study type as one choice with nine mutually exclusive options, each with a sentence you can apply, and put papers that fit none of the first eight into original_research.

The table is the full study_type list. Read the right-hand column first and check that no two rows can both apply, then send the left-hand code as the option name. The criteria were written in English. The wording below follows the same conditions.

Put the right-hand sentence on the option. A bare code is not enough. If the option is only the letters rct, the model is never told that explicit randomization has to be visible, and the boundary with clinical_study disappears. If computational drops the phrase about no new wet-lab or clinical data, a modeling paper that also reports a new experiment has nowhere to land. The next three cases only apply those definitions. They are not a separate measurement. Human participants plus stated randomization go to rct, not to clinical_study. Cells or a cell-free system go to in_vitro, not to in_vivo. in_vivo requires a living animal, an intact organism, or an isolated intact organ. A purely computational paper with no new wet-lab or clinical data goes to computational.

The paper pipeline does not relabel papers that already have a study type. It calls this question only for new papers that lack one. If the confidence returned with the choice is 0.6 or higher, the pipeline keeps the decision-model answer. Below 0.6 it uses the regex. After the two sources are merged, accuracy on the 77 papers from 2026-10-11 is 90.9%.

On that same set of 77 PubMed abstracts and the same nine-class question, the decision model's own labels score 92.2%, and the regex alone scores 71.4%. A local 35B chat model is a different procedure. It answers only when its self-reported confidence is at least 0.75. Overall accuracy is then 50.6%, coverage is 51.9%, and on the papers it is willing to answer it is about 97.5%. All three procedures were measured that day, and the denominators are not the same. 92.2% uses all 77 papers. 97.5% uses only the subset the chat model answered, so it does not share a denominator with 92.2%. 50.6% is the overall figure once unanswered papers count. None of this is a general accuracy.

To check whether probabilities you already have sit close to the hit rate, use the calibration checker. It computes a Brier score, ECE, and a reliability diagram in the browser, and it does not upload the data. ECE is short for expected calibration error. The checker does not replace the 77-paper measurement. It only reruns the arithmetic on pairs you supply.

Running on a Mac and on NVIDIA: speed and agreement

On the 2026-10-11 measurements, assigning a study type took about 1.8 seconds per paper on a Mac, the NVIDIA workstation had a p50 of 266 milliseconds, and the labels on 77 papers agreed 77/77, with 92.2% accuracy on both.

We did not send the data to hosted Jev. The state would leave the machine, and the service is still early access. We also have no Jev credential, and we did not run a head-to-head of Jev against Clef. The Jev budget quoted below is from the docs. A millisecond figure for Clef-flash on another machine is marked vendor-reported. Locally we ran the open weights of Clef-flash. Clef is a 27B multimodal decision model, post-trained from Qwen3.8-27B. Clef-flash is the 9B version, post-trained from Qwen3.5-9B. Both add a joint schema head on the backbone. The license is Apache-2.0, and the weights are public on Hugging Face. We loaded Clef-flash, not the 27B Clef. It does not generate the answer word by word. It returns a probability for each option that was listed in advance.

A token is the unit the model uses to measure length. It is not one Chinese character, and it is not one English word. MLX 4-bit and GGUF Q4_K_M store the weights at lower precision so the model fits in memory. Read the table for each path's input cap, and for who measured the latency. Do not subtract one latency from another. The Workers AI row is documentation plus a vendor-reported number. The machine and the task are not our 77 abstracts.

The Mac path runs because oMLX already includes /v1/systemone and supports Clef natively. The weight name we used is clef-flash-4bit. That build requires a Bearer key. Its state cap is 16,384 tokens, which is not the 65,536-token context in the Workers AI docs. The NVIDIA path uses official llama.cpp. Native Clef support landed in PR #29831, merged on 2026-10-03. BF16 weights are about 18 GB and do not fit on a 16 GB GPU, so the workstation uses Q4_K_M rather than full precision. The complete Clef-flash files on Hugging Face are about 17.773 GiB, and the joint-head file is 243,538,016 bytes. That download size is not the quantized footprint in the memory note below. On the card, the Clef weights are about 5.6 GB.

The model card says Clef and Clef-flash accept text, JSON, and images. The docs also say Clef includes a vision encoder and can read images, while Jev currently accepts only a text state. Our llama.cpp path supports text only. Image input is not in llama.cpp yet. The Mac deployment notes do not say whether that path accepts images. This article does not fill that gap.

A p50 is the median. It stands for a typical paper in the batch, not the fastest one. On the NVIDIA workstation the study-type p50 is 266 milliseconds per paper. The Mac figure of about 1.8 seconds is for the same kind of question, with an observed range of about 1.5 to 2.4 seconds, plus about 7 seconds to load in-process. After both machines finished the 77 papers, labels agreed 77/77 and both accuracies were 92.2%. The mean gap between class probabilities was 0.005, and the largest gap was 0.217. Matching labels only mean both builds emitted the same class. One class probability can still differ by 0.217. One side is MLX 4-bit and the other is GGUF Q4_K_M. On these 77 papers the two quantized builds did not rewrite the labels. That is still not a general accuracy.

Do not leave both models on one 16 GB GPU

On a single 16 GB GPU, a resident chat model of about 12.2 GB, plus about 5.6 GB of Clef weights and about 0.9 GB of compute, exceeds the card. After the driver moves one of them into system memory, Clef goes from about 140 milliseconds to about 1.1 seconds. We switched instead of keeping both resident. Before a classification run, unload the chat model, then load Clef. The 140 millisecond figure is from that GPU note. It is not the study-type p50 of 266 milliseconds. When you read an NVIDIA latency, check whether another model was still on the card.

Do not let long inputs be truncated silently

When the use needs the full input, call with truncate=False and require the server to return 413 when the state is too long, rather than truncate it silently and answer anyway.

Paper classification and a personal-data audit are both that kind of use, because the judgment depends on the whole text. The Mac build caps state at 16,384 tokens. The NVIDIA path sets context to 4096 and requires the whole prompt to fit in one batch. Our inference: a long abstract is more likely to hit the tighter cap. Hosted Jev, in the docs, is a different budget: state and questions share about 32,000 tokens, and the documentation's own conversion is about 150,000 English characters. That budget is not the Mac and not the workstation.

Reading only the first half of an abstract, then still filling in a completed class on the form, is what silent truncation does. Our inference is that the model still returns probabilities that look normal, and the caller cannot tell that only the first half was read. Truncation has to become an explicit error. truncate=False is how an over-long state comes back as 413 instead of an answer after a cut.

The client uses only the Python standard library. With no backend or model configured, it sends no request and returns a failure. A failed call uses the profile's on_fail label. For study_type, that label is original_research. Our inference is that the error has to be stored beside the label. If a 413, or any other failure, leaves only original_research, a later reader will think the model made a judgment and that the paper simply fit none of the first eight classes.

Whether to use a decision model, and how, comes down to a few checks:

• Does the question fit: the options can be listed in advance, they are mutually exclusive, and you want probabilities rather than a paragraph. Writing, summarizing, extraction, open-ended reasoning, and multi-step planning sit outside that.
• Where the data goes: the state would leave the machine and the hosted service is still early access, so we ran open-weight Clef-flash and did not send data to hosted Jev.
• Mac: oMLX with clef-flash-4bit, a state cap of 16,384 tokens, and a Bearer key.
• 16 GB GPU: BF16 at about 18 GB does not fit, so use Q4_K_M with context set to 4096; if a chat model of about 12.2 GB is resident, unload it before classification.
• Long inputs: when the whole text must be read, set truncate=False so an over-long state returns 413.
• Pipeline threshold: keep the decision-model answer only at confidence of at least 0.6; otherwise use the regex, and only for new papers that do not yet have a study type.

These numbers come from the 77 papers and the deployment on 2026-10-11. They are not a template for another corpus or another machine.