Ask before the agent acts: gate tool calls with a decision model

Before an agent runs a tool, hand the request and a set of pre-listed options to a decision model, get a probability for each option, and let code decide whether to run it, block it or ask a person. This part covers how we designed that checkpoint, the results on 20 test calls, and whether to choose a hosted service, open weights or raw logits.

Part 5 of the series | This part answers: should a decision model guard tool calls, and should you pick a hosted service, open weights, or read logits yourself?

Why an agent should ask before it acts

Before a tool call runs, it should get probabilities over a fixed set of options and let code decide what to do; the decision model supplies probabilities and never rewrites policy.

Agents delete files, send requests and read configuration, and some of those actions cannot be undone. A decision model suits a pre-execution check because it stops after one forward pass, writes no text and leaves nothing to parse (parts 1 and 2 cover the definition and mechanism).

Our decision-model client splits the gate into two questions. action is a choice with four defined options: allow is safe and reversible, or a local read the policy explicitly permits; deny violates standing policy, such as destructive deletion, secret exfiltration or unauthorized access; escalate is a borderline or high-impact case that needs a person or policy owner; ask_user means consent is missing or intent is unclear, so ask the user first. high_risk is a noul asking whether running the request could cause data loss, external side effects, credential exposure or an irreversible change. If the call fails, it falls back to action escalate and high_risk true.

Think of four pre-printed forms at a gate: the guard can stamp one of them, but cannot write a new rule on the spot.

The default is shadow mode, which logs without blocking; enforce needs two explicit switches, and any error fails closed (treated as not allowed). Tool input goes only to a local endpoint by default, file content is cut to 2,000 characters, and the shadow log keeps 200. One point is easy to miss: "is it loopback" is not the same as "does the data stay on this machine"; a local port that forwards to another machine is always treated as remote.

Rules first; the model may only tighten

Standing rules set the floor; the decision model may make a verdict stricter, never looser.

Strictness runs allow, then escalate and ask_user (same level), then deny; the merged verdict is the stricter of rule and model, and high_risk never drops below the rule's value. The rule came from a code review: an early draft let the model loosen escalate to allow, and review removed that.

The rule floor is an adjacent-token regular expression plus a linear token scan; any hit means deny, and every match runs in linear time. It went through six consecutive rounds of code review covering truncation blind spots, ReDoS, padding bypasses and parser differentials. Even so, it is a best-effort blocklist, not a security boundary.

PII columns can be added, never removed

The same tighten-only principle applies to auditing personal-data columns: the decision model casts a second vote that can only add suspected columns, never remove one. On the hard synthetic set from 2026-10-11 (disguised column names), the decision model alone caught 5/5; identifying columns scored 0.398 to 0.977 and non-identifying columns at most 0.032. The threshold has to be 0.3, because the ref_no column scored only 0.398. It takes about 1.2 seconds per column.

Results on 20 tool calls

On 20 synthetic tool calls on 2026-10-11, action accuracy was 85% with 0 critical misses; this describes that synthetic set only and is not a general accuracy.

The setup was a Mac running Clef-flash with MLX at 4-bit and an NVIDIA workstation running the GGUF Q4_K_M build with llama.cpp. high_risk accuracy was 90% on the Mac and 95% on NVIDIA. A critical miss is a call marked must-not-allow whose answer is deny, escalate or ask_user but which the model allowed; one such call counts. p50 latency was 826 ms in process on the Mac and 143 ms on NVIDIA; over HTTP end to end, 20/20 calls succeeded.

One call on either side of 0.5

The two machines differed on a single call: its high_risk probability was 0.4883 on one and 0.5039 on the other, only about 0.016 apart but on opposite sides of the 0.5 threshold, so the labels differed. Our inference: verdicts near a threshold flip easily, so cases close to it should take the stricter path. To check whether your own model's probabilities match its hit rate, use the calibration checker, which computes Brier, ECE and a reliability diagram in your browser without sending data.

Routing, search and compaction

These uses share one shape: the decision model answers a few small typed questions and code takes the action; that summary is our inference. The descriptions follow each project's README and code; we did not run them.

• pi-jev-router: a model-routing extension for the Pi coding agent. Jev answers only tier, standalone and high_consequence; a local deterministic policy then decides whether to switch models based on availability, context capacity and cost. The code pins jev-1.13.0.
• jev-search: uses Jev to understand a query, choose sources and a time window, and score result relevance with noul; the interface returns only links and snippets, never a generated answer.
• fast-jev-compaction: asks a noul question for every tool call and result in a conversation to keep, truncate or drop it, without rewriting user or assistant text.
• jev-security-scan: a standard-library Python CLI that can optionally call Jev to check agent skills, MCP configuration and source code for suspicious behavior before installation, reporting line numbers and redacted evidence.

Choosing a model: hosted, open weights or raw logits

If data must stay on your machine, self-host open weights or read an existing model's logits; only when the state can leave do hosted Jev and Workers AI become options.

We chose not to send data to hosted Jev and to run open-weight Clef-flash on our own machines instead, because the state would leave the machine and the service is still early access. The table reflects October 2026; numbers marked vendor- or author-reported are not our measurements, and we have no Jev credentials. "Depends on deployment" is our inference: those rows are public weights or code you can host yourself.

Details the table cannot hold: TypeSafe reports $0.042 per million input tokens with free output, and its "193.6× faster, 444.6× cheaper" headline comes from in-house workflow evals that the announcement says may be optimistic; per its model card (vendor-reported), d1-3B's license excludes commercial use at annual revenue of $10 million or more; in Cloudflare's Decision Index 0.2.1 table (H200 internal run, vendor-reported), BFCL is 98.8 for Clef-flash and 95.8 for Jev, and GPQA Diamond is 78.3 for Jev and 48.0 for Clef. TypeSafe's privacy policy says user input is not used to train models, but input is still processed by the hosted service.

A last point, also our inference: calibrating each question on its own does not make a guardrail that chains several questions calibrated as a whole; validate thresholds on the whole pipeline, not one question at a time.