TL;DR: In 2026, open-source models above 20B parameters run smoothly on consumer hardware. Plug them into an agent framework, though, and tool-call success rates often drop below 50%. After reviewing 27 top-venue papers, the pattern is clear: the bottleneck is in the framework, not the model.
The problem
Gemma4 26B on an M4 Pro with 48GB of unified memory. Single-turn Q&A works well. Reasoning quality is solid. But drop it into an agent system that needs tool calls and multi-turn conversations, and things fall apart: malformed JSON, misspelled tool parameters, instructions ignored after the third turn.
The instinct is to blame model size or quantization loss. But on the Berkeley Function Calling Leaderboard, xLAM-2-8b at just 8B parameters already tops the function-calling category (Zhang et al., 2024). Gemma4 26B exceeds that benchmark in both parameter count and instruction following. Model capability is not the issue.
Across 27 papers published between 2024 and 2026 at ICML, ACL, NeurIPS, and related venues, the failure modes almost always trace back to framework-level issues: prompt formatting, context management, and tool architecture.
One token in the prompt can break the entire tool chain
Ask a model to return JSON, and the output looks like JSON but fails the parser. The cause is surprisingly specific.
A 2026 study (arXiv:2605.02363) measured how prompt formatting affects structured output. Gemma models with a markdown code fence (```json ```) in the prompt produced valid structured output 0% of the time. Remove the code fence, and success jumps to 80%. Llama models show a similar pattern with LaTeX symbols: removing them raises the rate from 8.5% to 39%. The model can write JSON. Certain prompt tokens interfere with its output distribution.
The fix is straightforward. XGrammar (arXiv:2411.15100) is a structured decoding engine that constrains inference output to match a given JSON Schema. Success rate: 100%. Latency overhead: near zero. Speed: 100x faster than Outlines. Not a fancy algorithmic breakthrough. Just a grammar constraint layer in the inference pipeline. But it eliminates format failures entirely.
The longer the conversation, the less the model hears you
With formatting solved, a harder problem remains: semantic loss in multi-turn conversations.
Instruction comprehension drops by an average of 39% after multiple turns (arXiv:2505.06120). The model doesn't forget. The context window fills up with system prompts, tool definitions, and conversation history, pushing the actual instructions to the margins of the model's attention.
RECAP (arXiv:2509.04472) takes a pragmatic approach: at the end of each turn, rewrite the user's current intent as a single sentence. Subsequent turns condition on that sentence alone, not the full history. The bottleneck shifts from "how much the model can remember" to "how well the summary is written," and the latter is far easier to control.
A subtler form of waste hides in user preference files. Workflow rules, preference configs: these occupy substantial context and repeat every turn. POPI (arXiv:2510.17881) compresses them into 50-100 token summaries injected at inference time. Thousands of tokens freed up for the instructions that actually matter.
Compressing the system prompt itself also helps. LLMLingua-2 (Pan et al., 2024, ACL) uses token classification to shrink prompts by 2-5x with minimal task performance loss. Even with the 26B model's larger context window, compression measurably improves multi-turn stability, suggesting that context waste is independent of window size.
When the tool set runs out
The first two problems have clear solutions. The third is more open-ended: agent systems define a fixed set of tools, and user needs inevitably exceed that set.
One approach: stop insisting on predefined tools. CodeAct (Wang et al., 2024, ICML) lets the model generate Python code and execute it locally when no matching tool exists. Performance beats fixed JSON action formats by 20%, with zero tool registration overhead (arXiv:2402.01030). The tool set expands from a static list to the entire Python ecosystem.
ToolMaker (ACL 2025, arXiv:2502.11705) goes further: given a GitHub repository URL, it automatically installs dependencies and generates Python wrapper functions at an 80% success rate. It can also scan locally installed Python packages and convert them into callable tools. The trajectory is clear: from manually maintaining tool lists to automated discovery and generation.
The tradeoff is a blurred security boundary. A model running arbitrary Python locally can cause irreversible damage with a single os.remove. Production deployments need sandboxing at minimum, restricting filesystem and network access.
Is fine-tuning worth the effort?
Everything above works without touching model weights. What if you're willing to fine-tune?
One number worth considering: OPT-350M, a 350M-parameter model, reaches 77.55% tool-call accuracy after task-specific SFT, compared to 26% for ChatGPT with Chain-of-Thought prompting (arXiv:2512.15943). Training data can be synthesized automatically via ToolACE (arXiv:2409.00920), no manual labeling required. QLoRA fine-tuning on Apple Silicon is already a mature workflow. If 350M gets there, the headroom for a 26B model fine-tuned on a specific tool set is considerable.
Further out, SMITH (arXiv:2512.11303) merges tool libraries and memory systems into a unified architecture where agents dynamically create tools and share experience across tasks. Promising on benchmarks (GAIA 81.8% Pass@1), but still distant from production.
Others have tried splitting the problem: one model plans, another calls tools, a third summarizes results (arXiv:2401.07324). Reduces cognitive load per model but adds system complexity and latency. Whether that trade works for local deployment depends on how much inference overhead you're willing to pay for stability.
For now: clean up prompt formatting, add structured decoding, compress system prompts. No model swap, no fine-tuning, verifiable the same day. Fix the framework debt first. The model can wait.
FAQ
Q: What hardware do I need to run a 26B model on Apple Silicon? A: M4 Pro or M4 Max with at least 32GB of unified memory. 48GB comfortably runs a Q4-quantized 26B model; MLX inference runs at roughly 20-40 tokens per second.
Q: Does structured decoding slow down inference? A: XGrammar's overhead is near zero, running 100x faster than Outlines. On a 26B model, the difference is imperceptible.
Q: Are these techniques macOS-only? A: Structured decoding and prompt compression are platform-agnostic. MLX is Apple-exclusive; Windows and Linux can use llama.cpp or vLLM instead.
Q: Can these paper results be applied directly to production? A: Most studies use controlled benchmarks that differ from real-world conditions. Structured decoding and prompt compression have mature open-source implementations you can verify today. CodeAct and ToolMaker's security boundaries still need evaluation based on your deployment context. This article is a reading digest, not a systematic literature review.
References
- (2026). When Correct Isn't Usable: Structured Output Reliability in LLMs. arXiv:2605.02363
- Wang et al. (2024). XGrammar: Flexible and Efficient Structured Generation Engine. arXiv:2411.15100
- Wang et al. (2024). CodeAct: Executable Code Actions Elicit Better LLM Agents. ICML 2024. arXiv:2402.01030
- Pan et al. (2024). LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression. ACL 2024 Findings. arXiv:2403.12968
- Zhang et al. (2024). xLAM: A Family of Large Action Models to Empower AI Agent Systems. arXiv:2409.03215
- Wang et al. (2025). RECAP: Rewriting Conversations for Accurate Intent Preservation. arXiv:2509.04472
- Qian et al. (2025). POPI: Personalizing LLMs via Preference Inference from Conversations. arXiv:2510.17881
- Yuan et al. (2025). ToolACE: Winning the Points of LLM Function Calling. arXiv:2409.00920
- Shi et al. (2025). SMITH: Self-Evolving Agent with Dynamic Tool Creation and Memory. arXiv:2512.11303
- Shi et al. (2025). LLMs Get Lost in Multi-Turn Conversation. arXiv:2505.06120
- (2025). Small LLMs for Efficient Agentic Tool Calling. AAAI 2026 Workshop. arXiv:2512.15943
- Liu et al. (2025). ToolMaker: LLM Agents Making Agent Tools. ACL 2025. arXiv:2502.11705
- (2024). Small LLMs Are Weak Tool Learners: A Multi-LLM Agent. arXiv:2401.07324
CYHsieh | 2026-05-11 Reading notes from 27 papers (2024-2026). Not a systematic literature review.
Found this useful?
Follow for new AI × biomedical research notes:
Or buy me a coffee to keep new content coming.
☕ Buy Me a Coffee