Imagine a world-class chess grandmaster locked in a sensory deprivation chamber. They can think, analyze, and strategize at the highest level — but they can't see the board, can't move the pieces, and can't remember what happened in previous games unless someone reads the entire game history out loud before every move.
That's a large language model without a proper harness.
The Brain in the Vat Problem
LLMs are cognitively impressive but physically helpless. They cannot see your screen, execute code, open files, call APIs, or remember anything beyond what fits in their current context window. These aren't limitations of intelligence — they're limitations of embodiment.
In technical terms, the "body" we give an LLM is called a Harness: a complete system of perception, action, memory, and feedback that lets the model's reasoning actually accomplish things in the world.
The harness determines what the agent can perceive, what it can act on, how much it remembers, whether it can correct its own mistakes, and how gracefully it handles failures. It is, in the most literal sense, the difference between a brain and an agent.
The CPU and the Operating System
AI researcher @akshay_pachaar offers the clearest analogy:
LLM = CPU. Harness = Operating System.
Give an Intel processor DOS and you get a text terminal. Give it Windows 11 and you get a modern computing environment. Same silicon, radically different capability — because the operating system determines what the hardware can actually do.
The research evidence is compelling. @dotey's team ran a controlled experiment: improve the harness design within the LangChain framework, with no changes to the underlying model. The result? The same model jumped from 30th place to 5th place on TerminalBench. No new parameters. No additional training. Just a better body.
If that seems surprising, it shouldn't. We've been asking the wrong question for years. "Which model should I use?" is the wrong question. "What harness should I build?" is where the real leverage lives.
The Five Layers of a Complete Harness
A production-grade agent harness has five distinct components, and weaknesses in any one of them limit the whole system:
1. Sensory Input (Perception Layer) What the agent can see and read. Code files? Live data streams? Web content? Database records? Each additional perception channel expands the problem space the agent can operate in. An agent that can only read text lives in a fundamentally more limited world than one that can also analyze code structure, query databases, and monitor real-time events.
2. Motor Control (Action Layer) Tools are the agent's muscles. More tools mean more complex tasks are possible — but also more surface area for errors. Good harness design isn't just about adding tools; it's about defining clean interfaces so the agent knows when and how to use each one without ambiguity.
3. Memory Management The most overlooked layer, and frequently the most important. See below.
4. Feedback Loop Can the agent check its own work? Can it detect when it made an error and attempt correction? An agent with a well-designed feedback loop can recover from mistakes autonomously. One without it will confidently proceed down the wrong path until a human intervenes.
5. Fault Tolerance When things break — and in production, things always break — how does the system respond? Graceful degradation (reduced functionality but continued operation) beats catastrophic failure. This layer is often the difference between a demo that works and a system that's actually reliable.
Why Memory is the Make-or-Break Layer
Most agent failures trace back to memory. A well-designed harness implements three distinct memory types:
Short-Term Memory (Context Window): The current conversation and active context. Finite and expensive — every token costs. Good harness design carefully controls what occupies this space.
Long-Term Memory (Persistent Knowledge): Information written to databases or vector stores. The agent needs both the ability to write to long-term memory and the judgment to decide what's worth storing. Without the latter, memory systems become cluttered with noise.
Project-Level Memory (Task-Specific State): The intermediate steps, decisions, and partial results specific to the current task. Critical for multi-step work — and must be carefully isolated so state from one project doesn't contaminate another.
Three real-world harnesses make different bets on this tradeoff:
| Claude Code | Hermes | OpenClaw | |
|---|---|---|---|
| Memory approach | Context-only, no persistence | Self-managed MEMORY.md (~2,200 words) | Complex persistent layer |
| Best for | Single deep-focus tasks | Multi-model experiments, resource-constrained environments | Enterprise-scale, long-horizon workflows |
| Tradeoff | Simplest, but no cross-session continuity | Clean boundaries, lightweight | Maximum capability, maximum maintenance |
No single approach is universally best. The right choice depends on what the agent needs to remember and for how long.
A Cautionary Tale: MemPalace
The AI industry has a benchmark problem, and MemPalace illustrates it perfectly.
Last year, MemPalace claimed 100% benchmark accuracy in memory retrieval — an astonishing result that generated significant industry attention. The problem: they only tested information retrieval in isolation, not end-to-end task completion.
Worse, their AAAK compression algorithm actually dropped accuracy from 96.6% to 84.2% in practice. The benchmark looked spectacular. The real-world performance was worse than what it replaced.
The lesson isn't that MemPalace was dishonest — it's that component-level benchmarks mislead. What matters is end-to-end performance on real tasks. An agent that scores 99% on a memory retrieval benchmark but fails to correctly complete the full task is not a 99% agent. It's a failing agent with impressive-looking parts.
The Practical Implications
If you're building agents, these principles translate directly to decision-making:
Design the harness before you choose the model. Claude, GPT-4, Gemini — they're all capable enough for most tasks. The harness is where you'll spend your engineering effort, and getting it right matters more than model selection.
Invest heavily in memory architecture. The three memory layers must have clean boundaries. Without them, you get memory contamination — agents making decisions based on stale state from previous tasks, or confusing context between concurrent sessions.
Build feedback loops from the start. An agent that can detect and correct its own mistakes is dramatically more reliable than one that can't. This is not a feature to add later — it affects the entire architecture.
Evaluate end-to-end, not component-by-component. The only metric that matters is: does the agent complete the actual task reliably? Benchmark numbers on individual components can be actively misleading.
The Insight You Can't Unlearn
The five years of "bigger model = better agent" thinking has been costly. It led teams to spend millions on incremental model upgrades while ignoring architectural improvements that would have delivered an order-of-magnitude better performance for a fraction of the cost.
The leading agent teams today — the ones whose products actually work reliably in production — have shifted their attention. They're optimizing harness workflows, refining memory architectures, tightening tool interfaces, and engineering feedback loops. They're not waiting for the next model release.
The brain is already smart enough. The question is whether it has a good enough body to do the work.
References
- @akshay_pachaar. (2025). LLM-as-CPU, Harness-as-OS: A Framework for Agent Architecture. Social Media Technical Thread.
- @dotey. (2025). Harness Optimization Case Study: TerminalBench Benchmark Analysis. Social Media Technical Thread.
- LangChain framework architecture and harness design patterns. (2024–2025).
- MemPalace technical review: end-to-end evaluation critique. (2025).
- Comparative study: Claude Code vs. Hermes vs. OpenClaw architecture design. (2025).
- Three-layer memory system implementation patterns in agent design. (2025).
Found this useful?
Follow for new AI × biomedical research notes:
Or buy me a coffee to keep new content coming.
☕ Buy Me a Coffee