Every time you send a message to an AI, the system re-reads everything you've ever said in that conversation. Imagine calling customer support and having to re-introduce yourself from scratch every single time. Prompt caching lets AI remember what it's already read and only process what's new — cutting costs down to one tenth of the original.
The Hidden Cost of Repeated Reading
AI services bill a bit like a copy shop — you pay per "word processed." Every time you send a message, the AI doesn't just handle your new input. It reprocesses the entire conversation history, system instructions, tool definitions, and all other fixed preamble from scratch.
The longer the conversation, the higher this "repeated reading" cost gets. A conversation with detailed system instructions might see that fixed preamble — content that never changes — accounting for 80% of every single request's cost.
In multi-turn conversations, long document analysis, or scenarios requiring large amounts of background knowledge, this waste compounds quickly.
How Caching Works: Read Once, Remember Forever
The concept behind prompt caching is elegantly simple: if a piece of content has already been processed and hasn't changed, don't process it again.
Prompt caching logic: prefix matching vs cost structure comparison
Technically, the AI provider stores the intermediate result of processing your prompt prefix — system instructions, tool definitions, and other stable preamble — after the first computation. When you send a new message, the system loads those intermediate results directly from cache and only processes the new portion.
Anthropic's Claude API offers two modes: automatic caching (the system decides what to cache) and manual checkpoints (developers mark exactly where cache breakpoints should go). Manual mode supports up to 4 cache breakpoints, giving fine-grained control.
How Much Can You Save?
The numbers are concrete: cache-hit tokens are billed at just 10% of the base price.
Here's what that means in practice. If your system instructions contain 5,000 tokens and you run 20 conversation turns, without caching those 5,000 tokens get billed 20 times. With caching, the first turn costs the full amount (plus a 25% cache-write surcharge), and the remaining 19 turns each cost only one tenth.
The result: just the system instruction portion costs more than 85% less. If your stable prefix is longer — say, you're loading an entire product specification document — the savings go even higher.
Latency drops in parallel. Content that doesn't need reprocessing means response time drops significantly. In real-time interaction scenarios, users perceive the AI as noticeably "thinking faster."
Getting the Most Out of Caching
Turning on caching isn't enough — putting it in the wrong place actually wastes money. A few key principles:
Stable content goes first: System instructions, role definitions, and tool descriptions that almost never change should sit at the very beginning of your prompt. Caching uses prefix matching — it compares from the start and stops the moment it hits a difference.
Dynamic content goes last: The parts that change every turn — user messages, fresh query results — go at the end. That way the stable prefix in front keeps hitting the cache.
Don't switch models mid-conversation: Changing models invalidates the cache entirely because different models' intermediate representations are incompatible.
Respect minimum cache lengths: Not everything is eligible for caching. Claude Opus 4.6 requires at least 4,096 tokens to enable caching; Sonnet 4.6 and Haiku 4.5 require 2,048. A prefix that's too short isn't worth caching.
The Bigger Picture
Prompt caching is central to the "set good boundaries" strategy in AI application engineering. Anthropic recommends developers think in three layers: leverage what Claude already knows, reduce unnecessary human intervention, and design system boundaries carefully — cache optimization belongs to that third layer.
The significance goes beyond cost savings. Caching lets developers confidently load far more background knowledge into prompts (it's nearly free once cached), which directly improves AI answer quality. Context that used to get trimmed for cost reasons can now be included in full.
More context → better answers → more users → lower per-query cost. That's the positive feedback loop caching creates.
References
- Anthropic (2026). Prompt Caching. Claude API Documentation. https://docs.anthropic.com
- Martin, L. (2026). Harnessing Claude's Intelligence: 3 Key Patterns for Building Apps. Anthropic Blog.
- Kwon, W. et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
Found this useful?
Follow for new AI × biomedical research notes:
Or buy me a coffee to keep new content coming.
☕ Buy Me a Coffee