Prompt caching is one of the few levers that can materially lower AI spend without changing the model. If your app or coding agent keeps resending the same system prompt, tool schemas, or repo context, caching turns repeated input from a cost multiplier into a much smaller line item.
The catch is simple: caching only helps when your workload repeats enough context to matter. If every request is unique, the savings are tiny. If you’re running an agent loop with the same instructions and project context over and over, the difference can be huge.
Why caching matters more than most people think
Most developers look at model choice first, which is fair, but it’s not the whole bill. A coding agent can burn through input tokens faster than you expect, especially when it keeps reattaching the same project instructions, tool definitions, and file summaries. That repeated input is exactly where caching can help.
Think of it like this: if your agent sends a 20,000-token context block on every turn, and 18,000 of those tokens are effectively unchanged, you are paying to repeat yourself. Cache those repeated tokens once, and you stop paying full freight every time.
That’s also why the AI Cost Index is useful. It gives you a blended view of price pressure across the market, but your real bill depends on how much of your traffic is reusable versus genuinely new.
The token math, with a real example
Let’s do the arithmetic on a simple repeated-workload example. Suppose an agent makes 100 calls in a day. Each call sends 10,000 input tokens, but 8,000 of those tokens are identical every time, and only 2,000 are fresh task-specific context.
If the model is gpt-4o, input is $2.5 per 1M tokens and output is $10 per 1M tokens. We’ll ignore output for a moment and just look at input because that’s where caching helps most.
Without caching:
100 calls × 10,000 input tokens = 1,000,000 input tokens
Cost = (1,000,000 / 1,000,000) × $2.5 = $2.50With caching on the repeated 8,000-token block:
Repeated tokens = 100 × 8,000 = 800,000 tokens
Fresh tokens = 100 × 2,000 = 200,000 tokens
If the repeated block is billed at a much lower cached rate, the savings stack up fast.
Even a partial reduction on 800,000 tokens can move the daily bill materially.The exact savings depend on the provider’s cache pricing, but the structure is what matters: repeated context is the part worth attacking first. If you’re not measuring repeated input separately, you’re guessing.
Now compare that with deepseek-chat, where input is $0.28 per 1M tokens and output is $0.42 per 1M tokens. The same 1,000,000 input tokens cost:
(1,000,000 / 1,000,000) × $0.28 = $0.28That’s already cheap, so caching on top of that may still help, but the absolute dollar win is smaller. This is the part people miss: caching is most valuable when repeated input is large enough that the base rate still hurts.
What actually gets cached in real agent workflows
In practice, the best caching candidates are boring and repetitive:
- System prompts that define behavior and output format
- Tool schemas and function definitions
- Repository summaries and long-lived project instructions
- Repeated policy text or safety wrappers
- Static background context in multi-step agent loops
What usually does not cache well is the part you care about most: the changing task, the latest diffs, the new logs, the fresh user request. That means you should separate stable context from dynamic context in your architecture. If you jam everything into one giant prompt blob, you make caching harder and you also make debugging cost harder.
For a broader view of what’s expensive across the market, check the live model prices. It’s easier to decide whether caching or model switching is the bigger win when you can see the current input and output rates side by side.
When caching is worth the engineering time
Not every workload deserves a caching pass. Here’s a practical way to think about it.
| Workload shape | Cache value | Why |
|---|---|---|
| One-off chat or ad hoc prompts | Low | Little repeated input to reuse |
| Agent loop on one codebase | High | Same instructions, tools, and repo context recur constantly |
| Multi-agent orchestration | Very high | Shared prompts and schemas repeat across agents |
| Short-form completion API | Medium | Depends on how much fixed context you prepend |
If your app is mostly interactive and short-lived, you may save more by trimming prompts than by implementing cache logic. If you are running a long-lived coding agent, caching is usually one of the highest-ROI optimizations you can make.
It’s also worth comparing caching against model choice. A cheaper model with no caching may still lose to a pricier model with a highly reusable prompt structure, especially when the workload is dominated by repeated context rather than reasoning.
How to track this without guessing
You cannot optimize what you do not measure. The useful metrics are not just total tokens, but input, output, cache, and reasoning tokens, plus latency and success rate. You want to know which provider, model, platform, and use-case is generating repeated context and which one is actually finishing tasks.
That’s exactly why MyTokenTracker exists. Run npx mytokentracker to read the usage logs your coding agents already keep locally and print a 30-day cost table at API prices, with usage data staying on your machine. If you want a dashboard, create a free account and run npx mytokentracker init to upload daily totals per agent and model, then sync every 30 minutes. It works with Claude Code, Codex, GitHub Copilot CLI, Antigravity, OpenCode, Amp, Qwen Code and more.
For the bigger picture, the public State of AI page helps you see where the market is moving, and the open data set lets you build your own analysis if you want to go deeper than a dashboard.
What to change first if your bill is too high
If your spend is creeping up, start with the boring fixes before you chase model swaps:
- Separate static instructions from dynamic task context.
- Stop resending large repo summaries unless they actually changed.
- Deduplicate tool schemas and shared agent prompts.
- Measure repeated input by model and use-case.
- Check whether the expensive part is input, output, or reasoning.
Once you know where the repetition is, you can decide whether to cache, compress, or redesign the workflow. In many cases, the cheapest token is the one you never resend.
If you want a value-for-money lens instead of just raw price, the value view is a good place to sanity-check whether your current setup is paying for quality you actually use.
FAQ
Does prompt caching help every model equally?
No. The benefit depends on how much repeated input your workload sends and how the provider prices cached versus uncached tokens. If your prompts are mostly unique, caching won’t move the needle much. If your agent reuses the same instructions and context over and over, it can cut a meaningful chunk of input spend.
Is caching better than switching to a cheaper model?
Sometimes yes, sometimes no. If most of your cost comes from repeated context, caching can beat a model switch because it attacks the waste directly. If your workload is dominated by fresh reasoning or long outputs, changing models may matter more.
What should I measure before I optimize?
Measure input tokens, output tokens, cache tokens, reasoning tokens, latency, and success rate by model and use-case. Without that breakdown, you’re guessing which lever will save money and which one will just add complexity.
If you want to see where your repeated context is really going, try MyTokenTracker and look at the last 30 days before you change a single prompt.