Cache hit rate is one of the fastest ways to lower an AI bill, because cached input is usually the difference between “this agent feels expensive” and “why did that run barely cost anything?” If you’re using coding agents or multi-turn workflows, you should be watching cache hits alongside raw token counts.
Why cache hit rate matters more than most teams think
Token totals tell you how much text moved through a model. Cache hit rate tells you how much of that text the provider could reuse instead of charging you for it again at full input rates. That matters most in developer workflows, where the same system prompt, repo map, instructions, and tool context get sent repeatedly.
In practice, cache hit rate is a multiplier on everything else you already track. If a workflow has a lot of repeated context, then improving cache reuse can reduce the billed input side without changing the actual work. That is why two agents with the same output quality can have very different costs.
If you want the broader market context, the AI Cost Index is the quickest way to see whether model pricing is moving in your favor or not. But for your own bill, cache behavior often matters more than headline prices.
How cache hit rate changes the math
Let’s keep the arithmetic simple. Suppose a coding agent sends 1,000,000 input tokens in a month, but 600,000 of those tokens are repeated context that gets cached. That means only 400,000 tokens are effectively uncached on the expensive path.
Now compare two models using the real per-1M-token prices from the current catalog:
| Model | Input price per 1M | Output price per 1M | What cache changes |
|---|---|---|---|
| gpt-4o | $2.5 | $10 | Repeated input context can become much cheaper when cached |
| claude-sonnet-4-5 | $3 | $15 | Same story, but the savings scale with how much repeated context you send |
| deepseek-chat | $0.28 | $0.42 | Lower baseline input cost means cache still helps, but the absolute dollars are smaller |
Here’s the worked example. If your workflow uses gpt-4o and you send 1,000,000 input tokens plus 100,000 output tokens in a month, the uncached-cost baseline is:
Input: 1,000,000 / 1,000,000 * $2.5 = $2.50
Output: 100,000 / 1,000,000 * $10 = $1.00
Total = $3.50If 60% of the input is effectively reused through caching, the input side that actually needs fresh billing pressure is closer to 400,000 tokens. The same output stays the same, so the new arithmetic becomes:
Input: 400,000 / 1,000,000 * $2.5 = $1.00
Output: 100,000 / 1,000,000 * $10 = $1.00
Total = $2.00That is a $1.50 difference on the same work, just from repeated context being reused. The exact billing mechanics depend on the provider and the endpoint, but the directional lesson is stable: repeated prompt material is a tax, and cache hit rate is the rebate.
What actually drives cache hit rate in coding agents
Cache hit rate is not random. It is mostly shaped by how stable your prompts and context are from one request to the next. In coding tools, the usual suspects are easy to spot:
- Long system prompts that barely change.
- Repository summaries or file trees that are resent every turn.
- Tool instructions that stay identical across tasks.
- Agent loops that keep reloading the same context after every action.
- Chatty workflows where the model is asked to “look again” with almost the same prompt.
That means the best cache strategy is often not “use less AI,” it is “stop changing the boring parts.” If your agent can keep stable instructions stable, then the provider has a chance to recognize repeated input and reuse it. That is especially relevant in long-running dev sessions, where the first few turns may be expensive but later turns should get cheaper if the context stays well structured.
For a broader view of how expensive different models are when you normalize the math, the live model prices page is the place to sanity-check your assumptions before you wire a new tool into production.
Cache hit rate vs raw token count
Raw token count answers, “How much did we send?” Cache hit rate answers, “How much of that was actually new?” Those are related, but they are not the same metric, and treating them as the same is how teams end up with confusing invoices.
Here’s a simple comparison:
| Metric | What it tells you | Why it matters |
|---|---|---|
| Input tokens | Total prompt volume sent to the model | Useful for scale, but it overstates cost when context repeats |
| Output tokens | Total generated text | Usually the easiest part of the bill to explain |
| Cache hit rate | How much repeated input was reused | Directly impacts how much of the prompt side is effectively discounted |
| Success rate | Whether the task actually worked | A cheap run that fails is still wasteful |
This is why cost analysis gets more useful when you stop looking at totals in isolation. A workflow with a higher token count can still be cheaper if it has a strong cache hit rate and fewer retries. That is also why cost per successful task is often the metric that tells the real story, not just raw spend.
How to improve cache hit rate without making prompts worse
You do not need to turn your prompts into cryptic one-liners. You just need to make the repeated parts repeatable. A few practical patterns help a lot:
- Keep system instructions stable across sessions.
- Separate reusable policy text from task-specific text.
- Send compact repo summaries instead of dumping the whole tree every turn.
- Avoid rewriting the same instructions with tiny wording changes.
- Group related work so the agent can reuse context within a session.
There is a tradeoff, of course. Over-optimizing for cache reuse can make prompts harder to read or debug. But in most coding workflows, the best version is simple: stable scaffolding, fresh task details, and less accidental churn. That gives the model enough context to be useful while increasing the odds that repeated material is treated as repeated material.
If you want a live view of how pricing and model choice interact with these habits, the public State of AI page is a good companion to your own logs. It helps answer the bigger question: are you paying more because the model is pricier, or because your workflow is noisy?
How to track this
The easiest way to measure cache hit rate in your own workflow is to track real usage locally, then compare runs over time. MyTokenTracker is built for exactly that. Run npx mytokentracker and it reads the usage logs your coding agents already keep on your machine, then prints a 30-day cost table at API prices. Your usage data never leaves the machine.
If you want a dashboard, create a free account and run npx mytokentracker init. It asks for the API token, uploads history, then syncs every 30 minutes. It works with Claude Code, Codex, GitHub Copilot CLI, Antigravity, OpenCode, Amp, Qwen Code, and more. Only daily totals per agent and model are uploaded, never prompts, code, or file names.
For deeper analysis, the product also tracks cost, input/output/cache/reasoning tokens, latency, and success, broken down by provider, model, platform, and use-case. That makes it easier to see whether a workflow is expensive because of volume, retries, or poor context reuse. If you care about open data, the catalog is published under CC BY 4.0, and the broader dataset is available through the open data page.
FAQ
Is cache hit rate the same as prompt caching?
No. Prompt caching is the mechanism, cache hit rate is the outcome you measure. The mechanism is what the provider does with repeated context, while the hit rate is how often your repeated material actually gets reused.
Does a higher cache hit rate always mean lower cost?
Usually, yes on the input side, but the full answer depends on output volume, retries, and whether the task succeeds. A cached prompt that produces a bad answer can still be a bad deal if you need to rerun it three times.
What should I watch besides cache hit rate?
Watch input tokens, output tokens, latency, and success rate together. Cache hit rate explains reuse, but those other metrics tell you whether the workflow is actually efficient and reliable.
If you want to see your own cache behavior instead of guessing, install MyTokenTracker and start with the local 30-day cost table.