Cost Optimization illustration

Cost Optimization

Prompt caching vs raw tokens: what it really saves

Prompt caching can lower your bill, but the real savings depend on how much of your traffic is repeated context versus fresh output.

November 15, 2026 · 7 min read · By MyTokenTracker

Back to Blog

Prompt caching only helps when your requests reuse the same prefix, so the savings show up in repeated context, not in every token you send. If your workload is mostly fresh prompts or long generated answers, the bill barely moves.

The part developers miss is that caching changes the shape of cost, not the existence of cost. You still pay for uncached input, output, and any reasoning tokens, so the only way to know whether it matters is to measure your own mix of repeated versus unique tokens.

What prompt caching actually reduces

At a high level, caching is a reuse mechanism for prompt prefixes. The model does not need to reprocess the same system prompt, tool instructions, or long shared context every time if the provider recognizes it as cacheable. That means the repeated part of your input can become cheaper than sending the same text cold on every request.

But there are two important caveats. First, cached input only matters when the same prefix is reused often enough to be recognized. Second, output still bills normally, so a chatty agent that generates long answers can keep spending even if the prompt side is optimized.

If you want a mental model, think of caching as an input-side discount on repeated context. It is not a magic compression layer for the whole request, and it does nothing for a model that spends most of its tokens talking back to you.

Worked example: the arithmetic behind a cached request

Let’s use a simple example with gpt-4o, which is priced at $2.5 input and $10 output per 1M tokens. Suppose a request has 20,000 input tokens total, but 15,000 of those tokens are repeated prefix content that can be cached. The remaining 5,000 tokens are fresh input, and the model returns 2,000 output tokens.

Without caching, the input cost would be:

20,000 / 1,000,000 * $2.5 = $0.05

The output cost would be:

2,000 / 1,000,000 * $10 = $0.02

Total without caching:

$0.05 + $0.02 = $0.07

Now assume the repeated 15,000-token prefix is cached and the provider charges less for that reused portion. The exact cached rate depends on the provider and model, so the point here is not to invent a number. The important part is that only the repeated prefix is eligible for savings, while the 5,000 fresh input tokens and 2,000 output tokens still bill normally.

That means the practical question is not “Does caching exist?” It is “How much of my request is repeated prefix, and how often does it recur?” If the answer is “almost everything is unique,” caching won’t move the needle much.

Which workloads benefit most

Caching helps most when your app has stable, repeated scaffolding around a smaller changing core. That shows up in a few common patterns:

  • Long system prompts that stay the same across many requests
  • Agent loops that reuse the same tool instructions and project context
  • Document workflows where a shared base document is queried repeatedly
  • Multi-turn coding sessions with a persistent repo summary

It helps less when every request is a one-off, like ad hoc chat, short Q&A, or tasks where the prompt changes so much that there is no meaningful shared prefix. In those cases, you are paying mostly for fresh input and output, which caching does not eliminate.

There is also a behavioral trap. Teams sometimes add more context because caching makes it feel cheap. That can be backwards. If you stuff a prompt with extra repeated text just because it is cacheable, you may reduce the marginal cost of that prefix but still increase latency, complexity, and the uncached portion of the request.

How to think about savings with real price data

The cleanest way to reason about caching is to separate your bill into three buckets: repeated input, fresh input, and output. The AI Cost Index gives you a quick sense of the blended market baseline, with the Frontier basket at $4.64 per 1M tokens and the Budget basket at $0.74 per 1M tokens, both blended 3:1 input:output. Those numbers are useful for rough planning, but they do not tell you whether your own workload is cache-friendly.

Here is a small comparison of what changes and what does not:

Cost componentCaching helps?Why
Repeated prompt prefixYesThe same text can be reused instead of reprocessed cold
Fresh inputNoUnique tokens still need to be read
Output tokensNoThe model still has to generate them
Reasoning tokensNoThey are still part of the generation workload

If you want a more market-wide lens, the AI Cost Index is the easiest place to see how blended pricing behaves across the frontier and budget baskets. If you want to drill into specific models, the live catalog at model prices is the place to check current rates instead of guessing.

Why teams overestimate caching savings

People tend to overestimate caching for three reasons. First, they assume all input is equally reusable, which is rarely true. Second, they focus on prompt length instead of prompt repetition. A long prompt is not automatically cacheable. Third, they forget that output often dominates once the model starts drafting code, explanations, or multi-step reasoning.

Another subtle issue is that caching can make usage feel cheaper without making it structurally cheaper. If your agent is still taking the same number of turns, producing the same amount of output, and retrying on failures, the bill can stay stubbornly high even when the repeated prefix is discounted.

That is why a useful metric is not just cache hit rate, but cache hit rate alongside total output volume and success rate. A high cache hit rate with lots of retries is still a bad deal.

How to track this in your own stack

The easiest way to see whether caching is actually helping is to measure real usage, not just read provider docs. MyTokenTracker reads the usage logs coding agents already keep locally and prints a 30-day cost table at API prices, with usage data never leaving your machine. If you want to sync to a dashboard, you can create a free account and run npx mytokentracker init, which uploads only daily totals per agent and model, never prompts, code, or file names.

It works with Claude Code, Codex, GitHub Copilot CLI, Antigravity, OpenCode, Amp, Qwen Code and more. For drop-in app instrumentation, the wrappers auto-capture OpenAI, Anthropic, Gemini, and Mistral, and any other provider can be sent with a single POST to the events API. That makes it much easier to answer a practical question: how much of your bill is repeated context, and how much is actual generation?

If you want the broader ecosystem view, the State of AI page is a good companion read. If you want to compare value instead of just raw price, the value-for-money view is the right lens.

FAQ

Does prompt caching reduce output costs?

No. Caching is an input-side optimization for repeated context. Output tokens still bill normally, so long generations can remain expensive even with a strong cache hit rate.

Is caching worth it for short prompts?

Usually not by much. Short prompts do not have much repeated prefix to begin with, so the savings are often too small to matter unless the request volume is very high.

What should I measure before I optimize for caching?

Measure repeated prefix length, total input tokens, output tokens, retries, and success rate. If you only look at one number, you can end up optimizing the wrong part of the workload.

If you want to see what your own token mix looks like, run npx mytokentracker and check the 30-day table first.