Prompt Caching: What Actually Lowers Your AI Bill

Prompt caching only lowers your AI bill when you reuse the same input context enough times to matter. If your workload is mostly one-shot prompts, the savings are smaller than the hype makes it sound.

The useful way to think about it is simple: cached tokens change the price of repeated context, not the price of every request. That means the real question is not “does caching exist?” but “how much of my traffic is repeated input, and how often does it get reused?”

What prompt caching actually changes

Most AI spend comes from tokens, but not all tokens are equally expensive in practice. When a system can reuse previously processed prompt context, it avoids paying full freight for that repeated input every time. That matters most for tools that keep long instructions, repository context, or agent memory in every call.

For developers, the biggest wins usually show up in workflows like:

• Code assistants that resend the same system prompt on every turn
• Multi-step agents that keep large project context attached to each request
• RAG pipelines that repeatedly include the same retrieved chunks
• Tool-using workflows that rehydrate the same instructions across many calls

What caching does not do is magically reduce output cost. If the model still generates the same amount of text, that output is billed as output. Caching is about avoiding repeated input work, not making the model cheaper in every direction.

If you want a broader view of how pricing behaves across models, the public AI Cost Index is a better starting point than guessing from vendor marketing pages.

Worked example: when reuse starts to matter

Let’s use a concrete, boring example, because that is where the math gets honest.

Say your app sends a 20,000-token prompt prefix on every request, and 18,000 of those tokens are repeated instructions, docs, or project context. The remaining 2,000 tokens are genuinely new user content. You make 100 requests in a day.

If that repeated prefix is not cached, your input load is:

20,000 tokens/request × 100 requests = 2,000,000 input tokens

Now suppose the repeated 18,000-token prefix can be reused. Your uncached input becomes:

2,000 tokens/request × 100 requests = 200,000 input tokens

That is a reduction of 1,800,000 input tokens for the day. To turn that into dollars, you need the model’s input price per 1M tokens. For example, gpt-4o-mini is $0.15 input per 1M tokens, while gpt-4o is $2.5 input per 1M tokens.

So the avoided input cost on gpt-4o-mini would be:

1.8 × $0.15 = $0.27 saved

On gpt-4o, the same avoided input would be:

1.8 × $2.5 = $4.50 saved

That is the whole game. Caching is more valuable when the repeated context is large, when request volume is high, and when the underlying model has a higher input price. If your prompts are short, or the shared prefix changes constantly, there is not much to squeeze out.

Which models make caching feel worth it

Different models have very different input and output rates, so the same caching pattern can be trivial on one model and meaningful on another. Here is a small comparison using only the current figures we have:

For live model pricing across the catalog, check the model price list. If you care about relative value instead of raw price, the value-for-money view is the quicker sanity check.

When caching helps a lot, and when it barely matters

There are three patterns where caching tends to be genuinely useful:

• Stable system prompts, where the same instructions go into every request
• Large shared context, such as docs, specs, or repository snapshots
• Agent loops, where the model revisits the same workspace many times in one session

There are also three common cases where the savings disappoint:

• Short prompts, because there is not much repeated input to reuse
• Highly variable prompts, because the prefix keeps changing
• Output-heavy tasks, where generation cost dominates anyway

That last point is important. A lot of people focus on prompt length and forget that if the model is producing long responses, code patches, or reasoning-heavy answers, output can be the bigger bill driver. Caching does nothing to shrink that part.

If your usage pattern is mostly “same instructions, different question,” caching is worth investigating. If it is “different context every time,” it is probably not the lever you want.

How to estimate whether caching is worth engineering time

Before you build anything custom, do a rough token audit. You do not need exact precision, just enough to see whether repeated context is large enough to justify the work.

1. Measure the average prompt size per request.
2. Estimate how many tokens are repeated across calls.
3. Count how many times that same prefix is reused per day or per session.
4. Multiply repeated tokens by request volume.
5. Apply the model’s input price per 1M tokens.

If the resulting number is tiny, caching is a nice optimization but not a priority. If it is meaningful, then caching may be one of the few changes that actually moves spend without changing product behavior.

Here is the shortest possible formula:

saved dollars = (reused_input_tokens / 1,000,000) × input_price_per_million

That formula is intentionally boring, because token math is boring. It is also the only thing that keeps you from guessing wrong.

How to track this in real usage

The fastest way to see whether prompt caching would help is to inspect your real logs, not your intuition. MyTokenTracker reads the usage logs coding agents already keep locally and prints a 30-day cost table at API prices, so you can see where repeated input is actually showing up.

Try npx mytokentracker to get a local view first. No account is needed, usage data stays on your machine, and you get a plain cost breakdown from the logs your agent already wrote. If you want a dashboard, create a free account and run npx mytokentracker init to upload history and keep syncing every 30 minutes.

That is especially useful if you are comparing agents or providers across the same project. MyTokenTracker captures cost, input/output/cache/reasoning tokens, latency, and success, broken down by provider, model, platform, and use-case. For supported wrappers, it can auto-capture OpenAI, Anthropic, Gemini, and Mistral in Python or Node, and other providers can send events through the API.

If you want the raw dataset behind the public views, the open data page is the place to start. It is easier to reason about caching when you can see how much of your spend is actually repeated context versus fresh work.

FAQ

Does prompt caching reduce output cost?

No. Caching helps with repeated input context. If the model still generates the same amount of output, that output is billed normally. In many real workloads, output is still a major part of the bill.

Is caching worth it for small projects?

Sometimes, but only if the same long prefix is reused many times. For small apps with short prompts, the savings are often too small to justify extra complexity. Measure first, optimize second.

What is the easiest way to tell if I have a caching opportunity?

Look for repeated system prompts, repeated docs, repeated repo context, or agent loops that resend the same instructions. If those show up in your logs over and over, caching is worth a closer look.

If you want to see your own repeated token patterns instead of guessing, run MyTokenTracker and check the 30-day table against your actual usage.