LLM Cost Monitoring: Account for Cached Tokens

How to handle cached tokens in LLM cost monitoring: OpenAI and Gemini count cache hits inside input, Anthropic reports them separately. Normalize before pricing.

Ephraim Duncan

To account for cached tokens in LLM cost monitoring, normalize every provider response into three buckets before you price it: uncached input, cache reads and cache writes. OpenAI and Gemini count cache hits inside the input total, while Anthropic reports cache reads and writes separately from input_tokens, so one summing rule applied to all three providers will double count on some and undercount on others.

That is the usual reason a token dashboard disagrees with the provider invoice once prompt caching is on. Nothing is broken. Each provider reports usage under its own convention, and the numbers only line up after you translate them into one shape.

This post shows the three conventions, the normalization step, a synthetic example priced under two conventions, and what to record on each span. It makes no savings promise and offers no universal formula. Prices and fields change by model and by provider.

Why cached tokens break naive LLM cost monitoring

Cached tokens break naive LLM cost monitoring because a single call now carries tokens billed at different rates, and the providers disagree on whether the cheap ones are part of the input count or beside it.

A naive dashboard multiplies input tokens by one input price and output tokens by one output price. With caching on, that is wrong in both directions at once. If your provider counts cache hits inside input, you overcharge every hit at the full rate. If your provider reports cache activity separately, you drop it from the sum and undercount, including the cache writes that are priced above the base input rate.

The symptom is a gap against the invoice that moves with your cache behavior. Traffic with long, repeated prefixes drifts one way, cold traffic drifts the other. That moving gap means you are mixing conventions. The provider is not billing you wrongly.

This is a different failure from counting the same call twice across an app span and a gateway span. If you suspect that too, read LLM Cost Monitoring: Avoid Double Counting Gateway Spans. The two problems stack, so fix both.

How OpenAI, Anthropic and Gemini report cached tokens (inclusive vs separate counts)

OpenAI and Gemini report cache hits inside the input total, and Anthropic reports cache reads and writes in their own fields outside input_tokens.

OpenAI. The usage object has input_tokens, and under input_tokens_details a cached_tokens field. For current OpenAI Responses API usage, cached_tokens is reported within input_tokens_details and input_tokens includes cached and cache-write tokens; cached_tokens is a component of input_tokens, not additional tokens. Cache-write tokens are also reported separately. The docs' worked pricing example splits input into pieces and prices each at its own rate. Newer models also report cache_write_tokens in the same details object. Treat that as its own bucket priced at that model's published rates, and do not count those tokens again as ordinary input.

Anthropic. Usage reports input_tokens, cache_creation_input_tokens and cache_read_input_tokens as separate fields. input_tokens covers only the input after the last cache breakpoint, not the whole prompt, so total prompt size is the sum of the three. The pricing table on that page lists base input, 5-minute cache write, 1-hour cache write, cache read and output rates per model. Read the row for your model. Don't assume a multiplier.

Gemini. Gemini reports prompt token counts alongside cached content token counts. Do not add them. Uncached input is the difference. Explicit context caching also bills storage over the cache's TTL, a cost that never appears in per-call token fields, so check the current Gemini pricing page for your model before you model it.

Provider Cache hits vs input total Cache write field Uncached input is
OpenAI Inside the input total cache_write_tokens on newer models input minus cached
Anthropic Separate from input_tokens cache_creation_input_tokens input_tokens as reported
Gemini Inside prompt_token_count Storage billed over TTL for explicit caches prompt count minus cached

Normalize every call into uncached, cache-read and cache-write buckets

Normalize each call by mapping provider fields into three buckets, then price each bucket at its own rate, so no token is counted in two places.

The three buckets:

  • Uncached input: tokens billed at the standard input rate.

  • Cache read: tokens served from an existing cache, billed at the cache-read rate.

  • Cache write: tokens stored into the cache on this call, billed at the write rate for that model and TTL.

The mapping is a per-provider adapter, written once:

  • Inclusive providers (OpenAI, Gemini): cache read is the cached count. Uncached is the input total minus the cached count, minus any reported write tokens. Cache write is the reported write count, if the model reports one.

  • Separate provider (Anthropic): uncached is input_tokens, cache read is cache_read_input_tokens, cache write is cache_creation_input_tokens. No subtraction.

Then price: each bucket times its rate from that provider's current pricing page for that model, plus output tokens at the output rate. Keep the rates in a table keyed by model. Hard-coding them in the adapter is a mistake, because they change.

Two rules keep this honest. First, the three buckets should add up to the provider's own notion of total input. If they don't, your adapter is wrong for that model. Second, if a call reports no usage at all, record it as unknown, not as zero. A zero silently understates the bill.

A worked synthetic example: the same prompt under two reporting conventions

The same 10,000-token prompt produces different raw usage numbers under an inclusive and a separate convention, and normalization turns both into the same three buckets.

This example is invented for illustration. Rates are left symbolic on purpose: let U be the uncached input rate, R the cache-read rate and W the cache-write rate per token for the model in question, taken from its pricing page.

Suppose a call sends 10,000 prompt tokens, where 8,000 are a stable prefix already in the cache, and 2,000 are new text after it. No new cache entry is written.

Reported inclusively (OpenAI or Gemini style):

  • input total: 10,000

  • cached: 8,000

Normalized: uncached 2,000, cache read 8,000, cache write 0.

Reported separately (Anthropic style):

  • input_tokens: 2,000

  • cache_read_input_tokens: 8,000

  • cache_creation_input_tokens: 0

Normalized: uncached 2,000, cache read 8,000, cache write 0.

Same buckets, same cost: 2,000 U + 8,000 R, plus output.

Now the errors. Applying the Anthropic rule to the inclusive report would add 10,000 and 8,000 and bill 18,000 tokens of input, a double count. Applying the inclusive rule to the Anthropic report would read 2,000 as the whole prompt and miss the 8,000 cache reads, an undercount. Neither mistake throws an error. Both just make totals wrong.

A cold first call looks different. If the prefix is written instead of read, an inclusive provider that reports write tokens shows them inside the input details, and Anthropic shows them in cache_creation_input_tokens. In both cases the normalized write bucket is 8,000 and the read bucket is 0, priced at W.

What to record on each span, and what OpenTelemetry's GenAI conventions say

Record the raw provider usage fields and the three normalized buckets on the span for each billable provider call, so you can re-price later without re-running anything.

A workable set of attributes per call:

  • model name as the provider returned it

  • the provider's raw usage fields, unmodified

  • normalized uncached input, cache-read and cache-write token counts

  • output tokens

  • a flag for usage missing or incomplete

  • an attribution key such as feature, user or agent run

Keep raw and normalized side by side. When a provider changes a field or you fix an adapter bug, you can recompute from the raw values instead of losing history.

OpenTelemetry's GenAI semantic conventions define attributes for tracking input tokens and caching. As of the current spec, input_tokens should include cached tokens, both read and written, and the cache attributes are breakdowns of that total, not additions to it. That matches the inclusive convention, so Anthropic data has to be summed into the total before export: uncached plus reads plus writes. Conventions in this area are still evolving, so check the spec version your instrumentation targets. Don't assume attribute names.

An instrumentation library may not emit every cache attribute for every provider. Verify what actually leaves your app before you trust a dashboard. The method is in OpenTelemetry LLM Observability: Verify Exported Spans: send one known call, read the Collector's debug output, and confirm the attributes you expect are present.

Cache hit rate and cost: what you can and cannot conclude

A cache hit rate tells you how much of your input was served from cache. It does not tell you what you saved or what you will spend.

Compute it from normalized buckets: cache-read tokens divided by total input tokens, where total input is uncached plus reads plus writes. Using the buckets keeps the metric consistent across providers. Dividing raw cached counts by raw input_tokens gives a meaningless number on Anthropic, where input_tokens excludes the cached portion.

The rate can show whether a prompt change broke prefix stability, whether one feature gets almost no hits, or whether hit rate dropped after a deploy. On its own it cannot show:

  • Savings. Writes cost more than base input on some models, and a low-reuse prefix can cost more with caching than without. Whether you came out ahead depends on reuse and on each model's rates.

  • Storage. Explicit Gemini caches bill for storage over their TTL, which a per-call token view misses.

  • Expiry effects. Cache entries expire, so the same traffic can show different rates at different times of day.

Hit rate is a diagnostic. Priced buckets are the cost. Look at both, per feature and per model, before drawing a conclusion.

Check cached-token cost against the provider invoice, and see it in telemetry.dev

Validate your normalization by pricing a known window of calls and comparing the total to the provider invoice for the same window and model.

Pick one model and one day. Sum your priced buckets, add output cost, and compare. A gap that tracks cache activity points at an adapter or rate error. A gap that tracks retries or gateways points back at double counting. Count calls with missing usage separately and show that count next to the total. They contribute nothing to your sum and still appear on the invoice.

telemetry.dev is LLM observability built on OpenTelemetry that traces model calls, tool steps and retrievals and tracks cost, latency, tokens and errors. It ingests standard OpenTelemetry spans, allowing the span attributes described above to be sent directly from your instrumentation. That makes the invoice check above the right acceptance test, and it isn't optional.

The Free plan has 30k ingestion units and 14-day retention. How much traffic fits depends on the spans, logs and metrics your instrumentation emits.

Price buckets, not provider totals, then check one day against the invoice

Cached tokens are a reporting problem before they are a pricing problem. Once every call is mapped into uncached, cache-read and cache-write buckets, the three providers stop disagreeing and your cost numbers have a fixed shape. Do it this week: write the adapter for the one provider you spend the most on, record raw and normalized usage on each span, and compare one day of priced calls to the invoice. You can send that data to telemetry.dev on the free plan at telemetry.dev and run the comparison there.

Share this article

Related posts