A founder opens the provider dashboard on the first of the month and the number is three times last month's. The team has traces. It has a spreadsheet. Nobody can say which feature, which agent, or which loop caused it.
This is the normal state of a small team shipping LLM features. Provider dashboards show spend by API key and model. Application logs show what ran. Nothing joins the two, so the question "what did that agent run cost?" gets answered with a guess.
AI cost observability is the fix, and it is less magical than vendors make it sound. You join usage that the model provider reports to calls you have instrumented, roll those calls up into agent runs, and keep three different numbers apart: tokens, estimated cost, and the invoice. When usage is missing or a price does not match, you mark cost as unknown. You never write zero.
This guide is an investigation you can run in an afternoon, in TypeScript or Python, with or without a framework.
What AI cost observability means for a small team (and how it differs from FinOps and AI observability)
AI cost observability is the practice of tying money spent on model calls to the code paths that caused it. The unit of analysis is the instrumented call and the agent run that contains it, not the monthly invoice line.
It is not FinOps. FinOps is an organizational discipline: budgets, allocation policies, forecasting, committed-spend negotiations. A team of eight does not need a FinOps function. It needs to answer "why did Tuesday cost four times Monday?" with evidence.
It is also not the same as general AI observability. Tracing tells you which step was slow or which tool call errored. Cost observability asks a narrower question of the same data: which calls consumed tokens, how many, in what categories, and what those tokens plausibly cost. The two share a data source. A trace of model calls, tool steps and retrievals is the raw material for both, and that is the kind of trace telemetry.dev records alongside cost, latency, tokens and errors.
Here is the stance this article takes: cost observability is an investigation technique, not a dashboard. A dashboard shows you a number. An investigation tells you whether the number can be trusted.
That matters because LLM cost is derived, not measured. Your application never sees a dollar amount from the provider at call time. It sees token counts (if the provider returns them), looks up a price for that model, and multiplies. Every link in that chain can break quietly. Usage can be absent, the model name can fail to match a price, and the token categories can be double counted. The rest of this guide walks those failure points in the order you should check them.
Three numbers, not one: reported tokens, estimated cost and the provider invoice
Stop thinking of "cost" as one figure. You have three, and they answer different questions.
Reported tokens are the counts the provider returns in each response: input, output, and often cached or reasoning categories. This is the most trustworthy number you hold, because it comes straight from the party that bills you. It is a measurement.
Estimated cost is reported tokens multiplied by a price you looked up. It is a calculation. It is only as good as the usage it consumes and the price catalog it matches against. A span that carries an explicit cost value is a different thing from a computed one, and you should know which you are looking at.
The provider invoice is what you actually pay. It includes things your traces cannot see: calls made outside instrumented code, other teammates' experiments on the same API key, batch discounts, committed-use terms, rounding and billing-period boundaries.
Keep these in three separate columns. Do not overwrite one with another. When the estimate and the invoice disagree, the gap is information. A large gap usually means uninstrumented callers or a pricing mismatch, and you can only see that if you never collapsed the numbers together.
A practical habit: when you investigate a spike, write down all three for the same window. Tokens by model from traces. Estimated dollars from traces. Invoice dollars for the matching dates. If tokens in your traces are flat but the invoice jumped, the problem is not in the code you instrumented. That one comparison saves hours of staring at the wrong spans.
Step 1: Make sure every model call reports usage (missing usage is unknown, not zero)
Start by counting calls, not dollars. For the window you care about, how many model calls were instrumented, and how many of those carry token usage?
A call with no usage attached has unknown cost. It does not have zero cost. This is the most common way cost dashboards lie. A streaming response ends early, the final usage chunk never arrives, and the span records nothing. Summed naively, that call contributes $0.00. The real call still appeared on your invoice.
Streaming is the usual culprit. Some providers only send usage at the end of a stream, and some require you to opt in. If the connection drops or the consumer stops reading, the usage never shows up. The Bedrock streaming token usage guide covers how to handle incomplete streams so you can tell "zero tokens" from "never reported."
Run this check:
List model-call spans in the window.
Split them into "has input and output token counts" and "has neither."
For the second group, group by provider, model and whether the call was streamed.
If one model or one code path owns most of the missing group, you have found an instrumentation gap, not a spending pattern. Fix that before trusting any total.
Also check what the instrumentation does not cover. First-party SDKs and OTLP/HTTP exporters only record what you wrap or export. A background script, a notebook, or a second service calling the same provider key will not appear. Per the OpenAI integration docs, provider instrumentation is what produces the call-level detail, so a call made outside it is invisible to your traces however carefully you read them.
Write the count down: "412 calls, 371 with usage, 41 unknown." That sentence belongs next to every cost figure you share.
Step 2: Read cache and reasoning token categories without double counting
Token counts are not one pile. Providers split them into categories, and the categories overlap in ways that quietly inflate or deflate your estimate.
Three categories cause most of the trouble:
Cached input tokens. Prompt caching means part of your input was served from cache at a different rate. Some providers report cached tokens as a subset of the total input count. Others report them separately from it. If you add cached tokens on top of an input total that already includes them, you double count.
Cache write tokens. Some providers charge a premium to write a prefix into the cache. That is a separate line, not a free byproduct.
Reasoning tokens. Reasoning models spend tokens thinking before they answer. Depending on the provider, those may be included inside the output count or reported as a detail of it. They are billed as output either way, which is why a short visible answer can carry a large bill.
The rule: read the provider's documented definition for each field, and decide per provider whether a category is a subset of another or additive. Do not assume one provider's convention applies to the next. If you route through a gateway or aggregator like LiteLLM or OpenRouter, the field names may be normalized, and the normalization may or may not match what the underlying provider bills.
Test it on one call. Take a single response with cached tokens, pull the raw usage object, and compute the cost by hand using the provider's published rates. Then compare to what your pipeline estimated for that span. If they differ, you have a categorization bug, and it will scale with traffic.
When you cannot tell whether a category is included or additive, say so. An estimate with a stated uncertainty is more useful than a confident wrong one, and it keeps you from treating a guess as a measurement.
Step 3: Find retries, loops and repeated steps inside an agent run
Once usage is trustworthy, group calls into agent runs and look at shape, not totals. The expensive run is rarely one enormous call. It is usually forty ordinary calls that should have been eight.
Open a costly run as a trace and look for three patterns.
Retries. The same step appears multiple times in a row, often after an error or a validation failure. Each retry re-sends the full context, so the input tokens repeat. A retry loop on a long prompt multiplies cost fast, and each attempt looks innocent by itself.
Loops. The agent calls a tool, reads the result, calls the same tool with nearly identical arguments, and repeats. You will see this as a staircase of near-identical spans. The fix is usually a stop condition or a better tool result, not a cheaper model.
Growing context. Each turn appends the previous output, so input tokens climb from call to call. Plot input tokens per call in order. A steady upward slope means the agent carries its whole history every turn, and your cost grows faster than your step count.
Compare a cheap run and an expensive run of the same agent side by side. Count model calls, count tool calls, and note where they diverge. The divergence point is the story.
If your calls pass through a gateway, make sure the application span and the gateway request can be matched to each other. Otherwise you will count one call twice or miss a retry made by the gateway itself. The LiteLLM tracing guide shows how to correlate app and gateway requests.
Traces of model calls, tool steps and retrievals with tokens and errors are exactly the evidence this step needs. You are reading structure, and structure is what an aggregated invoice hides.
Step 4: Check the price-catalog match, and attribute spend with generic app instrumentation
Now the dollars. Estimated cost needs a price, and a price needs a match between the model name on the span and an entry in a price catalog. That match fails more often than people expect.
Common causes: a dated model snapshot name that the catalog does not list, a provider-prefixed name from a gateway, a fine-tuned model with its own rate, or a new model released after the catalog was last updated. When there is no match, the honest result is "cost unknown," the same as missing usage. Do not let an unmatched price fall through to zero or to a default rate that looks plausible.
Check it directly. Group spans by model name and list which ones got an estimate and which did not. Any model with tokens but no estimate has a catalog gap. Those tokens are real spend you have not priced.
Next, attribution. Cost by model is easy. Cost by feature, customer or team is a question about your application, and you answer it with your own instrumentation. telemetry.dev records the model, tool and retrieval traces. It is not described as doing per-user or per-team attribution for you, so add the dimension yourself.
The generic pattern works in any OpenTelemetry-style setup. When your app starts a request or agent run, attach attributes you choose, such as a feature name, a customer ID you are comfortable storing, or an environment, to the parent span or to a field on each call. Then group by that attribute when you investigate. This is ordinary application instrumentation using the TypeScript or Python SDK or an OTLP/HTTP exporter, not a built-in function.
Be careful with identifiers. Prompt capture can be turned off per environment before storage in telemetry.dev, and redaction patterns cannot catch all sensitive data, so decide deliberately what you attach.
Where telemetry.dev fits, and where other tools fit better
telemetry.dev is LLM observability built on OpenTelemetry. It traces model calls, tool steps and retrievals and tracks cost, latency, tokens and errors, with first-party TypeScript and Python SDKs and support for any OTLP/HTTP exporter or OpenTelemetry Collector. Provider instrumentation covers OpenAI, Anthropic, Google Gemini, AWS Bedrock and OpenRouter, plus LiteLLM in Python, and it has framework integrations for the Vercel AI SDK, TanStack AI and Vercel Eve. For the investigation above, that gives you the call-level trace and the token counts that Steps 1 through 3 depend on.
It is a good fit when you are a small team with an unexplained bill and want to see which step in which run consumed tokens, without hosting anything yourself. The Free plan is $0 with 30k units and 14-day retention, Pro is $25 per month with 500k units, and Team is $50 per month with 2M units and unlimited projects and seats.
Know its limits. Computed cost depends on reported token usage and a pricing match, so it will not be exact for every model. It does not reconcile against your provider invoice for you.
If you need to reconcile invoices across vendors, allocate spend to departments, or enforce budgets, you want a billing or FinOps tool alongside traces. If you need evaluations or prompt management, look elsewhere for those. If you want to see why a specific run cost what it did, tracing is the right place to start.
Conclusion
Run the investigation once, on last month's worst day. Count calls with and without usage. Split tokens into their categories. Open the most expensive agent run and look for retries and loops. List the models that never got a price. Write down tokens, estimated cost and the invoice as three separate figures, and treat every gap as unknown rather than zero.
If you do not have call-level traces yet, that is the first thing to fix, and it is the cheapest. Instrument one service with the telemetry.dev TypeScript or Python SDK on the free plan, and open one expensive agent run. If you use the Vercel AI SDK, the tracing walkthrough gets you there in an afternoon. You will know more about your bill by the end of it than the provider dashboard has told you all quarter.
Share this article
