An agent run finishes in 41 seconds, returns a correct answer, and the dashboard shows a red error. Another run finishes in 4 seconds, returns garbage, and the dashboard is green. If that sounds familiar, your telemetry is describing calls, not runs.
The usual cause is retries. A tool times out, the agent retries it, the second attempt works, and the run succeeds. But the first attempt left an error behind, and nothing in your data says whether that error mattered. Your on-call engineer sees a failure and can't tell if a user was affected.
This post is about agent telemetry for production AI agents: the traces, logs and metrics your own application emits about model calls, tool calls and retries. It is not about host monitoring agents, the ones that sit on a server and report CPU. We'll build a generic OpenTelemetry span hierarchy where every attempt is its own span, show how to record errors correctly, write a small retry wrapper, and be plain about what you won't see. All code here is application-owned instrumentation that you write and maintain.
What agent telemetry means for a production AI agent (and what it does not)
Agent telemetry is the data a running AI agent emits about its own work: spans for each model call and tool call, logs for notable events, and metrics for counts and durations. It exists to answer one question after the fact: why was this run slow, retried, or failed?
How does it work? Your application creates spans around units of work, attaches attributes such as the tool name or attempt number, and exports them over OpenTelemetry to a backend. The backend stitches spans into a trace through parent and child relationships. Nothing is observed unless your code or an instrumentation library creates the span.
That last point separates this from traditional monitoring. Host monitoring tells you a process is alive and how much memory it uses. It can't tell you that step 3 of a run called a search tool twice, or that a model call returned a rate limit error and the agent quietly retried it. Traditional monitoring watches the machine. Agent telemetry watches the work.
The data you'll collect is made of ordinary OpenTelemetry signals: traces (spans with timing and status), logs (discrete events), and metrics (counters and histograms). For agents, the useful attributes are model name, token counts, tool name, attempt number and error type.
Here's my stance: agent telemetry is not a special product category with its own data model. It's OpenTelemetry applied to a workload that happens to loop, branch and retry. The only new decision is where the span boundaries go, and that's what the next sections cover.
A failed attempt is not a failed run: the distinction to build around
A failed attempt is one try at one operation. A failed run is the end state the user experiences. They are different facts, and most agent dashboards collapse them into one.
Take a concrete case. An agent calls a fetch_invoice tool. Attempt one hits a 503 from your billing API. Attempt two, 800 milliseconds later, returns the invoice. The agent finishes and answers the user correctly. Attempts: two. Failed attempts: one. Failed runs: zero.
If you only record the final outcome, you lose the 503 and never learn your billing API is flaky. If you record every error as a run failure, your error rate is inflated and you page people for problems that healed themselves. Neither works.
The fix is to give each attempt its own span with its own status, and let the run span carry the final outcome. Then the questions separate cleanly:
How often do attempts fail? Count error spans at the attempt level.
How often do runs fail? Count error status on the run span.
How much latency do retries add? Sum attempt durations beyond the first, per run.
This is also where agent telemetry beats plain logs. A log line saying "retrying" has no duration and no parent. A span has both. When someone asks why a run took 41 seconds, you open the trace and see three attempts at the same tool, two of them waiting on a 15 second timeout. You find that in ten seconds instead of an afternoon of grepping.
A span hierarchy for retries and tool failures: run, step, model call, tool attempt
Use four levels. Each one maps to a question you'll actually ask.
Run. The root span. One per user request or agent invocation. It carries the final status, total duration, and identifiers like a run ID. This is the span you alert on.
Step. One iteration of the agent loop: decide, act, observe. A run with five reasoning loops has five step spans. Steps show you where in the loop time went.
Model call. A child of a step. One span per request to a model provider. Attributes include model name and token usage. If the provider call is retried, each try is a separate model call span.
Tool attempt. A child of a step. One span per try at one tool invocation. If the agent calls fetch_invoice and it takes three tries, that's three sibling tool attempt spans under the same step, with an attempt number attribute on each.
The shape looks like this:
agent.run
agent.step (1)
model.call
tool.attempt fetch_invoice (attempt 1, error)
tool.attempt fetch_invoice (attempt 2, ok)
agent.step (2)
model.callTwo design choices matter here. First, keep attempts as siblings, not nested. Nesting attempt 2 inside attempt 1 makes a retry look like a deeper call and distorts duration. Second, put the logical operation identity in an attribute, such as tool.name, so you can group attempts for the same call without a special span type.
The names and attribute keys above are an application convention you define, not a fixed standard. Pick them once and apply them everywhere. Consistency across services matters more than the exact spelling.
Recording errors correctly: span status, error.type, exceptions and retry attributes
Get error recording right and the rest of the hierarchy pays off. Get it wrong and every attempt looks the same.
The OpenTelemetry semantic conventions for recording errors are specific. On a failed operation, set the span status to Error and set the error.type attribute to a low-cardinality value that classifies the failure, such as a timeout, an HTTP status code or an exception class name. Keep error.type short and bounded so you can group by it. Put the long, specific message in the status description or an exception event, not in error.type.
If an exception was involved, record it as an exception event on the span, which carries the type, message and stack trace. That gives you the detail without bloating the attributes you group on.
Now the retry-specific part. The conventions describe a failed operation. They don't tell you how to mark "this attempt failed but the run recovered." That call is yours, and the cleanest answer is:
The attempt span gets status Error and an
error.type, because that attempt did fail.The run span gets status Error only if the run ultimately failed.
Add your own attributes for retry context, for example
retry.attempt(1, 2, 3),retry.max_attempts, and a boolean likeretry.will_retryon a failed attempt.
The retry.* keys are application-defined, not part of the official conventions, so treat them as yours. Document them in the repo.
Hold one rule firmly: never set the run to Error just because a child attempt errored. Status doesn't propagate upward automatically in OpenTelemetry, and it shouldn't here. The run's status reflects the outcome the user got.
Application-owned instrumentation: a retry wrapper example in TypeScript
This is application-owned instrumentation. You write it, you own it, and nothing here happens automatically. It uses the standard OpenTelemetry JavaScript API.
import { trace, SpanStatusCode, context } from "@opentelemetry/api";
const tracer = trace.getTracer("my-agent");
export async function withToolRetry<T>(
toolName: string,
maxAttempts: number,
fn: () => Promise<T>
): Promise<T> {
let lastError: unknown;
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
const span = tracer.startSpan(`tool.attempt ${toolName}`, {
attributes: {
"tool.name": toolName,
"retry.attempt": attempt,
"retry.max_attempts": maxAttempts,
},
});
try {
const result = await context.with(
trace.setSpan(context.active(), span),
fn
);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (err) {
lastError = err;
const willRetry = attempt < maxAttempts;
span.recordException(err as Error);
span.setAttribute("error.type", (err as Error).name ?? "Error");
span.setAttribute("retry.will_retry", willRetry);
span.setStatus({
code: SpanStatusCode.ERROR,
message: (err as Error).message,
});
} finally {
span.end();
}
}
throw lastError;
}Three things to notice. Each attempt gets its own span, started inside the loop and ended in finally so it always closes. The failing attempt records the exception and error.type, and sets retry.will_retry so you can tell a recoverable failure from the last one. And the wrapper makes no decision about the run. Whoever owns the run span sets its status after the loop resolves.
Backoff delay between attempts is not inside any attempt span here, on purpose. If you want to see time spent waiting, add a separate short span around the sleep. Otherwise that gap shows as empty space in the trace, which is informative in its own right.
The same structure ports to Python with tracer.start_as_current_span in a loop. The principle is identical: one span per attempt.
Instrumentation gaps: what you will not see and why
Spans only exist where something creates them. That sounds obvious until you're staring at a trace with a suspicious hole in it.
Here is what typically goes missing:
Retries inside a library. If an SDK retries a request internally, your code sees one call. Unless that library emits its own spans, you get one model call span that is simply long. You can't tell one slow call from three fast ones plus backoff.
Uninstrumented tools. A tool function you didn't wrap produces no span. The time still shows up as a gap inside the step.
Work outside the process. A queue, a webhook or a background job breaks the trace unless you propagate context across the boundary. Without it you get two disconnected traces for one run.
Token usage that was never reported. Cost and token figures depend on the provider returning usage and your instrumentation reading it. A streamed response that gets cut off may report nothing. Our post on Bedrock streaming token usage covers that failure in detail.
Prompt content you chose not to capture. If capture is off, you see timing and status but not the payload. See prompt capture and inspection before export.
A trace is a record of what your instrumentation observed, not a record of what happened. Treat every unexplained gap as a question about coverage before you treat it as a question about performance.
When you add a framework integration, don't assume it emits an attempt-level span for every retry. Open one real trace, find a call you know was retried, and confirm what appears. If it's one span, wrap it yourself.
Sending the spans to telemetry.dev via the quickstart and OpenTelemetry setup, and where other tools fit better
Everything above is standard OpenTelemetry, so it exports over OTLP/HTTP to any compatible backend. telemetry.dev accepts those spans and shows model calls, tool steps, cost, latency, tokens and errors in one trace view, so you can find the slow step and inspect the failed call.
To get data flowing, follow the quickstart for the first-party TypeScript or Python SDK, or the OpenTelemetry setup guide if you already run an OTLP exporter or an OpenTelemetry Collector. If you use the Collector, our note on environment variables for auth saves some fiddling.
Keep the division of labor straight. The attempt spans, error.type values and retry.* attributes in this post come from your instrumentation. telemetry.dev displays what you send. It doesn't infer retries that your code never marked, so the hierarchy is only as good as the wrapper you write.
The free plan gives you 30k ingestion units and 14-day retention with no credit card. That matters here: a run with 5 steps, 5 model calls and 8 tool attempts is about 19 spans, so retry-heavy agents burn through units faster. Budget for it. For spend specifics, see tracing AI cost to agent runs.
Where do other tools fit better? If you need evaluations, prompt management or self-hosting, telemetry.dev doesn't offer those, and a different tool will serve you better. If you need a general APM replacement for hosts and databases, that's also a different job.
Conclusion
Pick one agent in production this week and wrap its most failure-prone tool with a per-attempt span, using the retry wrapper above. Set error.type on each failed attempt, leave the run span alone until the run actually resolves, and look at one real trace to confirm the retries appear as siblings.
Then send it to telemetry.dev. Start with the quickstart on the free plan, no card needed, and open a run where a tool was retried. If you can read off which attempt failed, why, and how much time it cost, your agent telemetry is doing its job. If you can't, the gap is in your span boundaries, and now you know where to add one.
Share this article
