Trace every eval run like a production run
Evaluation calls from the AI SDK and TanStack AI arrive as Evaluate spans. See what each run cost and where it spent its time. AI SDK evals also show the questions and the answers.
Your coding agent adds the telemetry.dev integration to your evals and sends them to their own environment. Scores, datasets, and evaluators are not part of telemetry.dev yet.
- evaluate refund-policy41.8s
- support-agent · case 18.9s
- support-agent · case 27.4s
- support-agent · case 310.7s
- chat gpt-4.12.8s
- lookup_ordererror3.6s
- search_policies0.9s
17 more spans
- Duration
- 41.8s
- Tokens
- 58,204
- Cost
- $0.31
- Input · questions
- Does the agent refuse refunds above the limit?
- Does it ask for the order number first?
- Output · answers
- Yes, in 4 of 4 cases.
- No, in case 3 it asked after the lookup.
Look inside the eval, not only at its result
An eval run is a trace: the evaluate span, the model calls under it, and their tokens, cost, and errors.
Know what each model costs to evaluate
Run the same eval on two models, then compare their spend, latency, and errors in the evals environment.
- gpt-4.11411.6s$4.34
- claude-sonnet-4-5149.8s$3.72
- gpt-4.1-mini144.2s$0.84
- gemini-2.5-flash143.1s$0.41
See why a case went wrong
A failed tool call inside an eval shows its error, like it does in production.
- Error
- TimeoutError
- The operation was aborted due to timeout
- Parent
- chat gpt-4.1 · case 2
- evaluate refund-policy
Keep eval traffic out of production numbers
Send evals to their own environment. Their spans count as ingestion units.
- production41,206 spans
- evals1,344 spans
- staging3,980 spans
- development612 spans
Add the integration to the eval
Pass the telemetry.dev integration to experimental_evaluate, as you do for generateText.
- 01Install the AI SDK integration
- 02Add it to your eval
- 03Run evals in their own environment
Works with the AI SDK and TanStack AI.
import { experimental_evaluate } from "ai";
import { telemetryDev } from "@telemetry-dev/ai-sdk";
await experimental_evaluate({
model,
state: ticket,
questions,
telemetry: {
functionId: "refund-policy-evals",
integrations: [telemetryDev()],
},
});import { decide } from "@tanstack/ai";
import { telemetryDev } from "@telemetry-dev/tanstack-ai";
const result = await decide({
adapter,
state: ticket,
questions,
middleware: [telemetryDev()],
});Trace your next AI request
Choose an integration, add your project API key, and run an instrumented request.
Start free
30,000 ingestion units a month, one per span, log record, or metric data point. No credit card required.
npm install @telemetry-dev/sdkTalk to an engineer
Our team shows you tracing, cost tracking, and privacy controls for the frameworks and model providers that you use.