Get an alert when cost or errors break from normal
Alerts check your project every 5 minutes against its own history. Each alert says what changed, what it was compared with, and links to the spans that failed.
Your coding agent checks that your spans carry what the rules read. Alerts go out by email today. Slack, Discord, and webhooks are coming soon.
What happened
- What changed
- create_refund failed 42 of 233 calls in production since 09:04.
- Compared with
- 2.1% of calls failed over the last 30 days.
- Likely cause
- Most failures share one error type,
RefundLimitExceeded.
Rule
- Threshold
- production on track to spend more than $1,000 this month.
- Last alert
- Oct 24. production was on track for $1,180 of $1,000. It resolved when the projection fell under $1,000.
- This month
- $646 spent so far. At this pace, the month ends at $742.
Rule
- Threshold
- One agent turn that costs more than $2 or runs more than 30 spans.
- Last alert
- Oct 22. One support-agent turn cost $2.84 over 41 spans. It resolved after 30 minutes.
- Last 7 days
- 1,912 agent turns. The most expensive turn cost $1.12.
Rule
- Threshold
- Cost per successful trace in production reaches 1.5 times the usual and adds $5 or more a day, over 100 traces or more.
- Last alert
- None in the last 30 days.
- Last 7 days
- The average trace cost $0.075. The usual trace costs $0.072.
Rule
- Threshold
- One model that fails much more often than in the last 30 days, in 50 calls or more.
- Last alert
- Oct 18. gpt-4.1-mini failed 9% of calls in production. It resolved after 2 hours.
- Last 7 days
- gpt-4.1-mini failed 0.6% of calls, and claude-sonnet-4-5 failed 0.3%.
Rule
- Threshold
- Fewer than 90% of model calls with a price, in 100 calls or more.
- Needs
- 100 model calls in the last hour or the last 6 hours. Until then, it waits.
- Last 7 days
- No window had enough model calls to check.
- Tool errorscreate_refund fails 18% of callsAlerting · 38m
- Monthly budgetproduction was on track for $1,180 of $1,000Resolved · Oct 24
- Agent loopOne support-agent turn cost $2.84Resolved · Oct 22
- Model errorsgpt-4.1-mini fails 9% of callsResolved · Oct 18
- Tool errorsget_invoice times out on 6% of callsResolved · Oct 11
Each alert explains itself
Rules learn the usual numbers of your project. Each alert says what changed, what it was compared with, why it matters, and where to look.
Go from the email to the failed spans
The link opens Traces with the tool, model, or provider already in the filter. Reminders go out every 6 hours until the numbers settle.
Tool errors in support-agent: create_refund fails 18% of calls
- Failed calls
- 18.0%
- Usually
- 2.1%
- Calls
- 233
Why it matters
Customers who ask for a refund get an error from the agent, and the agent retries the tool.
Catch an agent that loops
Know when one agent turn costs more than $2 or runs more than 30 spans.
- support-agent38.2s
- chat claude-sonnet-4-5×14
- lookup_order×13
- check_inventory×12
- 1 more span
Stay inside the budget
Set a monthly budget in USD for an environment. Hear about it while there is still time.
- Spent
- $122
- Projected
- $236
- Budget
- $200
Send it where your team already looks
Email works today. Slack, Discord, and webhooks are coming soon.
- Emailephraim@telemetry.devLiveSlackA message in a channelSoon
- DiscordA message in a channelSoonWebhookA JSON POST to your URLSoon
- Microsoft TeamsA message in a channelPlannedPagerDutyAn incident for the on-call personPlanned
- OpsgenieAn alert for the on-call personPlannedGoogle ChatA message in a spacePlanned
Turn on the rules you need
Each project starts with recommended thresholds. Change one, and the preview tells you if the rule would alert now.
- 01Open Alerts in your project
- 02Turn on a rule
- 03Add who gets the alert
All clear nowat 5% failed calls for 50+ calls
Rules run every 5 minutes. An alert resolves after 3 healthy checks in a row.
| Rule | Fires when |
|---|---|
| Monthly budget | An environment is on track to spend more than its budget |
| Cost per trace jump | A trace costs much more than the usual trace |
| Agent loop | One agent turn costs more than $2 or runs more than 30 spans |
| Model errors | One model fails much more often than in the last 30 days |
| Tool errors | One tool fails much more often than in the last 30 days |
| Ingestion errors | The same ingest error occurs 3 times in 5 minutes |
| Missing prices | Fewer than 90% of model calls have a price |
| Quota running out | Your workspace nears its monthly ingestion units |
Trace your next AI request
Choose an integration, add your project API key, and run an instrumented request.
Start free
30,000 ingestion units a month, one per span, log record, or metric data point. No credit card required.
npm install @telemetry-dev/sdkTalk to an engineer
Our team shows you tracing, cost tracking, and privacy controls for the frameworks and model providers that you use.