Evals API

Read dataset items, run experiments of your code, and send scores over HTTP, with request fields, limits, and error codes.

The Evals API lets your code read the items of a dataset, send the outputs of an experiment, and send scores over HTTP. The TypeScript and Python SDKs use this API in runExperiment and score. Read Experiments and Scores first.

The API is at https://api.telemetry.dev, the same host as the OTLP ingest API. https://ingest.telemetry.dev also works. The OpenAPI document describes each route.

Endpoints

Authentication

Send a project API key as a bearer token:

Authorization: Bearer td_live_your_key

The Run experiments access of a key gives the three scopes datasets:read, experiments:write, and scores:write. Read Select the access of a key. A key without the scope of a route gets 403 insufficient_scope.

The key also selects the project. Dataset names and experiment IDs are those of the project of the key.

Get a dataset

curl "https://api.telemetry.dev/v1/datasets/support-golden?limit=100" \
  -H "Authorization: Bearer $TELEMETRY_DEV_API_KEY"

{
  "dataset": { "id": "…", "name": "support-golden", "version": 3 },
  "items": [
    {
      "id": "…",
      "number": 1,
      "input": { "question": "Where is my refund?" },
      "expectedOutput": "Refunds take 5 to 7 days.",
      "metadata": {},
      "tags": ["refunds"],
      "split": "test"
    }
  ],
  "nextCursor": "…"
}

• Items come in number order. The id of an item is the same in each version. Send it as itemId with the run of the item.
• A page ends before limit when its items have 4 MiB of JSON. This cut always leaves at least one item in the page.
• The cursor holds the version. Thus each page reads the same version, also when the dataset changes. nextCursor is null on the last page.
• A dataset before its first change is at version 0, with no items.

Run an experiment

An experiment of your code has three steps:

1. Start the experiment with POST /v1/experiments.
2. Run your code on each item, and send the runs with POST /v1/experiments/{id}/items.
3. Send POST /v1/experiments/{id}/complete. Then read the result with GET /v1/experiments/{id}.

Start the experiment

POST /v1/experiments takes a JSON body of up to 1 MiB:

The response is 201 with the id, the name, the datasetVersion, the itemsTotal (the items multiplied by the runs for each item), and the url of the experiment page. The experiment is running immediately.

This request makes a new experiment each time. Send it again only after a 429.

Send the runs

POST /v1/experiments/{id}/items takes 1 to 100 runs in items, in a body of up to 16 MiB:

The response is 200 with accepted and duplicates. A run that the experiment has, by itemId and runIndex, keeps its first post. Thus you can send a request again after a timeout. The request adds all of its runs or none.

Complete and read the result

POST /v1/experiments/{id}/complete answers 200 with the id and the status. The status stays running until telemetry.dev scores the last runs. Then the experiment completes, and telemetry.dev does a check of the pass marks.

GET /v1/experiments/{id} returns these fields:

The delta is in whole points for a pass rate, and in one decimal for a mean. An experiment that gets no runs for 24 hours ends with the runs that it has.

Send a score

curl https://api.telemetry.dev/v1/scores \
  -H "Authorization: Bearer $TELEMETRY_DEV_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"user_feedback","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","value":0,"comment":"Wrong amount"}'

Give traceId, sessionId, or the two. Give value, label, or the two. The label pass goes with the value 1, and fail with 0. A pass or fail label without a value gets 1 or 0. user_feedback takes the value 1 for up or 0 for down, and no label.

The body can have up to 64 KiB. The response is 202 with the id of the score. telemetry.dev applies the redaction of the project to the comment. If the redaction would change the session ID, telemetry.dev drops the session ID. A score without a traceId then gets 400. Each request adds a new score, also for the same name and trace.

Errors

An error has a JSON body with a code and a message:

{ "error": { "code": "insufficient_scope", "message": "This API key cannot send scores. Create a key with Run experiments access." } }

The Evals routes have their own limit of 6,000 requests for each project in a minute. This limit is separate from the limit of the OTLP routes. Read Rate limits and quotas.