Oh My Pi

Trace Oh My Pi sessions, prompts, model calls, tools, token usage, timing, retries, and compaction.

@telemetry-dev/omp is an extension for Oh My Pi (omp). Install it as an omp package, set an API key, and each omp session appears in telemetry.dev as a trace with its prompts, model calls, tool calls, and token usage.

By default the extension sends your prompts, the model’s text responses, and every tool call’s arguments and results, which include file contents and command output. You can turn content off with two options; see What leaves your machine.

Before you start

• Oh My Pi 17 or later. Version 0.1.9 of the extension is tested with omp 17.1.8.
• A telemetry.dev project and its API key. Keys start with td_live_.

Set up

export TELEMETRY_DEV_API_KEY=td_live_...
export TELEMETRY_DEV_ENVIRONMENT=development

omp install @telemetry-dev/omp

Load the extension from a file instead

Use an extension file when you want to set options in code, such as turning off content capture or adding a mask. Don’t also install the package with omp install: omp would load the extension twice and record every span twice.

Create ~/.omp/agent/extensions/telemetry-dev.ts for all projects, or .omp/extensions/telemetry-dev.ts for one project:

import { telemetryDevExtension } from "@telemetry-dev/omp";

export default telemetryDevExtension({
  captureInput: false,
  captureOutput: false,
});

The file must be able to import @telemetry-dev/omp, so install the package in a node_modules directory that Node.js module resolution finds from the file’s location. For example, run npm install @telemetry-dev/omp in ~/.omp/agent for the global file, or in the project for the project file.

Check that data arrives

Send a prompt and wait for omp to finish it. Then open the trace explorer. The extension starts sending a prompt’s spans when the agent loop ends.

If no trace appears, check that TELEMETRY_DEV_API_KEY is set in the shell that started omp; without a key the extension does nothing. omp writes extension load errors to ~/.omp/logs/.

What is recorded

All prompts in one omp session share one trace. Each prompt is an invoke_agent span:

invoke_agent
├── chat {model}
│   ├── execute_tool {toolName}
│   └── execute_tool {toolName}
└── chat {model}

• invoke_agent covers one prompt, including retries, compaction, and queued continuations. It records the prompt and the final assistant text.
• chat {model} records the provider, model, response ID, finish reason, token usage, duration, time to first token, assistant text, and errors. Timing comes from omp’s own message timestamps.
• execute_tool {toolName} records the tool call ID, arguments, result, and error. It sits under the chat span that issued the call, or under invoke_agent when no message matches.
• Session start, switch, branch, compaction, retry, and shutdown events become logs.

Only text blocks count as assistant output. Thinking and tool-call blocks aren’t recorded as output, though token usage includes reasoning tokens.

omp doesn’t report cost for these messages, so the extension sends none. telemetry.dev calculates cost from the token usage and the model’s price.

What leaves your machine

By default the extension sends:

• Your prompts (captureInput).
• Tool arguments (captureInput), such as shell commands and file paths.
• The model’s text responses (captureOutput).
• Tool results (captureOutput), such as file contents and command output.

Set captureInput and captureOutput to false in an extension file to stop sending this content. mask can change or drop individual values before they are sent. The package installed with omp install reads only environment variables, so it always uses the defaults.

These are sent whatever the options:

• Session IDs, the working directory (on the session start log), provider and model names, token counts, and timings.
• The result text of a failed tool, as the span’s error message. In version 0.1.9 this ignores captureOutput and mask.
• Provider error messages, and the error text in retry and compaction logs.

telemetry.dev’s privacy settings can also drop recognized prompt and response content before storage, but only after it has left your machine.

How it runs

agent_start opens the invoke_agent span. An agent_end event with willContinue: true keeps it open, so retries, compaction, and queued continuations stay in one prompt span. The extension also checks that an agent_end belongs to the latest assistant message, so a stale event doesn’t close the span. A final agent_end closes the span and any unfinished tool spans.

If a new prompt starts before the previous one closed, the extension marks the previous span incomplete. session_shutdown closes an unfinished span and waits for the last export.

The extension uses omp’s tool_execution_start and tool_execution_end events, not tool_call and tool_result. In omp, an error in a tool_call handler blocks the tool, and telemetry should never do that.

Configuration

In an extension file, telemetryDevExtension() accepts these options, which override the environment variables:

The extension also accepts the other TypeScript SDK init() options, except registerGlobal, which it always sets to false. The first extension instance to start in a process sets the options for that process.

Limitations

• Subagents started by the task tool run in separate omp processes with their own extension instance, so they appear as separate sessions, not inside the parent trace.
• An assistant message with stopReason: "aborted" gets the finish reason aborted, not an error. Only stopReason: "error" marks spans as errors.