Most teams find out what their LLM spans contain the same way they find out what their logs contain: after something sensitive shows up in a dashboard. Someone pastes a support ticket into a chat feature, the instrumentation records it, the exporter ships it, and now a hosted backend holds text the user never expected to leave your app.
The fix is boring and cheap. Before you point any exporter at any backend, run synthetic prompts through your app and read the payload that would have been sent. Skip the docs and the README of the instrumentation package. Read the actual bytes.
This post walks through that workflow for OpenTelemetry GenAI prompt capture in TypeScript and Python apps. We cover what the semantic conventions say, which behavior comes from the conventions and which comes from whichever instrumentation library you installed, how to dump exporter output locally, how to read it, and where to filter. The stance is simple: the only filtering you can fully trust is filtering that happens before data leaves the process.
What the GenAI conventions say about prompt and response content
The OpenTelemetry GenAI semantic conventions define a shared vocabulary for model calls: the operation name, the provider, the requested and response model, token usage, finish reasons, and so on. Those are metadata. They tell you what happened without telling you what was said.
Prompt and response content is a separate matter. The conventions treat message content as opt-in and potentially sensitive, because prompts routinely carry user data, retrieved documents, and system instructions you consider proprietary. Content, where it is recorded at all, is modeled as structured messages (system, user, assistant, tool) rather than one blob. The conventions have also shifted over time between carrying that content as span events and carrying it as attributes. The GenAI area is still marked as in development, so names and shapes have moved between releases (OpenTelemetry GenAI semantic conventions, 2026).
Two practical consequences follow.
First, do not hard-code attribute names from a blog post, including this one. Pin the version of the conventions your instrumentation targets and read that version's spec. Second, a convention describes where content goes if you choose to record it. It does not force anyone to record it, and it does not tell you what a given library does by default.
That gap between "the spec says content is opt-in" and "my app does not emit content" is where leaks happen. The spec is a contract for instrumentation authors. Your job is to check whether the library you installed honors it the way you assume.
Which behavior is general convention and which is instrumentation-dependent
Sort what you know into two buckets, because they fail differently.
The general bucket is what the conventions define: attribute and event names, the idea that content is opt-in, the message structure. If two libraries both claim to follow the same convention version, the names they emit should match.
The instrumentation-dependent bucket is everything else, and it is bigger than people expect:
Whether content capture is on or off by default.
What switch controls it: an environment variable, a constructor option, a config flag, or a different package entirely.
Whether capture covers only the final prompt or also tool arguments, tool results, and retrieved chunks.
Whether streaming responses are captured, and whether partial output is recorded when a stream is cut off.
Whether the library follows the current conventions or an older experimental shape.
A wrapper around the OpenAI client, a Vercel AI SDK telemetry setting, a LangChain callback, and a LiteLLM integration can each answer those questions differently within the same app. If your agent uses two of them, you have two capture policies, and one of them is probably undocumented.
That is why reading documentation is not verification. Docs describe intent at a point in time. A minor version bump can flip a default. A framework can add a new span type for tool calls that carries arguments verbatim. The only reliable statement you can make is "on this version, with this config, this synthetic prompt produced these attributes."
Re-establish that statement regularly. One audit is not enough.
Inspect exporter payloads locally with synthetic prompts
The setup has three parts: a recognizable prompt, a local exporter, and a real code path.
Start with synthetic prompts that are easy to grep. Include a fake email, a fake API key shaped like a real one, a fake name, and a canary string such as CANARY-7f3a-do-not-export. Put the canary in the system prompt, the user message, a tool argument, and a retrieved document. If it shows up in output, you know exactly which path leaked it.
Next, swap your production exporter for a local one. In TypeScript, configure a local exporter to inspect output or run test assertions. In Python, use ConsoleSpanExporter with a SimpleSpanProcessor, or InMemorySpanExporter for pytest. Use the simple processor locally so spans print immediately instead of waiting for a batch. If you also emit logs, attach the console log exporter too, since some instrumentation records message content as log records rather than span data.
If you want to see the exact OTLP shape, run a local OTLP/HTTP receiver (the OpenTelemetry Collector with a debug exporter works) and point your app's OTLP endpoint at localhost:4318. That shows you the serialized payload, which is what a hosted backend would receive.
Then exercise the real code path. Call the same function your users hit, with streaming on if production streams, with tools enabled if production uses tools. A unit test that mocks the model client proves nothing about what the instrumentation records.
Finally, make it a test. Assert that the canary string does not appear anywhere in the exported span, event, and log data when capture is meant to be off. Run it in CI. That turns a one-time audit into a regression guard.
Read the output: span attributes, events, and what to look for
A console dump is noisy. Read it with a checklist instead of scrolling.
Search for the canary first. A plain text search across the full dump beats eyeballing. Hits in a span attribute, a span event, or a log record each mean a different filter point later, so note which one.
Check span attributes for content. Metadata like model name and token counts is expected. Long string values are the red flag: anything holding a prompt, a completion, or a serialized message array. Also check attributes you did not expect, such as tool arguments, request bodies, or HTTP-level attributes added by a generic HTTP instrumentation that sits underneath the LLM client.
Check span events and log records. Depending on the convention version and library, content may live in events attached to the span rather than in attributes. People filter attributes, see nothing, and miss the events entirely.
Look at the spans you did not write. Agents produce spans for tool calls, retrievals, and framework internals. Your direct model call may be clean while the retrieval span carries the full document text.
Test the failure paths. Force an error, a timeout, and a cut-off stream. Exception events can embed the offending input in the message, and error handling often bypasses the normal capture switch.
Check resource attributes. Service name, environment, and host info are fine. Check that nothing user-specific got attached at the resource level by a custom setup.
Write down what you found as a short table: path, where content appeared, which switch controls it. That table is your actual capture policy, and it is more accurate than anything in a wiki.
Filter before transmission in the instrumentation or SDK layer
Once you know where content appears, remove it at the earliest point you control. In order of preference:
1. Turn capture off at the source. Use the instrumentation's own switch, if it has one, and confirm with the canary test that it works for every path in your table. Keep model, token, latency, and error metadata. Those are what you need to debug slow or failing calls.
2. Drop or rewrite in the SDK. If the library has no switch, or the switch misses a path, intervene in the OpenTelemetry SDK before export. You can intercept data in the SDK or wrap the exporter so sensitive fields are stripped before transmission. A small allowlist beats a denylist: export only attribute keys you recognize, and everything new is dropped until someone reviews it. A denylist fails open the day a library adds a new content attribute.
3. Control where the data is created. If a tool argument holds a secret, do not rely on the tracer to hide it. Pass an identifier into the span and keep the secret out of the traced call.
Whichever you choose, put the canary test in front of it. A processor that silently throws an exception can fall back to exporting unmodified spans in some setups, so test the failure behavior too.
telemetry.dev takes a similar view on the product side: prompt capture can be turned off per environment before storage, and redaction patterns apply, but redaction cannot catch all sensitive data. That is a reason to filter in your own process first, not a reason to skip it. Pointing the SDK at telemetry.dev with capture off in production and on in a dev workspace is a reasonable split, as long as you have verified what each environment actually emits.
Why backend or Collector-side filtering is not a substitute for not sending
The tempting shortcut is to export everything and scrub it downstream: a redaction rule in the backend, a Collector attributes or transform processor, a retention policy. These are useful. They are not equivalent to never sending.
By the time a Collector processor runs, the data has already left your application. It crossed a process boundary, possibly a network boundary, and was deserialized into memory somewhere else. If that Collector runs as a sidecar or in the same VPC and you control it, the exposure may be acceptable for your threat model. If it runs as a shared gateway, or the scrubbing happens inside a hosted backend, then the content was transmitted to a third party before the rule ran.
Three specific problems with scrubbing after the fact:
Pattern matching is incomplete. Regexes catch shaped data like emails and card numbers. They miss names, addresses, medical details, and free-text confessions. No pattern set catches all sensitive data, which is why we recommend not relying on one.
Misconfiguration fails open. A typo in a processor config, a pipeline that skips the processor for one signal type, or a log pipeline that was never wired up means content flows through untouched, and nothing alerts you.
Copies exist before the scrub. Debug exporters, error logs, retry queues, and dead-letter buffers can hold the original payload.
The Collector is still the right place for the things it does well: batching, retries, routing, and holding credentials so your app does not. We cover that in OpenTelemetry Collector Environment Variables for Auth. Use it as a second layer.
The first layer is your process. Defense in depth means the earlier layer does the real work and the later layer catches mistakes. If it runs the other way around, you are cleaning up after a leak.
A pre-export checklist
Run this before the first exporter points at a hosted backend, and again after any dependency upgrade that touches OpenTelemetry or an LLM client.
Pin versions. Record the GenAI convention version and the instrumentation package versions you tested against.
Write synthetic prompts with canaries. Place unique strings in the system prompt, user message, tool arguments, and retrieved documents.
Export locally only. Console exporter, in-memory exporter, or a local OTLP receiver. No hosted endpoint yet.
Run the real paths. Streaming and non-streaming, tools on, retrieval on, plus a forced error and a cut-off stream.
Search the dump for each canary. Record whether it appeared in span attributes, span events, or log records.
Fill in the capture table. Path, location of content, controlling switch.
Apply filters at the source or in the SDK. Prefer an allowlist of attribute keys. Keep model, token, latency, and error metadata.
Re-run and assert absence. Turn the canary check into a CI test that fails if any canary reaches the exporter.
Check every framework layer. If you use the Vercel AI SDK, read How to Trace Vercel AI SDK Calls with OpenTelemetry and verify its telemetry settings against your own dump rather than assuming them.
Lock down credentials. Keep auth headers out of app code and config files. The Collector auth post covers environment variable patterns.
Set capture per environment. Content off in production by default, on only in environments where you have decided that is acceptable.
Only then point the exporter at a backend. Send a small amount of traffic first and compare what arrives with what you saw locally.
If step 12 shows anything that step 5 did not, your local setup differs from production. Fix that before scaling traffic.
Conclusion
Prompt capture is a decision, and the only way to know what you decided is to read the payload. Do it this week: add the canary test to CI, run your app against a console exporter, and write down what appears. Then turn capture off in production, keep the metadata, and confirm the test still passes.
When you are ready to send real traces, telemetry.dev takes standard OTLP/HTTP, works with the TypeScript and Python SDKs, and lets you turn prompt capture off per environment before storage. Start on the free plan, send a handful of traces from a staging environment with capture off, and compare what lands in the dashboard against your local dump. If the two match, you can widen the rollout with evidence instead of hope.
Share this article
