OpenTelemetry Collector Environment Variables for Auth

A token pasted into collector.yaml tends to survive longer than anyone intended. It gets committed, copied into a Docker image, pasted into a Slack thread while someone debugs a 401, and then it lives in four places you forgot about. The Collector has a built-in answer for this: the ${env:VAR} syntax, which reads a value from the process environment when the config loads.

That feature is small, and people ask too much of it. It is not a secrets manager. It is not authentication. It does one thing: it substitutes text before the Collector parses your YAML. Everything else (what header gets sent, whether a backend accepts it, why you got a 403) is a separate problem with separate fixes.

This guide is narrow on purpose. It covers environment variable expansion for auth headers in the Collector: the syntax that works, the exporter-side and receiver-side auth extensions that people confuse, the YAML gotchas that bite, and how to debug HTTP errors without printing your token into a log. Examples use placeholder names only. Check anything version-sensitive against the official Collector configuration docs before you ship it.

What environment variable expansion does in Collector config (and what it doesn't)

Expansion is config-time text substitution. When the Collector loads your YAML, a config provider resolves references like ${env:MY_TOKEN} and replaces them with the variable's value from the Collector process environment. After that, the Collector parses the result as ordinary configuration. It never knows a variable was involved.

That definition rules out a lot of things people assume it does.

It does not authenticate anything. Setting an environment variable called BACKEND_TOKEN sends no credentials anywhere. A token only reaches a backend if some exporter setting, such as a headers entry, puts it on the wire.

It does not protect the secret at runtime. The value sits in the process environment, so anyone who can read the process environment or the container spec can read it. Expansion keeps the token out of your repository. It doesn't make the token safe.

It does not refresh. The value is resolved when the configuration loads. If you rotate the token, the running Collector keeps using the old one until it restarts or reloads config.

It does not validate. If the variable is unset, you don't get a helpful auth error. Depending on the syntax you used, you get an empty string, and the first sign of trouble is a 401 from your backend.

Treat expansion as a delivery mechanism for one value into one place in a file. Keep the secret in whatever your platform uses for secrets (Kubernetes Secrets, a CI variable store, a cloud secret manager), inject it into the Collector's environment, and let ${env:VAR} carry it the last inch into the YAML.

Supported syntax: ${env:VAR}, defaults, and the deprecated $VAR form

Use the explicit form. In current Collector versions the supported syntax is ${env:VAR_NAME}, where the env: prefix tells the config resolver to read from environment variables.

You can supply a default with a colon-dash after the name:

headers:
  x-environment: ${env:DEPLOY_ENV:-dev}

If DEPLOY_ENV is unset, the value becomes dev. Defaults are fine for labels and endpoints. They are a bad idea for tokens. A default token value means a missing secret turns into a plausible-looking wrong secret, and you debug a 401 instead of a clear startup failure. For credentials, skip the default.

The older forms, $VAR and ${VAR} without the env: prefix, are legacy shorthand that the Collector project has been moving away from. Older versions accepted them, newer versions have deprecated them, and behavior around them has shifted between releases. Don't write new config with them. If you inherit a file that uses them, convert it to ${env:VAR} and check your Collector version's release notes for the exact deprecation timeline. I'd rather you verify than trust my memory of a specific version number.

Three habits will save you time:

1. Always use the env: prefix so a reader of the YAML can tell where the value comes from.

2. Pick variable names that say what they hold and for whom, such as BACKEND_OTLP_TOKEN, not TOKEN.

3. Pin your Collector version in your deployment and read the configuration docs for that version. Expansion behavior is part of the configuration resolution machinery and has changed across releases.

If you need a literal dollar sign in a value, that is an escaping problem, covered in the gotchas section below.

Putting a token in an otlphttp exporter header

The most common job is sending an API token to an OTLP/HTTP backend. The otlphttp exporter takes a headers map, and that is where an Authorization header goes. Expansion fills in the value.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlphttp:
    endpoint: ${env:BACKEND_OTLP_ENDPOINT}
    headers:
      Authorization: "Bearer ${env:BACKEND_OTLP_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

Then start the Collector with the variables in its environment:

export BACKEND_OTLP_ENDPOINT="https://backend.example.com"
export BACKEND_OTLP_TOKEN="<your-token-here>"
otelcol --config collector.yaml

Notice the quotes around the Authorization value. The string Bearer followed by a space and an expansion is still a single YAML scalar, and quoting it keeps the parser from tripping on the colon-adjacent characters.

The header name and scheme depend on your backend, so read its docs. Some backends want Authorization: Bearer, others want a custom header such as an API key header. The Collector doesn't care. It sends whatever string you build.

If you point this at telemetry.dev, the same pattern applies, since telemetry.dev accepts any OTLP/HTTP exporter or OpenTelemetry Collector. Take the endpoint and credential format from your workspace settings and the OTLP reference rather than from an example in a blog post, and put the token in an environment variable, never the file.

If you run a TypeScript or Python app that already emits OTLP, you can point the SDK at a local Collector and keep the token in exactly one place: the Collector's environment. Your app code never sees it. For a worked instrumentation example on the application side, see how to trace Vercel AI SDK calls with OpenTelemetry.

Exporter auth vs receiver auth: bearertokenauth does two different jobs

This is where people lose an afternoon. The bearertokenauth extension is one component with two roles, and which role it plays depends entirely on where you attach it.

On the exporter side, it makes the Collector send a bearer token to a backend. You define the extension, then reference it from the exporter's auth setting. The token goes out on every request, in an Authorization header.

On the receiver side, it makes the Collector require a bearer token from whoever sends data in. You reference the same kind of extension from a receiver's auth setting, and incoming requests without a matching token get rejected.

Same extension name, opposite directions. A config that attaches it to the receiver does nothing to protect your outbound call, and a config that attaches it to the exporter does not lock down your OTLP port.

extensions:
  bearertokenauth/backend:
    token: ${env:BACKEND_OTLP_TOKEN}

exporters:
  otlphttp:
    endpoint: ${env:BACKEND_OTLP_ENDPOINT}
    auth:
      authenticator: bearertokenauth/backend

service:
  extensions: [bearertokenauth/backend]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp]

That is exporter-side auth. The extension also has to be listed under service.extensions, or it never starts.

Which should you use for outbound tokens: a raw headers entry or the extension? For a single static bearer token, the headers approach is shorter and easier to read. The extension fits better when your auth needs go beyond a plain header, and it keeps the credential in a dedicated named component. Either way, expansion is what keeps the secret out of the file.

If you also want to protect your own receiver, that is a separate decision with a separate extension instance and a separate token. Don't reuse the backend token to guard your inbound port. Different secret, different blast radius. Check the extension's README in the Collector contrib repository for the options your version supports.

Gotchas: YAML typing, quoting, escaping literal $, and the process environment

Most expansion bugs are YAML bugs wearing a disguise. Four to watch for.

YAML typing happens after expansion. Because substitution runs first, the resolved text is then typed by YAML rules. A token that happens to be all digits, or the word true or null, can be parsed as a number, boolean or null instead of a string. Quote values where the type matters. Authorization: "Bearer ${env:BACKEND_OTLP_TOKEN}" is safe because the literal prefix forces a string. A bare header value like x-api-key: ${env:API_KEY} is not, if the key could ever look like a number. Write x-api-key: "${env:API_KEY}" and move on.

Quoting and special characters. A secret containing a colon, hash or leading special character can change how YAML reads the line once substituted. Quoting the whole value is cheap insurance.

Escaping a literal dollar sign. If a value genuinely contains a dollar sign that must not be expanded, you have to escape it. The Collector's convention is to double it, so $$ becomes a literal $. Confirm the exact escape rule in the configuration docs for your version. A password with a dollar sign in it is exactly the kind of value that silently gets mangled.

The process environment is the only source. The variable has to exist in the environment of the Collector process itself. Setting it in your shell and then launching the Collector through systemd, Docker or Kubernetes does not carry it across. In Docker, pass it with an env flag or an env file. In Kubernetes, set it on the container from a Secret. In systemd, use Environment or EnvironmentFile in the unit. When expansion yields an empty string, nine times out of ten the variable simply never reached the process.

A fast check: run the Collector's validate command against your config in the same environment it will run in. It catches syntax errors, though it won't tell you the token is correct.

Debugging 401, 403 and other HTTP errors without printing secrets

When the backend rejects your export, the Collector log tells you the status code. Read it literally.

A 401 means the backend did not accept your identity. The usual causes: the variable was empty or unset, the token is wrong or expired, the header name or scheme is wrong (Bearer versus a custom key header), or you sent the token to the wrong endpoint. Suspect the empty variable first.

A 403 means the backend understood who you are and refused anyway. Look at permissions: a token scoped to a different project or workspace, a token without write access, or an IP or origin restriction. A 403 with a correct token is usually an authorization setting on the backend side, not a Collector bug.

A 400 or 404 points at the endpoint or path. The exporter appends the signal path (such as /v1/traces) to its configured base endpoint, and an incorrect path can cause request failures.

Now the secrets discipline, because debugging is where tokens leak.

1. Don't turn on verbose logging that dumps outgoing headers and then paste the output into a ticket. If you must raise log verbosity, treat the resulting logs as sensitive and scrub them before sharing.

2. The Collector can print its resolved configuration in some modes. Resolved config contains the expanded token. Never share that output, and know that your own log pipeline may be capturing it.

3. Confirm the variable without displaying it. Check that it is set and non-empty, or print its length, instead of echoing the value.

4. Test the token with a single curl call from a trusted shell, with the Authorization header read from the environment, so the secret never lands in shell history. A leading space on the command, or reading from a file, helps.

5. Rotate the token if it ever lands in a log, a chat message or a screenshot. Assume exposure and move on.

Diagnose from status codes and structure, not from the secret's value.

Where this fits with telemetry.dev's OpenTelemetry setup path

A Collector in front of your AI app is a clean place to hold credentials. Your TypeScript or Python service emits OTLP to a local or sidecar Collector with no secret attached. The Collector holds the token in its environment, expands it into the exporter header, and forwards to the backend. Rotating the token means restarting the Collector, not redeploying every service.

telemetry.dev is built on OpenTelemetry and works with any OTLP/HTTP exporter or OpenTelemetry Collector, so the otlphttp exporter pattern above is the whole integration. You point the exporter at your workspace's endpoint, put the credential in an environment variable, and use ${env:VAR} to carry it into the config. Exact endpoint and credential details live in the OTLP reference and your workspace settings, so use those, not placeholders from this post.

Two scoping notes. First, the Collector only forwards what your application emits. If a call isn't instrumented, it won't appear as a span no matter how clean your auth is. Second, telemetry.dev also offers first-party TypeScript and Python SDKs if you'd rather skip the Collector and send directly. The Collector route earns its keep when you already run one, want a single place for credentials and batching, or ship telemetry from several services.

The free plan covers 30,000 ingestion units per month with 14-day retention, which is enough to prove the pipeline end to end before you wire it into production.

Conclusion

Keep the rule short: the token lives in the Collector's environment, ${env:BACKEND_OTLP_TOKEN} carries it into one exporter header, and nothing secret ever appears in YAML, git or a pasted log. Quote the value, skip defaults for credentials, and read 401 as identity and 403 as permissions.

Then prove it with real data. Create a free telemetry.dev workspace, point an otlphttp exporter at it using the endpoint and credential from your workspace settings, and send traces from one TypeScript or Python AI call. If the span shows up with latency, tokens and errors attached, your auth path is correct and you can roll the same Collector config out to the rest of your services.