[FEAT] Support custom OTEL exporter headers for OTLP collector #4963

Summary

Hatchet's server-side OTLP export only allows a single Authorization header via SERVER_OTEL_COLLECTOR_AUTH. Many self-hosted users need additional headers (e.g.X-API-Key, X-Tenant) when integrating with managed observability backends like Volcengine, or self-hosted Tempo behind a reverse proxy.

This adds support for arbitrary custom headers on both trace and metrics OTLP exporters.

Proposed Solution

New env var SERVER_OTEL_COLLECTOR_HEADERS (YAML key otel.collectorHeadersString), comma-separated key=value pairs, matching the standard OTel SDK's OTEL_EXPORTER_OTLP_HEADERS:

SERVER_OTEL_COLLECTOR_HEADERS="X-API-Key=foo,X-Tenant=bar"

Behavior

• Empty by default → no change for existing deployments.
• Custom headers merged into the existing header map alongside Authorization.
• Backwards compatible: SERVER_OTEL_COLLECTOR_AUTH still works; if Authorization is set via the new var it wins.

Scope

Server/Engine only. Worker SDK can be a follow-up.

Implementation Sketch

1. Add loaderutils.ParseHeaders for k=v,k2=v2 parsing.
2. Add CollectorHeadersString / CollectorHeaders to OpenTelemetryConfigFile.
3. Bind SERVER_OTEL_COLLECTOR_HEADERS and split at loader time.
4. Add CollectorHeaders to TracerOpts; small buildHeaders(opts) helper used by InitTracer and InitMeter.
5. Thread through the four call sites.
6. Unit tests for parser + merge helper.
7. Update self-hosting docs.

Happy to take this on once accepted and assigned.