# OpenTelemetry tracing

Authority: pinned Jetstream V2 commit
`f29815c391fc2644f8a3dd36b899fb3697dd1ea6`, specifically
`internal/obs/tracing.go`, `internal/obs/observe.go`, and its pinned
OpenTelemetry Go SDK v1.44.0.

Stream uses the sibling `otel-zig` SDK and OTLP exporter. No endpoint means a
no-op global provider. Setting either endpoint below installs a batched
provider, merges detected process/resource attributes with `service.name` and
`service.version`, and flushes it within the ordinary graceful-shutdown budget.
Both modes install the same process-wide W3C TraceContext plus Baggage composite
propagator as upstream, so an absent exporter does not discard incoming context.

| Environment | Behavior |
|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | generic base URL; `/v1/traces` is appended |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | trace URL used verbatim; takes precedence |
| `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_TRACES_HEADERS` | comma-separated HTTP headers; trace-specific replaces generic |
| `OTEL_EXPORTER_OTLP_COMPRESSION`, `OTEL_EXPORTER_OTLP_TRACES_COMPRESSION` | exact value `gzip` enables gzip |
| `OTEL_EXPORTER_OTLP_INSECURE`, `OTEL_EXPORTER_OTLP_TRACES_INSECURE` | `true` selects HTTP; every other non-empty value selects HTTPS |
| `OTEL_EXPORTER_OTLP_TIMEOUT`, `OTEL_EXPORTER_OTLP_TRACES_TIMEOUT` | request timeout in milliseconds |
| `OTEL_EXPORTER_OTLP_CERTIFICATE`, `OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE` | PEM certificate-authority bundle; trace-specific replaces generic |
| `OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE`, `OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE` | PEM client certificate; used only with the matching client key |
| `OTEL_EXPORTER_OTLP_CLIENT_KEY`, `OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY` | PEM client private key; used only with the matching client certificate |
| `OTEL_BSP_SCHEDULE_DELAY` | maximum batching delay in milliseconds |
| `OTEL_BSP_MAX_QUEUE_SIZE` | bounded completed-span queue |
| `OTEL_BSP_MAX_EXPORT_BATCH_SIZE` | maximum spans per request |
| `OTEL_BSP_EXPORT_TIMEOUT` | maximum milliseconds for one complete export, including requests, retry backoff, and server throttling |
| `OTEL_TRACES_SAMPLER` | `always_on`, `always_off`, `traceidratio`, or their three `parentbased_` forms |
| `OTEL_TRACES_SAMPLER_ARG` | ratio in the inclusive range 0 through 1 |

Pinned upstream constructs `otlptracehttp` directly, so it always sends
OTLP/HTTP protobuf; `OTEL_EXPORTER_OTLP_PROTOCOL` and its trace-specific form do
not select JSON there and do not select JSON in Stream.

The executable accepts `--otel-service-name`; `OTEL_SERVICE_NAME` supplies its
default and `jetstream` is the fallback. CLI values take precedence.

## Production span boundaries

Stream carries one explicit trace context across the same asynchronous and
durability boundaries as pinned upstream. The implemented span names are:

| Scope | Spans |
|---|---|
| `jetstream/ingest/orchestrator` | `Run`, `runBootstrap`, `finishBootstrap`, `runMerge`, merge-runner `run`, `processSourceSegment`, `runDiscovery`, `runSteadyState`, `runDeleteCompaction`, `RunImport` |
| `jetstream/ingest/backfill` | `Run`, `handleRepo` (including retry and live resync, with `did`) |
| `jetstream/ingest/live` | `processBatch` |
| `jetstream/ingest` | `commitAsyncFlush`, `rotateIfFull`, `flushAndRotateLocked`, `rotateLocked` (with `active_idx`) |
| `jetstream/xrpcapi` | `getBlock` (with `segment.idx`, `block.index`, `block.compressed_size`, and `result`) |

Spans end at the operation boundary, record returned errors, and mark
successful operations `OK`. Repository downloads remain children of the
backfill run; `handleRepo` begins only after a CAR has been accepted, matching
upstream's materialization boundary. Writer spans occur per block or rotation,
never per event.

## Offline receipt

The `runtime/observability.zig` contract starts a loopback HTTP
collector, installs the production provider, records a sampled span, and then
performs bounded shutdown. The collector asserts the protobuf content type,
endpoint path, configured header, instrumentation scope, span name,
`service.name`, and `service.version` from the received request body. Separate
tests cover no-op behavior and environment precedence/fallbacks. This receipt
requires no external network. The no-endpoint receipt also inspects the global
propagator fields, while `otel-zig` exercises traceparent, tracestate, and
W3C baggage injection/extraction with upstream's strict invalid-header behavior.
The exporter receipt drives 503 and 429 responses before a successful 202,
checks integer `Retry-After`, and verifies both the request timeout and the BSP
export timeout cancel their respective I/O boundaries. Retry defaults match
upstream: 5-second initial interval, 1.5 multiplier with jitter, 30-second cap,
60-second retry budget, and only 429/502/503/504 responses are retried.

The pinned `otel-zig` receipt generates an ephemeral CA, server certificate,
and client certificate and drives two local HTTPS collectors. One verifies
the native Zig transport trusts only the configured CA. The other requires a
verified client certificate, first rejects a client without it, and then
accepts the OTLP exporter through its dynamically loaded libcurl transport.
Stream's parser receipt separately
covers trace-specific precedence and upstream's behavior of ignoring an
incomplete client certificate/key pair.

`runtime/tracing_integration_test.zig` then drives the production archive in
both asynchronous and synchronous modes. An OTLP collector requires the
exported parent `processBatch` plus `commitAsyncFlush`, `rotateIfFull`,
`rotateLocked`, and `flushAndRotateLocked` children and both canonical scopes.
The `xrpcapi.zig` receipt seals a JSS segment, serves its compressed block,
then exercises a malformed request. Its collector requires both exported
spans, canonical XRPC scope and attributes, `ok`/`bad_request` results, and the
recorded failure event.

The production Debian image installs `libcurl4` for the client-authenticated
path. Custom-CA-only and ordinary HTTPS exports use Zig's native TLS client and
do not load libcurl.