zat.dev / stream
jetstream v2 in zig stream.waow.tech
Star
Fork 0
Watch
stream docs opentelemetry.md
6.2 kB
Markdown
at zig-0.17

OpenTelemetry tracing #

Authority: pinned Jetstream V2 commit f29815c391fc2644f8a3dd36b899fb3697dd1ea6, specifically internal/obs/tracing.go, internal/obs/observe.go, and its pinned OpenTelemetry Go SDK v1.44.0.

Stream uses the sibling otel-zig SDK and OTLP exporter. No endpoint means a no-op global provider. Setting either endpoint below installs a batched provider, merges detected process/resource attributes with service.name and service.version, and flushes it within the ordinary graceful-shutdown budget. Both modes install the same process-wide W3C TraceContext plus Baggage composite propagator as upstream, so an absent exporter does not discard incoming context.

Environment Behavior
OTEL_EXPORTER_OTLP_ENDPOINT generic base URL; /v1/traces is appended
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT trace URL used verbatim; takes precedence
OTEL_EXPORTER_OTLP_HEADERS, OTEL_EXPORTER_OTLP_TRACES_HEADERS comma-separated HTTP headers; trace-specific replaces generic
OTEL_EXPORTER_OTLP_COMPRESSION, OTEL_EXPORTER_OTLP_TRACES_COMPRESSION exact value gzip enables gzip
OTEL_EXPORTER_OTLP_INSECURE, OTEL_EXPORTER_OTLP_TRACES_INSECURE true selects HTTP; every other non-empty value selects HTTPS
OTEL_EXPORTER_OTLP_TIMEOUT, OTEL_EXPORTER_OTLP_TRACES_TIMEOUT request timeout in milliseconds
OTEL_EXPORTER_OTLP_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CERTIFICATE PEM certificate-authority bundle; trace-specific replaces generic
OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE, OTEL_EXPORTER_OTLP_TRACES_CLIENT_CERTIFICATE PEM client certificate; used only with the matching client key
OTEL_EXPORTER_OTLP_CLIENT_KEY, OTEL_EXPORTER_OTLP_TRACES_CLIENT_KEY PEM client private key; used only with the matching client certificate
OTEL_BSP_SCHEDULE_DELAY maximum batching delay in milliseconds
OTEL_BSP_MAX_QUEUE_SIZE bounded completed-span queue
OTEL_BSP_MAX_EXPORT_BATCH_SIZE maximum spans per request
OTEL_BSP_EXPORT_TIMEOUT maximum milliseconds for one complete export, including requests, retry backoff, and server throttling
OTEL_TRACES_SAMPLER always_on, always_off, traceidratio, or their three parentbased_ forms
OTEL_TRACES_SAMPLER_ARG ratio in the inclusive range 0 through 1

Pinned upstream constructs otlptracehttp directly, so it always sends OTLP/HTTP protobuf; OTEL_EXPORTER_OTLP_PROTOCOL and its trace-specific form do not select JSON there and do not select JSON in Stream.

The executable accepts --otel-service-name; OTEL_SERVICE_NAME supplies its default and jetstream is the fallback. CLI values take precedence.

Production span boundaries #

Stream carries one explicit trace context across the same asynchronous and durability boundaries as pinned upstream. The implemented span names are:

Scope Spans
jetstream/ingest/orchestrator Run, runBootstrap, finishBootstrap, runMerge, merge-runner run, processSourceSegment, runDiscovery, runSteadyState, runDeleteCompaction, RunImport
jetstream/ingest/backfill Run, handleRepo (including retry and live resync, with did)
jetstream/ingest/live processBatch
jetstream/ingest commitAsyncFlush, rotateIfFull, flushAndRotateLocked, rotateLocked (with active_idx)
jetstream/xrpcapi getBlock (with segment.idx, block.index, block.compressed_size, and result)

Spans end at the operation boundary, record returned errors, and mark successful operations OK. Repository downloads remain children of the backfill run; handleRepo begins only after a CAR has been accepted, matching upstream's materialization boundary. Writer spans occur per block or rotation, never per event.

Offline receipt #

The runtime/observability.zig contract starts a loopback HTTP collector, installs the production provider, records a sampled span, and then performs bounded shutdown. The collector asserts the protobuf content type, endpoint path, configured header, instrumentation scope, span name, service.name, and service.version from the received request body. Separate tests cover no-op behavior and environment precedence/fallbacks. This receipt requires no external network. The no-endpoint receipt also inspects the global propagator fields, while otel-zig exercises traceparent, tracestate, and W3C baggage injection/extraction with upstream's strict invalid-header behavior. The exporter receipt drives 503 and 429 responses before a successful 202, checks integer Retry-After, and verifies both the request timeout and the BSP export timeout cancel their respective I/O boundaries. Retry defaults match upstream: 5-second initial interval, 1.5 multiplier with jitter, 30-second cap, 60-second retry budget, and only 429/502/503/504 responses are retried.

The pinned otel-zig receipt generates an ephemeral CA, server certificate, and client certificate and drives two local HTTPS collectors. One verifies the native Zig transport trusts only the configured CA. The other requires a verified client certificate, first rejects a client without it, and then accepts the OTLP exporter through its dynamically loaded libcurl transport. Stream's parser receipt separately covers trace-specific precedence and upstream's behavior of ignoring an incomplete client certificate/key pair.

runtime/tracing_integration_test.zig then drives the production archive in both asynchronous and synchronous modes. An OTLP collector requires the exported parent processBatch plus commitAsyncFlush, rotateIfFull, rotateLocked, and flushAndRotateLocked children and both canonical scopes. The xrpcapi.zig receipt seals a JSS segment, serves its compressed block, then exercises a malformed request. Its collector requires both exported spans, canonical XRPC scope and attributes, ok/bad_request results, and the recorded failure event.

The production Debian image installs libcurl4 for the client-authenticated path. Custom-CA-only and ordinary HTTPS exports use Zig's native TLS client and do not load libcurl.