Appview traces #

Appview tracing is optional. The exporter uses the OpenTelemetry OTLP/HTTP protocol. VictoriaTraces accepts traces at /insert/opentelemetry/v1/traces.

On Misaki, set this variable in the appview service environment:

PROVENANCE_TRACE_ENDPOINT=http://127.0.0.1:10428/insert/opentelemetry/v1/traces

Use the full ingest URL. Port 10429 serves the read-only proxy, not the ingest API. An empty variable disables tracing. The service name is provenance-appview.

In the NixOS module, set the optional endpoint:

services.provenance.appview.traceEndpoint =
  "http://127.0.0.1:10428/insert/opentelemetry/v1/traces";

The default is null, which disables tracing.

Traces cover database open, migrations, index preparation, each index drop and rebuild, and replay transactions. Batch spans include state lookups, subject lookups, record writes, and commits. Each batch uses a separate trace to limit trace size during a full archive replay. Progress logs include the worker trace ID. Startup and index-operation logs include their trace and span IDs. For batch-operation start logs, set PROVENANCE_LOG_LEVEL=debug. An operation-start log identifies a stalled operation before its span ends. VictoriaTraces receives only completed spans.

The exporter uses a non-blocking queue of 1,024 spans and batches of up to 128 spans. It drops spans when the queue is full. Export attempts have a two-second timeout and no retry. Shutdown has a three-second limit. Collector failures do not change replay results or readiness.

Traces exclude credentials, DIDs, record keys, SQL text, SQL parameters, and error messages. Error spans contain only a generic failure status. Batch attributes contain the batch size and checkpoint. Endpoint URLs cannot contain credentials, query parameters, or fragments. Keep the collector on loopback or use HTTPS across a network.

OTLP exporter environment variables also apply, including OTEL_EXPORTER_OTLP_TRACES_HEADERS for collector authentication. Keep authentication headers in secret service configuration. Do not put them in source files.