# Jetstream V2 and Stream Grafana dashboards Stream reuses the upstream dashboard. The reviewed artifact is: - upstream commit: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` (upstream's repository, not this one — the path below is upstream's too) - path: `contrib/grafana/jetstream.json` - SHA-256: `ef6033c182d8c3f9d88af7d91e61126724efbe72e6b940984c9de12e9530cd91` ## Empty panels during bootstrap are expected, not broken Every metric named in this file has been checked against a live instance, but a panel only has data once the code path behind it has run at least once. During bootstrap that leaves three absent from the metrics endpoint: | metric | appears when | |---|---| | `jetstream_getblock_requests_total` | serving ungates; `getBlock` returns 503 until steady state | | `jetstream_orchestrator_phase_transitions_total` | the lifecycle first leaves bootstrap | | `jetstream_orchestrator_state_duration_seconds` | same — needs a completed phase | Verified on experiment 6 at 54 h elapsed: `max_over_time(jetstream_orchestrator_phase[54h])` was still `1`, so no transition had ever occurred and those series could not exist. Do not read their absence as a wiring fault. **Also: histograms have no bare series.** A `*_duration_seconds` or `*_wait_seconds` metric exists only as `_bucket`, `_count` and `_sum`, so grepping the metrics endpoint for the bare name finds nothing while the histogram is present and working. Nine of the names in this file behave that way. Query `histogram_quantile(...)` or `rate(..._sum[5m]) / rate(..._count[5m])`, and when checking whether a metric exists, look for the suffixed series. The exact reviewed bytes are vendored at `deploy/grafana/jetstream-upstream.json`; deployment does not depend on the network or a moving branch. Stream provisions `deploy/grafana/stream.json` (SHA-256 `d00582d478cdf0c8b00d348dd953c79651c4961e161592bf5eee9f389b3b51d9`). It is deterministically generated by `tools/adapt_dashboard.py` from the pinned source. The generator refuses a changed source checksum, and the tests prove every upstream non-runtime panel remains structurally identical. Stream adds two explicit final rows. The first displays Stream's current dispatch-batch counters. Those counters are not whole-network progress, and the 2026-07-24 audit found that their durability boundary is not equivalent to upstream's per-repository completion batcher. The second covers Stream's scheduler, which has no upstream Go equivalent: ticket boundaries, unsettled stage gaps, current fatal state, and first-failure counters partitioned by stage. `just dashboard-test` verifies both checksums, regeneration, and the contract parser. `just dashboard-contract ` extracts every metric family referenced by the Stream dashboard's PromQL and compares it with the candidate's exposition, including histogram child series. It exits non-zero and prints every missing family. This is a schema-presence check, not evidence that a producer has the upstream semantic boundary or remains correct across restart. ## Reuse boundary Protocol and lifecycle rows are reused verbatim: overview, live firehose, ingest, backfill/retry, serving, storage, integrity, cutover/merge, and timestamp import. Structural reuse does not establish semantic parity. The current producer audit is in [semantic-parity.md](semantic-parity.md), and any family whose underlying behavior is blocked must be treated as diagnostic rather than equivalent. The Go runtime row is the sole adaptation. Stream does not synthesize `go_*` series. It retains upstream's same-semantics standard process measurements for start time, cumulative CPU, current RSS, and file descriptors; adds current virtual memory and OS thread count; and replaces Go GC and per-process network panels with cumulative page-fault and context-switch counters. Values come from `getrusage` plus Linux procfs or Darwin `proc_pidinfo`. Unavailable measurements are omitted, never reported as zero. `just process-metrics-contract` launches the production binary with only loopback dependencies and validates types, values, invariants, monotonicity, and the absence of `go_*` series across independent HTTP scrapes. The same receipt proves real startup RocksDB reads, point writes, and batch commits populate the canonical store histogram. A real RocksDB unit receipt covers point-read `ok`/`notfound`, durable `set`/`delete`, and atomic `batch_commit` boundaries; registered error series remain zero until RocksDB reports one. `just http-metrics-contract` seeds a real sealed archive and exercises the ReleaseSafe binary through ordinary HTTP, Range/conditional getBlock requests, and a real WebSocket upgrade/close. It proves the same public-mux middleware boundary and `xrpc/` label as upstream, all four getBlock outcomes, exact histogram counts, and full-200-only served-byte accounting. Upstream's debug `/metrics` and `/healthz` endpoints and unmatched public requests remain outside the HTTP histogram rather than manufacturing extra handler labels. The pinned official-client archive receipt also proves the resident manifest: startup loads exactly the sealed-segment count, archive-to-live replay produces real block-index hits, unknown-index unit coverage produces misses, and refcounted readers survive a verified metadata refresh. Compaction and timestamp rewrite tests prove the resident checksum/index is refreshed before the rewritten generation is published to later serving work. ## Implemented canonical producers The following dashboard families already have non-placeholder producers: - `process_start_time_seconds` - `process_cpu_seconds_total` - `process_resident_memory_bytes` - `process_virtual_memory_bytes` - `process_open_fds` - `process_max_fds` - `process_minor_page_faults_total` - `process_major_page_faults_total` - `process_voluntary_context_switches_total` - `process_involuntary_context_switches_total` - `stream_process_threads` - `stream_pipeline_ticket` - `stream_pipeline_stage_gap` - `stream_pipeline_fatal` - `stream_pipeline_failures_total` - `jetstream_build_info` - `jetstream_livestream_events_received_total` - `jetstream_livestream_decode_errors_total` - `jetstream_livestream_reconnects_total` - `jetstream_livestream_sequence_gaps_total` - `jetstream_livestream_sequence_gap_missed_seqs_total` - `jetstream_livestream_unknown_events_total` - `jetstream_livestream_stream_error_frames_total` - `jetstream_livestream_last_seen_upstream_event_timestamp_seconds` - `jetstream_livestream_replayed_account_events_dropped_total` - `jetstream_livestream_replayed_identity_events_dropped_total` - `jetstream_ingest_events_appended_total` (counted in the shared archive writer, so live, bootstrap, retry, and resync producers all contribute) - `jetstream_ingest_blocks_flushed_total` - `jetstream_ingest_segments_rotated_total` - `jetstream_ingest_append_errors_total` - `jetstream_ingest_active_segment_bytes` - `jetstream_ingest_readable_log_bytes` - `jetstream_ingest_readable_log_pinned_bytes` - `jetstream_ingest_readable_log_pinned_overrun_bytes` - `jetstream_ingest_dropped_events_total` - `jetstream_segment_seal_duration_seconds` - `jetstream_data_dir_free_bytes` - `jetstream_http_request_duration_seconds` - `jetstream_getblock_requests_total` - `jetstream_getblock_served_bytes_total` - `jetstream_getblock_duration_seconds` - `jetstream_manifest_segments_loaded` - `jetstream_manifest_block_index_cache_hits_total` - `jetstream_manifest_block_index_cache_misses_total` - `jetstream_manifest_block_index_load_seconds` - `jetstream_store_op_duration_seconds` - `jetstream_orchestrator_phase` - `jetstream_orchestrator_phase_transitions_total` - `jetstream_orchestrator_state_duration_seconds` - `jetstream_orchestrator_merge_events_kept_total` - `jetstream_orchestrator_merge_events_dropped_total` - `jetstream_orchestrator_merge_segments_consumed_total` - `jetstream_orchestrator_merge_did_lookups_total` - `jetstream_orchestrator_merge_repo_revs_updated_total` - `jetstream_orchestrator_merge_dids_discovered_post_bootstrap_total` - `jetstream_verifier_failures_total` - `jetstream_backfill_discovered_total` - `jetstream_backfill_completed_total` - `jetstream_backfill_failed_total` - `jetstream_backfill_active_flips_total` - `jetstream_backfill_on_fail_store_errors_total` - `jetstream_backfill_handle_repo_duration_seconds` - `jetstream_backfill_progress_completed` - `jetstream_backfill_completion_queued_total` - `jetstream_backfill_completion_queue_depth` - `jetstream_backfill_completion_durable_batches_total` - `jetstream_backfill_completion_durable_repos_total` - `jetstream_backfill_completion_stage_errors_total` - `jetstream_backfill_completion_queue_wait_seconds` - `jetstream_backfill_forced_checkpoint_flushes_total` - `jetstream_backfill_failed_repo_retry_passes_total` - `jetstream_backfill_failed_repo_retry_candidates_total` - `jetstream_backfill_failed_repo_retry_attempts_total` - `jetstream_backfill_failed_repo_retry_succeeded_total` - `jetstream_backfill_failed_repo_retry_failed_total` - `jetstream_backfill_failed_repo_retry_skipped_host_parked_total` - `jetstream_subscribe_subscribers` - `jetstream_subscribe_events_sent_total` - `jetstream_subscribe_bytes_sent_total` - `jetstream_subscribe_bytes_encoded_total` - `jetstream_subscribe_events_filtered_total` - `jetstream_subscribe_events_oversize_total` - `jetstream_subscribe_events_skipped_total` - `jetstream_subscribe_encode_errors_total` - `jetstream_subscribe_options_updates_total` - `jetstream_subscribe_options_update_errors_total` - `jetstream_subscribe_cursor_requests_total` - `jetstream_subscribe_cursor_resolve_seconds` - `jetstream_subscribe_hot_reads_total` - `jetstream_subscribe_cold_reads_total` - `jetstream_subscribe_adversarial_drops_total` - `jetstream_subscribe_clean_disconnects_total` - `jetstream_import_jobs_total` - `jetstream_import_job_duration_seconds` - `jetstream_import_phase` - `jetstream_import_rows_parsed_total` - `jetstream_import_rows_rejected_total` - `jetstream_import_rows_mutated_total` - `jetstream_import_rows_matched_total` - `jetstream_import_rows_corrupt_offset_total` - `jetstream_import_dids_matched_total` - `jetstream_import_segments_examined_total` - `jetstream_import_segments_patched_total` - `jetstream_import_bytes_rewritten_total` - `jetstream_compaction_passes_total` - `jetstream_compaction_pass_duration_seconds` - `jetstream_compaction_passes_early_total` - `jetstream_compaction_tombstones_collected_total` - `jetstream_compaction_tombstone_set_entries` - `jetstream_compaction_tombstone_set_bytes` - `jetstream_compaction_segments_examined_total` - `jetstream_compaction_segments_rewritten_total` - `jetstream_compaction_segments_skipped_clean_total` - `jetstream_compaction_manifest_reconciled_total` - `jetstream_compaction_rows_dropped_total` - `jetstream_compaction_bytes_rewritten_total` - `jetstream_compaction_watermark_seq` - `jetstream_compaction_watermark_lag_seconds` The Stream-only backfill chart plots absolute repository counts across the current run over time: `discovered`, `processed`, and `durably committed`. It deliberately does not present a whole-network percentage while `listRepos` is still discovering the denominator. Upstream requests are pages of 1,000 entries; the production default accumulates roughly 100,000 entries into a shuffled dispatch/checkpoint batch before advancing durable progress. The backfill completion families are measured at the writer durability boundary, independently of the dispatch/checkpoint batch. A successful CAR records its final archive sequence; ordinary writer fsyncs commit every covered repository state, and the terminal dispatch drain refuses to advance the listRepos cursor if any queued completion is not covered. Queue and durable batch/repository metrics therefore describe the same boundaries as upstream. The Prometheus counters themselves are process-local; restart-stable progress must be reconstructed from durable repository-state counts. Compaction metrics are emitted at the actual fold, worker rewrite, manifest refresh, watermark commit, live-set, and timer boundaries. The production offline receipt reports `required=104 present=104 missing=0`. That result means only that every queried family is exposed; it does not establish producer semantics or close the deployment gate. ## Dashboard inventory (source of truth: deploy/grafana/) | uid | title | audience | notes | |---|---|---|---| | `stream-glance` | stream / at a glance | operator | 4 rows: pipeline alive (ingest rate, upstream freshness, reconnects+gaps) / consumers served (subscribers, delivery, archive bytes) / margins+faults (disk free, watermark lag, flat-zero fault rates) / memory-allocator (rss vs malloc live, free-held, arena heaps — zlay lesson 8) | | `stream-consumers` | stream / archive consumers | operator | per-key throughput, cumulative, 429s, bucket headroom, configured rates | | `stream-public-full` | Stream / Jetstream V2 | public | adapted upstream dashboard (see top of this file) | To edit: change the JSON in `deploy/grafana/`, then POST it — python3 -c 'import json; d=json.load(open("deploy/grafana/.json")); \ print(json.dumps({"dashboard": d, "overwrite": True}))' > /tmp/post.json # box: admin password in /opt/stream-experiment/grafana-password.txt # POST /tmp/post.json to http://localhost:3000/api/dashboards/db inside # the grafana container with basic auth (busybox wget needs an explicit # "Authorization: Basic " header; URL-embedded creds are ignored) Edits made in the Grafana UI are not durable state — anything worth keeping gets exported back into `deploy/grafana/`. Panels must only name series that exist on the live instance: a panel over an unshipped metric renders "No data" and reads as an outage. Series shipping with the next stream deploy — import their panels after it, not before: admission counters (`stream_subscribe_capacity_rejects_total`, `stream_subscribe_cold_contended_total`) and the all-arena malloc gauges (`stream_malloc_system_bytes`, `stream_malloc_free_total_bytes`, `stream_malloc_arena_heaps` — the glance memory row depends on these).