From dae40f726faa63132f6811977d0cdfb972a41a35 Mon Sep 17 00:00:00 2001 From: zzstoatzz Date: Thu, 30 Jul 2026 05:57:49 -0500 Subject: [PATCH] docs: say which dashboard panels are expected empty during bootstrap Checked all 118 metric names in grafana-dashboard.md against a live instance's series list. Every one is real. Thirteen appeared missing, and the reason splits two ways worth writing down, because both make a working dashboard look broken. Nine are histograms, which exist only as _bucket/_count/_sum -- grepping the metrics endpoint for the bare *_duration_seconds name finds nothing while the histogram is fine. Recorded that, with how to query them. Three are genuinely absent from the endpoint and should be: getblock_requests_total needs serving to ungate, and the two orchestrator phase-transition metrics need the lifecycle to leave bootstrap. All three are defined in src, awaiting a first observation. Confirmed empirically on experiment 6 -- max_over_time(jetstream_orchestrator_phase[54h]) was still 1, so no transition had occurred and those series could not exist. An operator opening this dashboard mid-bootstrap sees empty panels and has no way to tell "not wired" from "hasn't happened yet". Now there is a table. Also marks the upstream commit and contrib/ path as upstream's, consistent with the other docs citing upstream sources. Co-Authored-By: Claude Opus 5 (1M context) --- docs/grafana-dashboard.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/docs/grafana-dashboard.md b/docs/grafana-dashboard.md index 0e2e64a..6570926 100644 --- a/docs/grafana-dashboard.md +++ b/docs/grafana-dashboard.md @@ -3,10 +3,35 @@ Stream reuses the upstream dashboard rather than maintaining a lookalike. The reviewed artifact is: -- upstream commit: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` +- upstream commit: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` (upstream's + repository, not this one — the path below is upstream's too) - path: `contrib/grafana/jetstream.json` - SHA-256: `ef6033c182d8c3f9d88af7d91e61126724efbe72e6b940984c9de12e9530cd91` +## Empty panels during bootstrap are expected, not broken + +Every metric named in this file is real — checked against a live instance — +but a panel only has data once the code path behind it has run at least once. +During bootstrap that leaves three genuinely absent from the metrics endpoint, +because nothing has exercised them yet: + +| metric | appears when | +|---|---| +| `jetstream_getblock_requests_total` | serving ungates; `getBlock` returns 503 until steady state | +| `jetstream_orchestrator_phase_transitions_total` | the lifecycle first leaves bootstrap | +| `jetstream_orchestrator_state_duration_seconds` | same — needs a completed phase | + +Verified on experiment 6 at 54 h elapsed: `max_over_time(jetstream_orchestrator_phase[54h])` +was still `1`, so no transition had ever occurred and those series could not +exist. Do not read their absence as a wiring fault. + +**Also: histograms have no bare series.** A `*_duration_seconds` or +`*_wait_seconds` metric exists only as `_bucket`, `_count` and `_sum`, so +grepping the metrics endpoint for the bare name finds nothing while the +histogram is present and working. Nine of the names in this file behave that +way. Query `histogram_quantile(...)` or `rate(..._sum[5m]) / rate(..._count[5m])`, +and when checking whether a metric exists, look for the suffixed series. + The exact reviewed bytes are vendored at `deploy/grafana/jetstream-upstream.json`; deployment does not depend on the network or a moving branch. Stream provisions -- 2.51.2