diff --git a/docs/grafana-dashboard.md b/docs/grafana-dashboard.md index 0e2e64a..6570926 100644 --- a/docs/grafana-dashboard.md +++ b/docs/grafana-dashboard.md @@ -3,10 +3,35 @@ Stream reuses the upstream dashboard rather than maintaining a lookalike. The reviewed artifact is: -- upstream commit: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` +- upstream commit: `f29815c391fc2644f8a3dd36b899fb3697dd1ea6` (upstream's + repository, not this one — the path below is upstream's too) - path: `contrib/grafana/jetstream.json` - SHA-256: `ef6033c182d8c3f9d88af7d91e61126724efbe72e6b940984c9de12e9530cd91` +## Empty panels during bootstrap are expected, not broken + +Every metric named in this file is real — checked against a live instance — +but a panel only has data once the code path behind it has run at least once. +During bootstrap that leaves three genuinely absent from the metrics endpoint, +because nothing has exercised them yet: + +| metric | appears when | +|---|---| +| `jetstream_getblock_requests_total` | serving ungates; `getBlock` returns 503 until steady state | +| `jetstream_orchestrator_phase_transitions_total` | the lifecycle first leaves bootstrap | +| `jetstream_orchestrator_state_duration_seconds` | same — needs a completed phase | + +Verified on experiment 6 at 54 h elapsed: `max_over_time(jetstream_orchestrator_phase[54h])` +was still `1`, so no transition had ever occurred and those series could not +exist. Do not read their absence as a wiring fault. + +**Also: histograms have no bare series.** A `*_duration_seconds` or +`*_wait_seconds` metric exists only as `_bucket`, `_count` and `_sum`, so +grepping the metrics endpoint for the bare name finds nothing while the +histogram is present and working. Nine of the names in this file behave that +way. Query `histogram_quantile(...)` or `rate(..._sum[5m]) / rate(..._count[5m])`, +and when checking whether a metric exists, look for the suffixed series. + The exact reviewed bytes are vendored at `deploy/grafana/jetstream-upstream.json`; deployment does not depend on the network or a moving branch. Stream provisions