TASK: Observability node #

Working notes only — never commit. Source note: ../.sethetter/notes/20260816080133.md

Goal #

New node (devict-observability) running the Grafana LGTM stack via docker compose. Alloy on every monitored host pushes telemetry to it.

Decisions #

• Hostname grafana.devict.org via Caddy on the node itself
• Alloy per-host, push model; one reusable role/config for all hosts
• Apps send OTLP straight to Tempo; Alloy not in the trace path
• Grafana-managed alerts via file provisioning; embedded Alertmanager
• Access control for ingest ports: DO cloud firewall + VPC private networking (docker bypasses ufw). WireGuard mesh deferred — pure upgrade path, no rework needed.
• No infisical on this node; PDS-style env-file secrets from 1Password

Open questions #

• Alert destination: email (needs SMTP creds) vs Slack/Discord webhook
• Dashboard provisioning strategy (UI-first then export, or jsonnet)
• Exact image version pinning once validated

Links #

• Planning note: ../.sethetter/notes/20260816080133.md