# TASK: Observability node

Working notes only — never commit. Source note: `../.sethetter/notes/20260816080133.md`

## Goal

New node (`devict-observability`) running the Grafana LGTM stack via docker
compose. Alloy on every monitored host pushes telemetry to it.

## Decisions

- Hostname `grafana.devict.org` via Caddy on the node itself
- Alloy per-host, push model; one reusable role/config for all hosts
- Apps send OTLP straight to Tempo; Alloy not in the trace path
- Grafana-managed alerts via file provisioning; embedded Alertmanager
- Access control for ingest ports: DO cloud firewall + VPC private
  networking (docker bypasses ufw). WireGuard mesh deferred — pure
  upgrade path, no rework needed.
- No infisical on this node; PDS-style env-file secrets from 1Password

## Open questions

- Alert destination: email (needs SMTP creds) vs Slack/Discord webhook
- Dashboard provisioning strategy (UI-first then export, or jsonnet)
- Exact image version pinning once validated

## Links

- Planning note: ../.sethetter/notes/20260816080133.md