# Observability

Central observability node for devICT infrastructure: the Grafana LGTM
stack (Loki, Mimir, Tempo + Grafana), with Grafana Alloy as the collector
on every monitored host.

- Terraform provisions the droplet (`devict-observability`) + cloud firewall.
- Ansible installs Docker + renders one compose stack per role under
  `/opt/devict/<role>/`, each wrapped in a systemd unit.
- Caddy terminates TLS for `grafana.devict.org` -> `grafana:3000`.

## Architecture

- **Alloy runs on every host** (this one included) and pushes: host metrics
  (embedded node_exporter) + its own metrics -> Mimir, systemd journal +
  docker container logs -> Loki. Push model: monitored hosts need no
  inbound ports, only outbound to this node.
- **Apps send OTLP traces directly to Tempo** (:4317). Alloy is not in the
  trace path.
- **Backends run unauthenticated** (`auth_enabled: false`). All telemetry
  crosses a **WireGuard mesh** (hub-and-spoke, this node is the hub,
  `10.77.0.0/24`; shared `wireguard` role in `../shared/roles`). The
  ingest ports are not opened in the cloud firewall at all — the tunnel
  is the only path in, and every peer is authenticated by key. Note:
  docker-published ports bypass ufw and wg0 traffic bypasses the cloud
  firewall (it only sees UDP 51820 on the NIC), so the mesh *is* the
  enforcement point.
- **Alerting is Grafana-managed, provisioned as code**
  (`roles/svc-grafana/templates/alerting-*.yaml.j2`) via Grafana's embedded
  Alertmanager. If critical alerting ever needs to survive Grafana being
  down, the upgrade path is Prometheus-style rule files evaluated by
  Mimir's/Loki's ruler + a standalone Alertmanager.
- **Dashboards are provisioned as code** — JSON committed under
  `roles/svc-grafana/files/dashboards/boards/`, loaded read-only from the
  `Infrastructure` folder.
- **Why WireGuard over VPC + firewall**: VPC rules are IP-based,
  unencrypted, region-locked, and DO-only. The mesh adds encryption in
  transit and per-node key identity, and works for any future host
  anywhere. Private keys live in 1Password (`WG_PRIVATE_KEY_*` in
  `env-devict-infra`), injected per host via each service dir's
  `.env.local.op`; public keys are committed in the inventories.

## Prerequisites

Create `observability/.env.local.op` (1Password-backed, not committed) with:

```
TF_VAR_do_token=op://Private/env-devict-infra/DIGITALOCEAN_TOKEN
TF_VAR_ssh_public_key=op://Private/env-devict-infra/SSH_PUBLIC_KEY
CADDY_EMAIL=op://Private/env-devict-infra/CADDY_EMAIL
GRAFANA_ADMIN_PASSWORD=op://Private/env-devict-infra/GRAFANA_ADMIN_PASSWORD
GRAFANA_ALERT_EMAIL=op://Private/env-devict-infra/GRAFANA_ALERT_EMAIL
WG_PRIVATE_KEY=op://Private/env-devict-infra/WG_PRIVATE_KEY_OBSERVABILITY
# Optional: SMTP for alert email delivery (alerts are UI-only without it)
# GRAFANA_SMTP_HOST=...
# GRAFANA_SMTP_USER=...
# GRAFANA_SMTP_PASSWORD=...
# GRAFANA_SMTP_FROM_ADDRESS=...
```

`DIGITALOCEAN_TOKEN` / `SSH_PUBLIC_KEY` are reused from the other services;
create the rest in 1Password. `GRAFANA_HOSTNAME` defaults to
`grafana.devict.org`.

DNS (out of band): `grafana.devict.org` -> droplet IP.

## Usage

Prefix commands with `op run --env-file=.env.local.op --` to inject secrets:

```
just init     # terraform init
just plan     # terraform plan
just apply    # terraform apply
just setup    # ansible-playbook site.yaml (full setup)
just deploy   # re-render service templates + restart (no base setup)
just ssh      # ssh devict@<ip>
```

## Adding monitoring to another host

The `wireguard` and `alloy` roles live in `../shared/roles` (on every
dir's `roles_path`). On the target host:

1. Generate a keypair (`wg genkey`), store the private key in 1Password,
   add `WG_PRIVATE_KEY=op://...` to that dir's `.env.local.op`.
2. Pick the next free `10.77.0.x/24` address. Add the host as a peer in
   this dir's `inventory.yaml` (hub side) and add the hub as a peer in
   the host's inventory (spoke side, with `endpoint` +
   `persistent_keepalive`).
3. Add the `wireguard` + `alloy` roles to the host's playbook with:

```yaml
alloy_docker_network: ""
alloy_loki_url: "http://10.77.0.1:3100/loki/api/v1/push"
alloy_mimir_url: "http://10.77.0.1:9009/api/v1/push"
```

## Open questions / TODO

- Alert destination is email for now; consider Slack/Discord webhook.
- Dashboards are provisioned as code (see below); new ones: add JSON under
  `roles/svc-grafana/files/dashboards/boards/`, validate queries against
  Mimir/Loki, then `just deploy`.
- Consider pinning image tags to exact versions once validated.
- Sizing: `s-1vcpu-2gb` to start; bump `size` if Mimir gets OOM-happy.