devict.org / infra
This repository has no description
Star
Fork 0
Watch
infra observability
2 folders · 4 files
ansible
terraform
justfile
PLAN.md
README.md
TASK.md
README.md

Observability #

Central observability node for devICT infrastructure: the Grafana LGTM stack (Loki, Mimir, Tempo + Grafana), with Grafana Alloy as the collector on every monitored host.

  • Terraform provisions the droplet (devict-observability) + cloud firewall.
  • Ansible installs Docker + renders one compose stack per role under /opt/devict/<role>/, each wrapped in a systemd unit.
  • Caddy terminates TLS for grafana.devict.org -> grafana:3000.

Architecture #

  • Alloy runs on every host (this one included) and pushes: host metrics (embedded node_exporter) + its own metrics -> Mimir, systemd journal + docker container logs -> Loki. Push model: monitored hosts need no inbound ports, only outbound to this node.
  • Apps send OTLP traces directly to Tempo (:4317). Alloy is not in the trace path.
  • Backends run unauthenticated (auth_enabled: false). All telemetry crosses a WireGuard mesh (hub-and-spoke, this node is the hub, 10.77.0.0/24; shared wireguard role in ../shared/roles). The ingest ports are not opened in the cloud firewall at all — the tunnel is the only path in, and every peer is authenticated by key. Note: docker-published ports bypass ufw and wg0 traffic bypasses the cloud firewall (it only sees UDP 51820 on the NIC), so the mesh is the enforcement point.
  • Alerting is Grafana-managed, provisioned as code (roles/svc-grafana/templates/alerting-*.yaml.j2) via Grafana's embedded Alertmanager. If critical alerting ever needs to survive Grafana being down, the upgrade path is Prometheus-style rule files evaluated by Mimir's/Loki's ruler + a standalone Alertmanager.
  • Dashboards are provisioned as code — JSON committed under roles/svc-grafana/files/dashboards/boards/, loaded read-only from the Infrastructure folder.
  • Why WireGuard over VPC + firewall: VPC rules are IP-based, unencrypted, region-locked, and DO-only. The mesh adds encryption in transit and per-node key identity, and works for any future host anywhere. Private keys live in 1Password (WG_PRIVATE_KEY_* in env-devict-infra), injected per host via each service dir's .env.local.op; public keys are committed in the inventories.

Prerequisites #

Create observability/.env.local.op (1Password-backed, not committed) with:

TF_VAR_do_token=op://Private/env-devict-infra/DIGITALOCEAN_TOKEN
TF_VAR_ssh_public_key=op://Private/env-devict-infra/SSH_PUBLIC_KEY
CADDY_EMAIL=op://Private/env-devict-infra/CADDY_EMAIL
GRAFANA_ADMIN_PASSWORD=op://Private/env-devict-infra/GRAFANA_ADMIN_PASSWORD
GRAFANA_ALERT_EMAIL=op://Private/env-devict-infra/GRAFANA_ALERT_EMAIL
WG_PRIVATE_KEY=op://Private/env-devict-infra/WG_PRIVATE_KEY_OBSERVABILITY
# Optional: SMTP for alert email delivery (alerts are UI-only without it)
# GRAFANA_SMTP_HOST=...
# GRAFANA_SMTP_USER=...
# GRAFANA_SMTP_PASSWORD=...
# GRAFANA_SMTP_FROM_ADDRESS=...

DIGITALOCEAN_TOKEN / SSH_PUBLIC_KEY are reused from the other services; create the rest in 1Password. GRAFANA_HOSTNAME defaults to grafana.devict.org.

DNS (out of band): grafana.devict.org -> droplet IP.

Usage #

Prefix commands with op run --env-file=.env.local.op -- to inject secrets:

just init     # terraform init
just plan     # terraform plan
just apply    # terraform apply
just setup    # ansible-playbook site.yaml (full setup)
just deploy   # re-render service templates + restart (no base setup)
just ssh      # ssh devict@<ip>

Adding monitoring to another host #

The wireguard and alloy roles live in ../shared/roles (on every dir's roles_path). On the target host:

  1. Generate a keypair (wg genkey), store the private key in 1Password, add WG_PRIVATE_KEY=op://... to that dir's .env.local.op.
  2. Pick the next free 10.77.0.x/24 address. Add the host as a peer in this dir's inventory.yaml (hub side) and add the hub as a peer in the host's inventory (spoke side, with endpoint + persistent_keepalive).
  3. Add the wireguard + alloy roles to the host's playbook with:
alloy_docker_network: ""
alloy_loki_url: "http://10.77.0.1:3100/loki/api/v1/push"
alloy_mimir_url: "http://10.77.0.1:9009/api/v1/push"

Open questions / TODO #

  • Alert destination is email for now; consider Slack/Discord webhook.
  • Dashboards are provisioned as code (see below); new ones: add JSON under roles/svc-grafana/files/dashboards/boards/, validate queries against Mimir/Loki, then just deploy.
  • Consider pinning image tags to exact versions once validated.
  • Sizing: s-1vcpu-2gb to start; bump size if Mimir gets OOM-happy.