diff --git a/.env.example b/.env.example index df83900..85b5bfa 100644 --- a/.env.example +++ b/.env.example @@ -138,6 +138,15 @@ TAP_ADMIN_PASSWORD="tap_dev_secret" # Log level: trace, debug, info, warn, error, fatal LOG_LEVEL="info" +# ============================================================================== +# Monitoring Proxy -- requires docker-compose.monitoring-proxy.yml +# ============================================================================== +# The monitoring stack (Prometheus + Grafana) runs as a separate compose stack +# from singi-labs/monitoring. This variable tells Caddy which domain to proxy. + +# Domain for the Grafana dashboard (needs a DNS A record pointing to this VPS) +# GRAFANA_DOMAIN="monitor.barazo.forum" + # ============================================================================== # Backups (Production only) # ============================================================================== diff --git a/.github/workflows/deploy-staging.yml b/.github/workflows/deploy-staging.yml index 0278837..af1162f 100644 --- a/.github/workflows/deploy-staging.yml +++ b/.github/workflows/deploy-staging.yml @@ -23,7 +23,7 @@ on: env: REGISTRY: ghcr.io DEPLOY_PATH: /opt/barazo - COMPOSE_FILES: "-f docker-compose.yml -f docker-compose.staging.yml" + COMPOSE_FILES: "-f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.monitoring-proxy.yml" concurrency: group: staging-deploy diff --git a/Caddyfile b/Caddyfile index f223546..7660119 100644 --- a/Caddyfile +++ b/Caddyfile @@ -62,3 +62,12 @@ docs.barazo.forum { reverse_proxy barazo-web:3001 } } + +# Optional: Grafana monitoring dashboard proxy. +# Only active when running a separate monitoring stack (e.g., singi-labs/monitoring) +# with docker-compose.monitoring-proxy.yml. Without GRAFANA_DOMAIN set, this block +# defaults to "monitor.localhost" and has no effect. +{$GRAFANA_DOMAIN:monitor.localhost} { + header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" + reverse_proxy grafana:3050 +} diff --git a/docker-compose.monitoring-proxy.yml b/docker-compose.monitoring-proxy.yml new file mode 100644 index 0000000..076e484 --- /dev/null +++ b/docker-compose.monitoring-proxy.yml @@ -0,0 +1,30 @@ +# Barazo Monitoring Proxy -- Docker Compose Overlay (Optional) +# +# This overlay is NOT required for running Barazo. It is only needed if you +# run a separate Prometheus/Grafana monitoring stack on the same VPS and want +# Caddy to reverse-proxy to it. Self-hosters can safely ignore this file. +# +# Connects Caddy to the monitoring stack's Docker network so it can +# reverse-proxy to Grafana. The monitoring stack (singi-labs/monitoring) +# runs as a separate compose stack on the same VPS. +# +# Usage: +# docker compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.monitoring-proxy.yml up -d +# +# Requires: +# - The monitoring stack running (docker compose up -d in /opt/monitoring) +# - GRAFANA_DOMAIN set in .env +# - DNS A record for GRAFANA_DOMAIN pointing to this VPS +# - Grafana subdomain block in Caddyfile + +services: + caddy: + environment: + GRAFANA_DOMAIN: "${GRAFANA_DOMAIN}" + networks: + - monitoring-proxy + +networks: + monitoring-proxy: + external: true + name: monitoring-proxy diff --git a/docs/administration.md b/docs/administration.md index 4610448..1a9e6b4 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -43,6 +43,29 @@ Run the smoke test to verify everything is working: docker stats --no-stream ``` +### Grafana Dashboard + +The monitoring stack (Prometheus + Grafana) provides real-time dashboards and alerting for all Singi Labs VPSes. Access it at `https://monitor.barazo.forum`. + +Available dashboards: +- **Singi Labs -- VPS Overview**: All VPSes at a glance (CPU, RAM, disk, network, container metrics) +- **Node Exporter Full**: Deep per-VPS host metrics +- **Docker Container & Host**: Per-container resource usage + +The monitoring stack runs from a separate repo: [singi-labs/monitoring](https://github.com/singi-labs/monitoring). See its README for setup, maintenance, and adding new VPSes. + +### Prometheus Targets + +Check that all scrape targets are healthy: + +```bash +# From the VPS +curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep -E '"health"|"job"' + +# Reload config after editing prometheus.yml +curl -X POST http://localhost:9090/-/reload +``` + ## Backups ### Automated Backups (Recommended) diff --git a/docs/security-hardening.md b/docs/security-hardening.md index 3513637..de04788 100644 --- a/docs/security-hardening.md +++ b/docs/security-hardening.md @@ -156,6 +156,12 @@ services: Never mount the Docker socket (`/var/run/docker.sock`) into any container. None of the Barazo services require it. +**Exception: cAdvisor** -- The monitoring stack includes cAdvisor, which requires read-only access to `/var/run` (containing the Docker socket) and `/sys` to collect per-container metrics. This is a deliberate, documented exception: +- cAdvisor is a widely-used, Google-maintained monitoring tool +- The mount is read-only (`:ro`) -- cAdvisor does not write to the socket +- cAdvisor runs with a read-only root filesystem +- No other monitoring container has socket access + ### Image Updates Keep base images updated. Dependabot is configured in the repo for automated image update PRs. On the server: @@ -282,7 +288,7 @@ Backups must be encrypted before off-server storage. See [Backup & Restore](back Use this as a post-deployment verification: - [ ] SSH: root login disabled, password auth disabled -- [ ] Firewall: only 22, 80, 443 (TCP), 443 (UDP) open +- [ ] Firewall: only 22, 80, 443 (TCP), 443 (UDP), and 51820 (UDP, WireGuard) open - [ ] Unattended upgrades enabled - [ ] Resource limits set in `docker-compose.yml` - [ ] No `CHANGE_ME` in `.env`: `grep CHANGE_ME .env` returns nothing diff --git a/infrastructure/staging.md b/infrastructure/staging.md index a81ebae..81e5c93 100644 --- a/infrastructure/staging.md +++ b/infrastructure/staging.md @@ -111,12 +111,35 @@ Use the `/staging-status` Claude Code skill for quick SSH-based triage: This checks container health, recent logs, disk/memory usage, missing env vars, and image versions. +## Monitoring + +This VPS is the monitoring hub for all Singi Labs VPSes. It runs Prometheus, Grafana, node_exporter, and cAdvisor locally, and scrapes remote VPSes over WireGuard. + +| Component | Purpose | +|-----------|---------| +| Prometheus | Scrapes and stores metrics (30-day retention) | +| Grafana | Dashboards and alerting at `monitor.barazo.forum` | +| node_exporter | Host CPU, RAM, disk, network metrics | +| cAdvisor | Per-container resource metrics | +| WireGuard | VPN tunnel to remote VPSes (10.10.0.0/24) | +| socat | Port forwards from localhost to WireGuard IPs for Prometheus | + +The monitoring stack runs from a separate repo: [singi-labs/monitoring](https://github.com/singi-labs/monitoring). See its README for full setup instructions, WireGuard configuration, and how to add new VPSes. + +### WireGuard Peers + +| VPS | WireGuard IP | Ports forwarded via socat | +|-----|-------------|--------------------------| +| Barazo staging (this VPS) | 10.10.0.1 | n/a (local Docker network) | +| Sifa production | 10.10.0.2 | 9101->9100, 9102->8080 | +| Barazo production (future) | 10.10.0.3 | 9103->9100, 9104->8080 | + ## Compose Files Staging uses the overlay pattern: ```bash -docker compose -f docker-compose.yml -f docker-compose.staging.yml up -d +docker compose -f docker-compose.yml -f docker-compose.staging.yml -f docker-compose.monitoring-proxy.yml up -d ``` The staging override (`docker-compose.staging.yml`) sets: @@ -125,6 +148,8 @@ The staging override (`docker-compose.staging.yml`) sets: - `LOG_LEVEL: debug` - Relaxed rate limits for testing +The monitoring proxy overlay (`docker-compose.monitoring-proxy.yml`) connects Caddy to the monitoring stack's Docker network so it can reverse-proxy Grafana. The monitoring stack itself (Prometheus, Grafana, node_exporter, cAdvisor) runs from a separate repo at `/opt/monitoring`. + ## Image Tags Each deploy produces two tags per image: