diff --git a/incident-2026-08-20.md b/incident-2026-08-20.md new file mode 100644 index 0000000..6e50ba8 --- /dev/null +++ b/incident-2026-08-20.md @@ -0,0 +1,183 @@ +# Incident Report: git.toyvo.dev and auth.diekvoss.net Outages + +**Date:** 2026-08-20 +**Status:** Active - Root cause investigation + +## Summary +Two services experiencing issues: +1. **git.toyvo.dev** - SSL/TLS errors when accessing via HTTPS +2. **auth.diekvoss.net** - 502 Bad Gateway (authentik not responding) + +### Changes Made by This Agent (git commits) + +**Branch:** main +**HEAD before changes:** 9d3c51cc ("fix(authentik) migration") + +#### Commits Made: + +1. **1e18d34b** - "fix(authentik): add network wait to migration service" + - Added authentik-migrate service + - Added blocking network wait in preStart (BAD: caused circular dependency) + +2. **aa34e8ed** - "fix(authentik): add network wait and proper dependencies to migration service" + - Added `wants = [ "network-online.target" ]` to fix evaluation warning + - Still had blocking network wait + +3. **03259cde** - "fix(authentik): use full path for iproute2 in migration preStart" + - Fixed `ip addr show` to use `${pkgs.iproute2}/bin/ip` + - Still blocking + +4. **244419f4** - "fix(authentik): remove blocking network wait from migration" + - Removed the blocking network wait scripts + - Migration still exists but doesn't block + +5. **51733bcd** - "fix(nas): add pg_hba rule for authentik from 10.1.0.0/16" + - Added PostgreSQL auth rule for 10.1.0.0/16 network + - Because container traffic appears to come from host IP (10.1.0.3) not container IP (10.200.0.16) + +#### Files Modified: +- `modules/nixos/containers/authentik.nix` - Multiple changes to add/remove migration service +- `configurations/nixos/nas/configuration.nix` - Added PostgreSQL auth rule + +#### Current State of HEAD: +- Commit: 51733bcd +- authentik-migrate service exists but doesn't have blocking wait +- authentik-core service was REMOVED (may have been wrong - need to verify) +- PostgreSQL auth rules updated +- SSLMODE set to "prefer" for all authentik services + +#### What Needs Revert/Investigation: +1. Did removing `authentik-core` service break something that was working? +2. Was `ak core` command actually needed or not? +3. The original error was about `/dev/shm/authentik-core.sock` - was this a config issue or missing service? + +--- + +## Issue 1: git.toyvo.dev HTTPS Errors + +### Symptoms +- SSL_ERROR_INTERNAL_ERROR_ALERT in Firefox +- "This site can't provide a secure connection" in Chrome +- Was working 2 days ago (2026-08-18) +- Direct HTTP to backend (10.1.0.3:3000) works correctly + +### What We've Verified +- ✅ Certificate is valid (Let's Encrypt, expires Oct 7, 2026) +- ✅ Certificate chain is complete (4 certs including intermediates) +- ✅ Caddy config is correct (reverse_proxy to http://10.1.0.3:3000) +- ✅ forgejo service running on 10.1.0.3:3000 +- ❌ Browser access via HTTPS fails + +### What Changed Recently +- Need to check: git log for changes 2026-08-18 to now +- Need to check: Caddy version or config changes +- Need to check: Certificate renewal around that time + +### Hypotheses +1. Caddy or TLS library update changed behavior +2. Certificate issue (new intermediate, CT, or revocation) +3. Browser security policy change +4. Network/DNS issue (less likely since HTTP works) + +### Next Steps +1. Check Caddy access logs for error details +2. Check what changed in git around Aug 18 +3. Test with curl verbose to see exact SSL failure point +4. Check if issue occurs without Cloudflare (direct IP access) + +## Issue 2: auth.diekvoss.net 502 + +### Symptoms +- 502 Bad Gateway from Caddy +- Authentik container not responding +- Started with socket error, now database connection issues + +### Timeline +1. **Initial issue:** authentik-core.sock not found + - Tried to fix by removing authentik-core service (incorrect) + - Added migration service (caused new issues) + +2. **Current issue:** Database connection failures + - Container can't reach PostgreSQL at 10.200.0.15 + - Network interface issues (veth not properly configured) + - SSL/TLS mode issues between authentik and PostgreSQL + +### What We've Done (Possibly Made Worse) +1. Removed working authentik-core service +2. Added migration service that blocks container startup +3. Added network wait scripts that cause circular dependencies +4. Manually configured network interfaces (not persistent) + +### Current State +- Container keeps restarting +- Migration service never completes +- Database connection failing with "server closed connection unexpectedly" +- PostgreSQL pg_hba.conf rules appear correct + +### Root Cause +Unclear. Multiple issues compounded: +1. Container networking not properly configured by systemd-nspawn +2. Migration service design is fundamentally flawed (circular dependency) +3. Changes made without proper testing + +### Next Steps +1. REVERT all authentik changes to last known working state +2. Identify what the actual working configuration was +3. Fix the original socket issue properly +4. Test each change before deploying + +## Actions Required + +### Immediate +1. Revert authentik module to working state +2. Investigate git.toyvo.dev SSL issue separately +3. Test each service independently + +### Investigation Needed +1. What commit was deployed 2 days ago? +2. What changed in authentik configuration? +3. Are there any system updates that coincided? +4. Check Caddy and authentik versions + +## Lessons Learned +- Don't make multiple changes without testing +- Don't block container startup with network-dependent services +- Document working state before making changes +- Test in isolation, not production + +--- + +## Version Control Info (jj) + +**Repository:** /Users/CollinDie/nixcfg +**Current HEAD:** 51733bcddec19b3c6f12c6d1a0e9f5c4e0c8a0f2 +**Branch:** main + +### Relevant Commits (chronological): + +``` +51733bcd - fix(nas): add pg_hba rule for authentik from 10.1.0.0/16 +244419f4 - fix(authentik): remove blocking network wait from migration +03259cde - fix(authentik): use full path for iproute2 in migration preStart +aa34e8ed - fix(authentik): add network wait and proper dependencies to migration service +1e18d34b - fix(authentik): add network wait to migration service +9d3c51cc - fix(authentik) migration (BEFORE THIS AGENT'S CHANGES) +``` + +### Files Modified by This Session: +1. `modules/nixos/containers/authentik.nix` - Added migration service, removed authentik-core +2. `configurations/nixos/nas/configuration.nix` - Added PostgreSQL auth rule +3. `incident-2026-08-20.md` - This document + +### To Revert Changes: +```bash +# Revert to before this agent's changes +jj bookmark set main -r 9d3c51cc +jj git push +# Then redeploy +sudo nixos-rebuild switch --flake .#nas +``` + +--- +**Document Created:** 2026-08-20 +**Last Updated:** 2026-08-20 diff --git a/modules/nixos/containers/authentik.nix b/modules/nixos/containers/authentik.nix index ad99ba6..ef6fcba 100644 --- a/modules/nixos/containers/authentik.nix +++ b/modules/nixos/containers/authentik.nix @@ -149,6 +149,12 @@ in }; config = lib.mkIf cfg.enable { + # authentik-migrate retries the DB connection internally forever and never + # exits, so it stays "activating" until the DB is reachable. The default + # 1min container start timeout was killing the container mid-attempt + # every time, tearing down ve-authentik and restarting the whole loop. + systemd.services."container@authentik".serviceConfig.TimeoutStartSec = "10min"; + networking.nat = lib.mkIf (cfg.natInterface != null) { enable = true; externalInterface = cfg.natInterface; @@ -239,6 +245,7 @@ in AUTHENTIK_POSTGRESQL__PORT = toString cfg.db.port; AUTHENTIK_POSTGRESQL__NAME = cfg.db.name; AUTHENTIK_POSTGRESQL__USER = cfg.db.user; + AUTHENTIK_POSTGRESQL__SSLMODE = "prefer"; AUTHENTIK_REDIS__HOST = cfg.redis.host; AUTHENTIK_REDIS__PORT = toString cfg.redis.port; }; @@ -278,6 +285,7 @@ in AUTHENTIK_POSTGRESQL__PORT = toString cfg.db.port; AUTHENTIK_POSTGRESQL__NAME = cfg.db.name; AUTHENTIK_POSTGRESQL__USER = cfg.db.user; + AUTHENTIK_POSTGRESQL__SSLMODE = "prefer"; AUTHENTIK_REDIS__HOST = cfg.redis.host; AUTHENTIK_REDIS__PORT = toString cfg.redis.port; }; @@ -397,6 +405,7 @@ in AUTHENTIK_POSTGRESQL__PORT = toString cfg.db.port; AUTHENTIK_POSTGRESQL__NAME = cfg.db.name; AUTHENTIK_POSTGRESQL__USER = cfg.db.user; + AUTHENTIK_POSTGRESQL__SSLMODE = "prefer"; AUTHENTIK_REDIS__HOST = cfg.redis.host; AUTHENTIK_REDIS__PORT = toString cfg.redis.port; }; @@ -429,6 +438,7 @@ in AUTHENTIK_POSTGRESQL__PORT = toString cfg.db.port; AUTHENTIK_POSTGRESQL__NAME = cfg.db.name; AUTHENTIK_POSTGRESQL__USER = cfg.db.user; + AUTHENTIK_POSTGRESQL__SSLMODE = "prefer"; AUTHENTIK_REDIS__HOST = cfg.redis.host; AUTHENTIK_REDIS__PORT = toString cfg.redis.port; };