diff --git a/.gitea/workflows/deploy.yml b/.gitea/workflows/deploy.yml index 8952c6d..aff1771 100644 --- a/.gitea/workflows/deploy.yml +++ b/.gitea/workflows/deploy.yml @@ -88,6 +88,7 @@ jobs: run: | docker build \ --platform linux/amd64 \ + --build-arg BUILD_COMMIT=${{ steps.sha.outputs.short }} \ -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ steps.sha.outputs.short }} \ -t ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest \ gateway/ diff --git a/docs/DEPLOYMENT-INVENTORY.md b/docs/DEPLOYMENT-INVENTORY.md new file mode 100644 index 0000000..b3d9fe3 --- /dev/null +++ b/docs/DEPLOYMENT-INVENTORY.md @@ -0,0 +1,111 @@ +# ATAuth Deployment Inventory + +**Last updated**: 2026-03-01 + +## Single Instance (Consolidated) + +### K8s Instance (DigitalOcean) + +| Field | Value | +|-------|-------| +| **Public URL** | `https://apricot.workingtitle.zip` | +| **Internal** | `atauth.atauth.svc.cluster.local:3100` | +| **Version** | v2.2.0 | +| **Routing** | Cloudflare Tunnel → K8s service | +| **Image** | `registry.digitalocean.com/ghostmesh-registry/atauth` | +| **Data** | K8s PersistentVolume (SQLite) at `/app/data/gateway.db` | +| **CI/CD** | Gitea Actions → DO registry → kubectl (currently broken as of 2026-02-25) | +| **Kubectl** | `KUBECONFIG=~/.kube/atauth-kubeconfig.yaml kubectl --insecure-skip-tls-verify -n atauth` | +| **Admin Token** | In K8s Secret `atauth-secrets` key `ATAUTH_ADMIN_TOKEN` | + +**All apps use this instance.** + +### Decommissioned: PDS Server Instance (Hetzner) + +Stopped 2026-03-01 after consolidation. Data preserved at Docker volume `atauth_atauth-data` on pds-hetzner. + +| Field | Value | +|-------|-------| +| **Former URL** | `https://atauth.arcnode.xyz` (tunnel route still exists, returns 502) | +| **Data backup** | `/var/lib/docker/volumes/atauth_atauth-data/_data/gateway.db` | +| **Config** | `/opt/atauth/.env`, `/opt/atauth/docker-compose.yml` | + +## Registered Apps + +### Proxy Origins (forward-auth) + +| Origin | Name | Notes | +|--------|------|-------| +| `search.arcnode.xyz` | SearXNG | nginx auth_request → K8s atauth | +| `blog.arcnode.xyz` | Arcnode Blog | Redirects to proxy login, validates ticket server-side | + +### OIDC Clients + +| ID | Name | Redirect URI | +|----|------|-------------| +| `bits-console` | BITS Backup Console | `bits-console.cloudforest-basilisk.ts.net/auth/callback` | +| `gitea` | Gitea/Forgejo | `gitea.cloudforest-basilisk.ts.net/user/oauth2/atauth/callback` | +| `audiobookshelf` | Audiobookshelf | `audiobookshelf.cloudforest-basilisk.ts.net/auth/openid/callback` | +| `mydrawings` | MyDrawings | `rock.workingtitle.zip/api/auth/atproto/callback` | + +### Legacy HMAC Apps + +| ID | Name | Callback URL | +|----|------|-------------| +| `mspdesk` | MSP-Desk | `tickets.brooksitservices.com/auth/callback` | +| `mdeditor` | Markdown Editor | `md.bkb.cx/api/auth/callback` | + +## Critical Rules + +1. **Never reroute `apricot.workingtitle.zip`** — All apps depend on this single atauth instance. Rerouting the tunnel will break everything. + +2. **PDS nginx must set Sec-Fetch headers for `/oauth/authorize`** — Cloudflare Tunnel strips browser security headers. Config at `/pds/nginx/nginx.conf` on pds-hetzner. Required headers: + ```nginx + proxy_set_header Sec-Fetch-Site "cross-site"; + proxy_set_header Sec-Fetch-Mode "navigate"; + proxy_set_header Sec-Fetch-Dest "document"; + ``` + +3. **HMAC secrets are per-app** — Each registered app has its own HMAC secret. Token verification uses `createHmac('sha256', secretString)` with UTF-8 encoding of the hex secret. + +4. **SQLite single-writer** — Do NOT scale beyond 1 replica. PVC is RWO. + +## Cloudflare Tunnel Routes + +Tunnel: `k3s-pds-tunnel` (`600f069c-4c53-4a59-b1fd-57c5ad09c044`) + +| Hostname | Destination | Notes | +|----------|-------------|-------| +| `apricot.workingtitle.zip` | K8s service `atauth:3100` | Primary — DO NOT CHANGE | + +`atauth.arcnode.xyz` route and DNS CNAME removed 2026-03-01. + +## Kubectl Access + +```bash +# Save kubeconfig (shared separately, do not commit) +export KUBECONFIG=~/.kube/atauth-kubeconfig.yaml + +# Note: TLS verify fails with DO CA cert, use --insecure-skip-tls-verify +kubectl --insecure-skip-tls-verify -n atauth get pods +kubectl --insecure-skip-tls-verify -n atauth logs deployment/atauth -f +kubectl --insecure-skip-tls-verify -n atauth rollout restart deployment/atauth + +# Get admin token +kubectl --insecure-skip-tls-verify -n atauth get secret atauth-secrets \ + -o jsonpath='{.data.ATAUTH_ADMIN_TOKEN}' | base64 -d +``` + +See `docs/DO-HANDOFF.md` for full operations guide. + +## Cleanup TODO + +- [ ] Fix Gitea CI pipeline (broken since 2026-02-25) +- [x] Remove stale `atauth.arcnode.xyz` Cloudflare tunnel route + DNS (done 2026-03-01) +- [ ] Remove `/opt/atauth/` directory and Docker volumes from pds-hetzner +- [ ] Add `/version` endpoint to atauth for quick identification +- [ ] Add uptime monitoring +- [x] Consolidate to single K8s atauth instance (done 2026-03-01) +- [x] Decommission PDS atauth sidecar (done 2026-03-01) +- [x] Fix PDS nginx Sec-Fetch headers (done 2026-03-01) +- [x] Remove stale app registrations from PDS instance (done 2026-03-01) diff --git a/docs/RCA-2026-03-01-dual-atauth-instances.md b/docs/RCA-2026-03-01-dual-atauth-instances.md new file mode 100644 index 0000000..fa70d2b --- /dev/null +++ b/docs/RCA-2026-03-01-dual-atauth-instances.md @@ -0,0 +1,121 @@ +# Root Cause Analysis: Dual ATAuth Instances & PDS Server Version Drift (2026-03-01) + +## Incident + +Blog authentication flow (`blog.arcnode.xyz`) failed with "OAuth state not found or expired" after PDS OAuth completed successfully. Investigation revealed a second undocumented atauth instance on the PDS Hetzner server running v1.0.0 (3+ months behind production v2.2.0). Attempted fix temporarily broke authentication for all apps using the production K8s atauth. + +**Duration**: ~3 hours active debugging +**Impact**: +- Blog login broken (original issue) +- ~10 minute outage for all apps using K8s atauth (mspdesk, search, etc.) due to incorrect tunnel reroute +- PDS server running atauth v1.0.0 without passkey, proxy auth, OIDC, or admin dashboard features +**Severity**: High (P2) — partial auth outage, undocumented infrastructure drift + +## Timeline + +1. Blog login flow completed PDS OAuth but atauth returned "OAuth state not found" +2. Traced root cause: blog server-side called local atauth (`172.17.0.1:3100` on PDS server) for `/auth/init`, creating state locally. But PDS redirected browser to `apricot.workingtitle.zip/auth/callback` (K8s atauth) which had no knowledge of that state. +3. **Mistake**: Rerouted `apricot.workingtitle.zip` tunnel from K8s atauth to PDS server atauth — immediately broke all apps (mspdesk, search, etc.) that use K8s atauth with different HMAC secrets and app registrations. +4. Reverted tunnel within ~10 minutes. +5. **Fix**: Created `atauth.arcnode.xyz` as a separate public URL for the PDS server's atauth instance, keeping `apricot.workingtitle.zip` pointed at K8s. +6. Discovered PDS server was running atauth v1.0.0 (Dec 2025 image) while K8s was running v2.2.0 (Feb 2026). Built v2.2.0 from source and deployed. + +## Root Causes + +### 1. Undocumented second atauth instance + +The PDS Hetzner server (`46.224.33.19`) had atauth deployed at `/opt/atauth/` using Docker image `ghcr.io/cache8063/atauth-gateway:latest` (v1.0.0, built Dec 11 2025). This instance was: +- Not documented in any infrastructure docs +- Not tracked in the Cloudflare tunnel config +- Running a version 3+ months behind production +- Had its own separate SQLite database with its own app registrations and HMAC secrets +- Accessible only via Docker bridge IP (`172.17.0.1:3100`) — no public URL + +### 2. Mixed internal/external URL configuration + +The atauth instance on the PDS server had: +- `OAUTH_CLIENT_ID=https://apricot.workingtitle.zip/client-metadata.json` +- `OAUTH_REDIRECT_URI=https://apricot.workingtitle.zip/auth/callback` + +This meant it told the PDS "I am `apricot.workingtitle.zip`" but it was NOT `apricot.workingtitle.zip`. The PDS would redirect the browser to the real `apricot.workingtitle.zip` (K8s atauth) which had no knowledge of the state created by the PDS server's atauth. + +### 3. Stale container image — no CI/CD pipeline to PDS server + +The Gitea CI workflow builds images and pushes to `gitea.cloudforest-basilisk.ts.net/arcnode.xyz/atauth-gateway`. But: +- The PDS server was pulling from `ghcr.io/cache8063/atauth-gateway:latest` (different registry) +- No Watchtower or auto-update mechanism for atauth on the PDS server +- The Gitea CI has been failing since Feb 25 (runner issues), so even the Gitea registry image was stale (Dec 2025) +- Had to build v2.2.0 from source directly on the server + +### 4. Shared tunnel namespace without isolation + +Both atauth instances shared the same `apricot.workingtitle.zip` identity. Changing the tunnel route for one immediately broke the other because: +- Different app registrations (different HMAC secrets) +- Different session stores (SQLite on each) +- Different feature sets (v1.0.0 vs v2.2.0) + +## What Data Was At Risk + +Checked and confirmed all data survived: +- All 5 registered apps intact (mspdesk, apricot, md, mdeditor, arcnode-blog) +- All HMAC secrets preserved (64 bytes each) +- All sessions preserved (4 active) +- K8s atauth completely untouched +- Docker volume `atauth_atauth-data` used by both old and new containers (same project prefix) + +## Fix Applied + +1. Created `atauth.arcnode.xyz` DNS + tunnel route → PDS server atauth (`46.224.33.19:3100`) +2. Updated PDS server atauth env: + - `OAUTH_CLIENT_ID=https://atauth.arcnode.xyz/client-metadata.json` + - `OAUTH_REDIRECT_URI=https://atauth.arcnode.xyz/auth/callback` +3. Built and deployed atauth v2.2.0 from source on PDS server +4. Enabled passkey (RP ID: `atauth.arcnode.xyz`), forward-auth proxy +5. Updated blog to use atauth proxy login flow (no more client-side callback delay) +6. K8s atauth (`apricot.workingtitle.zip`) left completely untouched +7. Fixed PDS nginx `Sec-Fetch-Mode` header — PDS started enforcing this header on `/oauth/authorize`, and Cloudflare Tunnel strips it. Added `Sec-Fetch-Mode: navigate` and `Sec-Fetch-Dest: document` to the nginx proxy config alongside the existing `Sec-Fetch-Site: cross-site`. This was breaking both search.arcnode.xyz (K8s atauth) and blog.arcnode.xyz (PDS atauth) since both OAuth flows go through the same PDS. +8. Cleaned up stale PDS atauth app registrations — removed duplicate `mdeditor` (same product as `md`), stale `mspdesk` and `apricot` entries that only belong on K8s instance + +## Current State (Post-Fix) + +| Instance | URL | Version | Apps | +|----------|-----|---------|------| +| K8s (DO) | `apricot.workingtitle.zip` | v2.2.0 | search, mspdesk, others | +| PDS (Hetzner) | `atauth.arcnode.xyz` | v2.2.0 | arcnode-blog, md | + +**Note**: Consolidation target is the K8s instance (`apricot.workingtitle.zip`). Once kubeconfig access is available, migrate PDS instance apps to K8s and decommission the PDS sidecar. + +## Prevention + +### Immediate + +1. **Document all atauth instances** — Update infrastructure overview with both instances, their URLs, versions, and which apps use which +2. **Each instance MUST have its own unique public URL** — Never configure `OAUTH_CLIENT_ID`/`OAUTH_REDIRECT_URI` pointing to a different instance +3. **Add version endpoint** — `GET /version` returning build commit and version to quickly identify running versions + +### Short-term + +4. **Fix Gitea CI pipeline** — Runner has been broken since Feb 25. Fix it so images auto-build on push +5. **Add Watchtower or auto-pull** for atauth on PDS server to track Gitea registry +6. **Consolidate to single atauth instance** — Both instances have diverged data. Pick one as source of truth and migrate (see cleanup task below) + +### Long-term + +7. **Migrate atauth to dedicated instance** — Currently running as a sidecar on the PDS server. Should be its own container/VM with proper CI/CD, monitoring, and backup +8. **Infrastructure-as-code** — Tunnel routes, DNS, and service configs should be in version control, not manually configured +9. **Add health monitoring** — Alert if any atauth instance returns non-200 or version mismatch + +## Cleanup Task + +Migrate atauth to a dedicated instance off the PDS server. This was identified as necessary during this incident. Track in project TODO. + +## Files Changed + +- `/opt/atauth/.env` (PDS server) — Updated OAUTH_CLIENT_ID/REDIRECT_URI to atauth.arcnode.xyz +- `/opt/atauth/docker-compose.yml` (PDS server) — Switched from ghcr.io to locally built v2.2.0 image +- `/pds/nginx/nginx.conf` (PDS server) — Added `Sec-Fetch-Mode` and `Sec-Fetch-Dest` headers to `/oauth/authorize` location +- Cloudflare tunnel — Added `atauth.arcnode.xyz` ingress rule +- `arcnode-blog/src/lib/auth.ts` — Rewrote to use proxy auth flow +- `arcnode-blog/src/pages/write.astro` — Redirect to atauth proxy login instead of inline form +- `arcnode-blog/src/pages/api/publish.ts` — Use session cookies instead of HMAC token +- Deleted: `arcnode-blog/src/pages/api/login.ts`, `api/session.ts`, `auth/callback.astro` diff --git a/gateway/Dockerfile b/gateway/Dockerfile index 4cd712a..3a43301 100644 --- a/gateway/Dockerfile +++ b/gateway/Dockerfile @@ -46,6 +46,10 @@ RUN mkdir -p /app/data && chown atauth:atauth /app/data # Switch to non-root user USER atauth +# Build metadata +ARG BUILD_COMMIT=unknown +ENV BUILD_COMMIT=${BUILD_COMMIT} + # Environment defaults ENV NODE_ENV=production \ PORT=3100 \ diff --git a/gateway/src/index.ts b/gateway/src/index.ts index a872137..a14f6aa 100644 --- a/gateway/src/index.ts +++ b/gateway/src/index.ts @@ -250,6 +250,17 @@ async function main(): Promise { }); }); + // Version endpoint + const pkgPath = path.join(process.cwd(), 'package.json'); + const pkgVersion = JSON.parse(fs.readFileSync(pkgPath, 'utf-8')).version; + app.get('/version', (_req, res) => { + res.json({ + service: 'atauth-gateway', + version: pkgVersion, + commit: process.env.BUILD_COMMIT || 'unknown', + }); + }); + // Forward-auth proxy routes MUST be mounted before the general /auth rate limiter. // /auth/verify is called by nginx auth_request on every subrequest from a single // pod IP, so per-IP rate limiting would block legitimate traffic.