From d032d40be24154b5d2f3673d0722ec4831843d28 Mon Sep 17 00:00:00 2001 From: iacore Date: Mon, 03 Aug 2026 12:25:31 +0000 Subject: [PATCH] docs: log S023 - invite error passthrough + token refresh race (roadmap slice line + follow-up) --- roadmap.dj | 9 +++++++++ log/s023.dj | 84 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 file(s) changed, 93 insertion(s)(+), 0 deletion(s)(-) diff --git a/roadmap.dj b/roadmap.dj --- a/roadmap.dj +++ b/roadmap.dj @@ -118,6 +118,12 @@ the sign-up credential, invite codes gate spaces only); bootstrap swaps home-PDS tokens for full-scope zds creds via getServiceAuth + zds service-auth createAccount + zds OAuth. [log](log/s020.dj) +- **S023** ✓ — invite errors surfaced verbatim: the BFF forwards + zds's structured `{error, reason}` on invite preview/join instead of + collapsing every upstream failure into 403 "denied" (transport + failures get a distinct 502); oauth.refresh serialized against the + concurrent-401 replay race that split token families and locked the + app out of zds. [log](log/s023.dj) ## Open questions @@ -140,6 +146,9 @@ - Invite revocation UI (deleting the invite record — CLI/API only). (s018) - Bootstrap invite code is a shared 500-use fly secret; revisit if sign-ups grow. (s020) +- zds-linji runs an older binary than HEAD (2h access tokens, no + replay-revocation path); redeploy to current source deliberately. + (s023) Resolved: chat publicity (member-only spaces in phase 001, public records in phase 003); invitee PDS (hosted zds — load-bearing for spaces); stack (Zig + diff --git a/log/s023.dj b/log/s023.dj new file mode 100644 --- /dev/null +++ b/log/s023.dj @@ -0,0 +1,84 @@ +# S023 - the "denied" invite incident: divergent token families, missing JWT secret, masked errors + +Date: 2026-08-02 +Slice: ops incident + hardening. A freshly minted tongxi invite link +reported "denied" (403) when opened; the link itself was never invalid. + +## What happened + +The BFF's invite preview/join is a zds passthrough; any upstream failure +collapsed into a generic 403 "denied". For the user's account +(`1a-insec.net`, `did:plc:73xkqje76lj5tostbsqa74b7`) the upstream was +broken: the stored access token expired at 12:18Z, and the stored refresh +token was unknown to zds (`invalid_grant: Invalid refresh token`), so the +401 -> refresh -> retry loop died on the first hop. + +DB archaeology (zds `oauth_tokens`) showed why: two token families were +granted in the same second (10:18:47Z, jti counters -0/-1 and -2/-3). +Family A landed in zds; family B landed in the BFF's `accounts.json` and +was never persisted by zds. The app held a pair zds could not honor. + +Root cause is a client-side race, not a zds one: the BFF is threaded +(`std.Io.Threaded`, 8 concurrent) and two concurrent requests hitting 401 +for the same account both call `oauth.refresh()` with the same refresh +token. Responses race on the shared `Account` struct; the app can end up +holding a token pair zds never persisted. (Current zds additionally +revokes the family on replay, so the same race can lock an account out.) + +Two aggravating findings: + +1. **`ZDS_JWT_SECRET` was unset** on zds-linji. zds signs session tokens + with `config.jwtSecret()`, which defaults to a hardcoded public string + (`zds-local-development-jwt-secret-change-me`). Anyone reading the + source could forge zds access tokens for any DID. `ZDS_DPOP_SECRET` + fell back to the same value. +2. **The BFF masked every upstream error as 403 "denied"** - the browser + could not distinguish a bad invite from a broken token leg, and the + server-side debug print was the only trace. + +## Fix + +- **zds**: set `ZDS_JWT_SECRET` and `ZDS_DPOP_SECRET` (random 32-byte + values, fly secrets). Old access tokens fail signature on the next use, + but the 401 -> refresh leg self-heals them (refresh lookup is by DB + string, not signature); DPoP nonce staleness is handled by zat's + nonce-challenge retry. +- **app0 (`src/core/oauth.zig`)**: serialize `refresh()` behind a mutex, + locked before reading the account's refresh token. Concurrent 401s for + the same account now refresh sequentially, each using the previous + rotation's token - one consistent pair on both sides. +- **app0 (`src/core/pds.zig`, `src/core/spaces.zig`, `src/server.zig`)**: + invite preview/join use a raw passthrough that forwards zds's status + + body verbatim (200 `{valid:false, reason}` previews; 403 + `{"error":"InviteInvalid","reason":...}` joins). Only transport + failures (broken tokens, network) return a distinct 502 instead of a + misleading "denied". +- **web (`src/api.ts`, `src/App.tsx`)**: API errors carry the zds + `reason`; the join screen shows the human text (expired / not-found / + wrong-authority / wrong-space) instead of the raw body. +- **Data repair**: aligned zds's `oauth_tokens` row for the affected DID + with the BFF's stored pair (old refresh kept in grace) so the stored + refresh token became findable again. + +## Verification + +- `zig build test` PASS; `zig build scenario-serve` PASS (invite section + incl. bogus-join 403 and idempotent rejoin); web `bun run test` 53/53. +- After deploys + secret change: all three accounts' `/api/spaces` 200 + (self-heal through 401 -> refresh -> retry, stale DPoP nonce included). +- As `1a-insec.net` on prod: login redirect 302 -> bsky.social with valid + PAR; invite preview 200 `valid:true`; join 200 `alreadyMember`; + thread + post created, read back visible; bogus invite now yields 403 + `{"error":"InviteInvalid","reason":"not-found"}` (was "denied"). + +## Notes / follow-ups + +- The full interactive consent leg (bsky.social password entry) is the + user's browser step; every server-side leg of sign-in is verified. +- The exact interleaving that lost family B's DB write predates the + current zds source (deployed binary is older: 2h access tokens vs the + source's 15 min, no replay-revocation path) and was not reproduced; + the app0-side serialization removes the trigger regardless. +- zds-linji is running an older binary than HEAD; a redeploy to current + source brings 15-min access tokens and replay revocation - worth doing + deliberately, not in an incident. -- tangled.sh