diff --git a/architecture.dj b/architecture.dj new file mode 100644 index 0000000..3f7987d --- /dev/null +++ b/architecture.dj @@ -0,0 +1,118 @@ +# architecture + +How linji is built. Why: [vision](vision.dj). When: [roadmap](roadmap.dj). + +## Stack + +- **Zig (0.16) everywhere.** App code builds on **zat** + (`tangled.org/zat.dev/zat`) — the same primitives library our PDS is built + on. JWT/DPoP/OAuth helpers, XRPC client, DID/handle resolution, MST, + firehose/jetstream all ship in zat. No TS/Bun, no node_modules. +- **Headless core, thin shells.** All app logic lives in a library + (`app0/src/core/`): accounts, storage backends, community model, records. + The `linji` CLI is shell #1; the web appview (phase 001+) is shell #2. + Tests drive the core directly — no UI in the loop. +- **PDS: our zds fork** at `./zds/` (ours to patch). Upstream checkout: + `~/computing/lib/zds`. + +## One schema, two publicity modes + +`at.linji.*` records are canonical; the lexicon is the interop layer. +Storage is a per-community choice, never a data migration: + +| | public mode | member-only mode | +|---|---|---| +| storage | author's public repo | zds permissioned space | +| PDS requirement | any | space-capable (zds) | +| reads | firehose → appview indexer | space routes (`listRecords`, `listRepoOps`) | +| readable by | the network | members only | +| phase | 003 | 001 | + +The headless core implements both behind one substrate-agnostic API +(`post` / `read` / `watch`). Fallback is symmetric: if proposal 0016 dies, +the spaces backend gets swapped for records and nothing else changes. + +## Co-location principle + +A space has two server roles: + +- **authority host** (community account's PDS): space config, member-list + policy, notify fanout, and the writer registry — rev + digest per writer, + no content. +- **repo host** (each writer's PDS): that writer's records in a per-writer + permissioned repo, signed with their keys. + +During the invite-only era both roles live on **one hosted zds**: the whole +space is one SQLite file, aggregation is a local query — no indexer, no +syncer. Messages still belong to each poster's own repo: per-DID, signed, +portable via `getRepo` full-state export. + +The constraint this creates: every space writer needs a space-capable PDS, +so hosted zds accounts for invitees are load-bearing, not just convenient. +When writers later distribute across hosts, the same model degrades to +fan-out queries (few hosts) or a syncer (many) — a cost paid when +distribution arrives, not before. + +## Testing: the simulation rig + +We build on an experimental PDS surface; the counterweight is that every +failure must be reproducible. The zds fork gains dev-only tooling behind +`ZDS_DEV_TOOLS=true`: + +- one centralized clock module with freeze/offset (replacing the six + scattered `now()` call sites); +- `dev.zat.debug.snapshot` / `restore` (SQLite + blobstore); +- `dev.zat.debug.setClock` — fast-forward token expiry, replay expiry races; +- later, as needed: scripted faults (dropped notifications, scheduled + revocations). + +The scenario suite (Zig tests driving the headless core against a fresh zds) +pins every behavior: multi-account login, post/read/watch, refresh after +fast-forward, member denial. Rule: **no zds upgrade reaches the class before +the scenario suite passes on it.** + +## Conventions + +- **IDs**: project-wide ids are ≥3 digits, zero-padded — phases 000–003, + milestones M001…, slices S001…. +- **Accounts**: the session store is multi-account from the first struct — + keyed by DID, per-account DPoP key + tokens, file mode 0600. +- **Thin space module**: all `com.atproto.space.*` access behind one core + module; if the upstream surface shifts, one file changes. +- **Lexicons**: JSON is the source of truth; structs hand-rolled until zlex + exists. + +## zlex (lexicon codegen, post-M001) + +No Zig lexicon codegen exists; zat's roadmap defers it ("probably a separate +project"). We build it as a separate tangled repo (`zlex/`): a build-step +generator, lexicon JSON → `.zig` structs with parse + validation. Not +comptime (cross-file refs, unions, error quality). Starts after M001 proves +real usage; M001 hand-rolls the one `at.linji.post` struct, written exactly +as the generator will emit it. + +## Collaboration with zat.dev + +Load-bearing, not opportunistic: phase 001 is gated on proposal 0016 +stabilizing, so contributing *is* the roadmap. Queue, in order: + +1. simulation-rig patches (deterministic testing for zds); +2. typed repo write helpers (`createRecord` / `applyWrites`, `Tid.now`) — + already on zat's near-term roadmap; +3. space-scope client ergonomics; +4. concrete 0016 feedback from a real group-chat consumer; +5. zlex, once M001 proves usage. + +## Workspace + +| dir | repo | what | +|---|---|---| +| `app0/` | tangled `did:plc:jdoqsgjhdnixt3zkmma6ows6` | linji CLI → app | +| `zlex/` | tangled `did:plc:vokl5wj3yzftb5tequiawxih` | lexicon codegen (post-M001) | +| `zds/` | tangled `did:plc:rox3gwjqr7dwrwyiifpav6t7` | our zds fork (rig patches) | +| `docs/` | — | this documentation | +| `www/` | — | Lume landing site | +| `liputoki/` | — | finished Gleam experiment; patterns only, no code reuse | + +Upstream checkouts live in `~/computing/lib/` (zds at `zat.dev/zds`). Never +edit the sibling zat repo from zds work without making the change explicit. diff --git a/atproto.dj b/atproto.dj new file mode 100644 index 0000000..e805f9a --- /dev/null +++ b/atproto.dj @@ -0,0 +1,61 @@ +# atproto glossary + +Concept budget: 6. Everything else is boxed out (see bottom). +Rule: a concept only earns a place here after we've needed it. +Write entries in our own words; delete entries that stop being true. + +## The 6 + +**repo** — a user's signed document store. Like a personal git repo hosted by +their PDS. Every record linji writes lives in somebody's repo: chat messages +in the author's, community definitions in the community account's. + +**record** — one JSON document in a repo. Validated against a lexicon (which +is just a JSON schema). The same `at.linji.*` shape works in both publicity +modes. + +**collection** — a folder of same-typed records, named by NSID: +`at.linji.post`, `at.linji.membership`. A repo is just a set of collections. + +**AT-URI** — `at://did/collection/rkey`. The universal path. Records reference +each other with these; post → channel → community is just AT-URI fields on +records. Space URIs extend the shape: `at://{authorityDid}/space/{type}/{skey}`. + +**DID + handle** — identity. DID is the permanent id (`did:plc:…`), handle +(`@you.linji.at`) is the readable alias that resolves to it. One DID, one +repo, present in many communities. + +**space** — a permissioned data space (proposal 0016): where member-only +records live. Two server roles: an authority (policy + writer registry, no +content) and each writer's repo host (their signed records). Access is by +space credential, exchanged from a short-lived delegation token. During the +invite-only era, one zds plays both roles. + +## Frozen defaults (reversible, don't re-litigate) + +- AI role: deferred to phase 001. Chat first. +- Chat publicity: member-only (spaces) in phase 001; public records in + phase 003. Same schema in both modes. +- Invitee PDS: hosted zds. Load-bearing — space writers need a space-capable + PDS. +- Iteration sandbox: local zds fork (`./zds/`), 2–3 test accounts. No + federation, no relay. + +## The box (do not open until forced) + +Relay, firehose, jetstream, AppView-as-concept, MST, CAR files, PLC +directory, DID methods, labelers, federation. Network plumbing. Matters when +we interoperate with apps we didn't write — phase 002 earliest. Until then +"the network" is a single opaque dependency behind zat's XRPC client. + +## Reference notes + +- roomy.space's published lexicons are event-stream schemas, not records; + their record-based redesign is gated on the same permissioned-data proposal + as our spaces mode. Watch, don't copy. +- zds permissioned spaces (`com.atproto.space.*`, simplespace member lists) + are the member-only substrate. Experimental — all access behind one thin + core module, and every zds upgrade must pass the scenario suite before the + class touches it. +- Lexicon conventions: [ref/lexicon-style-guide.dj](ref/lexicon-style-guide.dj) + (Bluesky's guide, CC-BY). diff --git a/plan/atproto.dj b/plan/atproto.dj deleted file mode 100644 index ea4772b..0000000 --- a/plan/atproto.dj +++ /dev/null @@ -1,56 +0,0 @@ -# atproto for linji — working glossary - -Concept budget: 5. Everything else is boxed out (see bottom). -Rule: a concept only earns a place here after we've needed it. -Write entries in our own words; delete entries that stop being true. - -## The 5 - -**repo** — a user's signed document store. Like a personal git repo hosted -by their PDS. Every record linji writes lives in somebody's repo: chat -messages in the author's, community definitions in the community account's. - -**record** — one JSON document in a repo. Validated against a lexicon -(which is just a JSON schema; treat our lexicons as generated from the TS -types, not hand-designed). - -**collection** — a folder of same-typed records, named by NSID: -`at.linji.post`, `at.linji.membership`. A repo is just a set of collections. - -**AT-URI** — `at://did/collection/rkey`. The universal path. Records -reference each other with these; that's the whole linking model -(post → channel → community is just AT-URI fields on records). - -**DID + handle** — identity. DID is the permanent id (`did:plc:…`), -handle (`@you.linji.at`) is the readable alias that resolves to it. -One DID, one repo, present in many communities — this is the Discord-shape -we get for free. - -## Frozen defaults (reversible, don't re-litigate) - -- AI role: deferred to Phase 1. Chat first. -- Chat publicity: member-only for now; revisit before 同喜班 dogfood. -- Invitee PDS: hosted zds. -- Iteration sandbox: local zds + 2–3 test accounts. No federation, no relay. - -## Milestone 1 (current) - -Two local users exchange messages in one topic on local zds. -Concepts touched: the 5 above, plus OAuth login. Nothing else. - -## The box (do not open until forced) - -Relay, firehose, jetstream, AppView-as-concept, MST, CAR files, PLC -directory, DID methods, labelers, federation. These are network plumbing. -They matter when we interoperate with apps we didn't write — Phase 2 -earliest. Until then "the network" is a single opaque dependency behind -`@atproto/api`. - -## Reference notes - -- roomy.space's published lexicons are event-stream schemas, not records; - their record-based redesign is gated on the same permissioned-data - proposal as our zds plan. Watch, don't copy. -- zds permissioned spaces (`com.atproto.space.*`, `managing-app` mode) - are the private-data primitive. Experimental — all access behind one - thin appview module (see vision.dj). diff --git a/plan/vision.dj b/plan/vision.dj deleted file mode 100644 index 7a20c40..0000000 --- a/plan/vision.dj +++ /dev/null @@ -1,120 +0,0 @@ -# linji.at — vision & plan - -## What it is - -A place for 修学, built on atproto. Two commitments from day one: - -1. **A home for study.** A community platform where 修学 (自修 + 共修) happens, - with an AI participant that helps rather than lectures. -2. **A better shape for community.** Zulip's topics (the part worth keeping), - Discord's account model (one global identity, many communities), atproto's - ownership (your identity and your words are yours, not the platform's). - -Seeded by one real community (同喜班), invite-only, then opened. - -## Why atproto fits - -- "One account for many servers" is not a feature to build — it is atproto's - native shape. Identity (DID + handle) is global and user-owned; communities - are spaces you join, not accounts you create. Nobody has to resolve into a - single context; the same person can be present in many. -- Data portability: messages as records in user repos means leaving the - platform doesn't erase you. -- Existing TS tooling: `@atproto/api`, lexicon codegen, OAuth, firehose. - -## Data model (lexicons under `at.linji.*`) - -- `community` — definition record, lives in the community's own repo - (a community account run by its moderators). -- `channel` — belongs to a community; name + description. -- `membership` — record in the member's repo: community ref, role, invitedBy. -- `invite` — record in the community repo: code, maxUses, expiry. This is the - invite-only gate. -- `post` — record in the author's repo: community ref, channel, **topic - (string, Zulip-style)**, text, optional reply ref. Topic-first threading is - the core reading experience. **Decided: each user owns their chat messages - as records in their own repo.** -- `study.journal` — 心得/自修 notes. **Decided: private data lives in - permissioned spaces, not public repos** (zds, below). - -Realtime: optimistic write + confirmation over the firehose; the appview -fans out to websocket clients. Second-scale latency is fine — this is a -study community, not a trading floor. - -**Consequence to keep in view:** repo records are world-readable — anyone -with relay access can read community chat. For an invite-only community that -means "invite-only" gates *participation*, not *readership*. If 共修 -discussion should be member-readable-only, chat itself moves into -permissioned spaces. Decide before dogfooding with the class. - -## Architecture - -- **Identity/auth**: atproto OAuth from day one. Members can bring their own - PDS; default to hosted PDS for invitees who don't have one. -- **Appview**: TS/Bun service. Indexes community + post records, serves the - web client, runs invite validation, moderation hooks, websocket fanout. -- **Client**: web first, bilingual zh/en from the start — equal citizens, - not a translation pass added later. -- **AI**: two distinct roles, keep them separate: - - **Bot participant** (`@…linji.at` account): joins topics when invited, - asks Socratic questions on the text under discussion, summarizes 共修 - threads. Its replies are ordinary records — the AI is a community member, - not a sidebar. - - **Private 自修 companion**: journaling, reflection prompts, RAG over the - community's own course materials. This handles sensitive content. - -## Private data: zds permissioned spaces - -**Decided.** Private data (journals, 心得, anything member-only) uses the -permissioned-data implementation in **zds** (`tangled.org/zat.dev/zds`) — an -atproto PDS in Zig by zzstoatzz.io, tracking Bluesky proposal 0016 -(permissioned data). What it gives us: - -- `com.atproto.space.*` protocol routes: spaces as the private-data primitive, - records keyed per writer repo, delegation tokens, sync. -- `com.atproto.simplespace.*` management routes with three access modes. - The **`managing-app` mode is the linji fit**: the PDS calls our appview's - `checkUserAccess` hook, so community roles/membership stay application - records — the PDS stays a dumb substrate, linji owns group semantics. -- Space URIs: `at://{authorityDid}/space/{spaceType}/{skey}`. - -Risk: this surface is experimental (`ZDS_PERMISSIONED_DATA=true` gate), -upstream proposal still moving (member lists were just pulled from the -protocol down to apps), zds is pre-1.0. Mitigation: all private-data access -goes through one thin module in the appview; if zds's surface shifts, only -that module changes. - -## Honest constraints - -- atproto group-chat-at-scale is unsolved territory. Bluesky DMs are - centralized; there is no mature open lexicon for Discord-shaped group chat. - We are building in the open part of the map. -- Moderation is a real cost the moment a second community joins. Roles, - labelers, and community-level autonomy need design before phase 2. - -## Phases - -**0 — Foundation.** Lexicons published; dev PDS; OAuth login; one hardcoded -community; channels/topics/posts CRUD; realtime delivery. -Done when: two accounts hold a topical conversation from two browsers. - -**1 — Dogfood.** Migrate 同喜班 off Zulip. Invite codes. AI bot in topics -(questions on the 课文, 八步骤-style reflection prompts). -Done when: the class holds one full 共修 session in linji.at and doesn't -go back to Zulip to finish it. - -**2 — Multi-community.** One account, many communities; community creation -by invitation; moderator roles. -Done when: a second community runs for a month without the seed community's -involvement. - -**3 — Open.** Public communities, directory, federation hardening. - -## Open questions - -1. AI emphasis first: in-topic participant, or private 自修 companion? - (Phase 1 concern — doesn't block foundation.) -2. Chat publicity: world-readable repo records, or member-only spaces for - chat too? (Must decide before 同喜班 dogfood.) -3. Default PDS for invitees without one: host our own zds instance, or - point at pds.zat.dev? diff --git a/lexicon-style-guide.dj b/ref/lexicon-style-guide.dj similarity index 100% rename from lexicon-style-guide.dj rename to ref/lexicon-style-guide.dj diff --git a/roadmap.dj b/roadmap.dj new file mode 100644 index 0000000..3b6bdbe --- /dev/null +++ b/roadmap.dj @@ -0,0 +1,72 @@ +# roadmap + +When, in what order, gated by what. Why: [vision](vision.dj). How: +[architecture](architecture.dj). + +## Phases + +**000 — Foundation.** +Lexicons drafted; dev PDS; OAuth login; one hardcoded community; +channels/topics/posts CRUD; delivery. +Done when: two accounts hold a topical conversation. + +**001 — Dogfood.** **Locked: we stay here until proposal 0016 +(permissioned data) is stable and broadly adopted.** +Migrate 同喜班 off Zulip. Chat runs member-only on spaces (same schema). +Invite codes. AI bot in topics (questions on the 课文, 八步骤-style +reflection prompts). Roles v0. Contribute to 0016 stabilization. +Done when: the class holds one full 共修 session in linji.at and doesn't go +back to Zulip to finish it. + +**002 — Multi-community.** (post-lock) +One account, many communities; community creation by invitation; moderator +roles; managing-app access hooks (`checkUserAccess`); BYO-PDS for members on +space-capable PDSes. +Done when: a second community runs for a month without the seed community's +involvement. + +**003 — Open.** (post-lock) +Public communities in records mode (any PDS); appview indexer over the +firehose; directory; federation hardening. + +## M001 (current) + +One `linji` CLI, multiple logged-in accounts, one topic on local zds. + +``` +linji login alice.test # OAuth loopback, adds to multi-account store +linji login bob.test +linji post --topic "课文讨论" "…" +linji post --as bob.test --topic "课文讨论" "…" +linji read # merged across member repos +``` + +Substrate: public records + `listRecords` (2–3 accounts need no indexer). +Concepts touched: repo, record, collection, AT-URI, DID+handle, OAuth login. +Nothing else. + +Done when: one binary holds alice + bob sessions; messages posted as each +appear merged in one topic; a scenario test proves it against the rig, +including clock-fast-forwarded token refresh for both accounts. + +## Slices + +- **S001** — app0 scaffold: build.zig + zat dep, `core/` layout, + multi-account session store, `tools/sandbox.sh` (fresh zds from the fork, + permissioned data + dev tools on, alice/bob/tongxi accounts). +- **S002** — rig patch into the zds fork: clock centralization, + snapshot/restore, setClock behind `ZDS_DEV_TOOLS`. +- **S003** — `linji login`: OAuth loopback against zds. +- **S004** — `linji post` / `read` / `watch`: `at.linji.post` records. +- **S005** — M001 scenario test (the done-when proof). +- **S006** — spaces backend: same schema, member-only mode (phase 001 entry). + +## Open questions + +1. AI emphasis first: in-topic participant, or private 自修 companion? + (Phase 001 concern — doesn't block foundation.) + +Resolved: chat publicity (member-only spaces in phase 001, public records in +phase 003); invitee PDS (hosted zds — load-bearing for spaces); stack (Zig + +zat); codegen (zlex, post-M001); phase 001 gate (proposal 0016 stable + +adopted). diff --git a/vision.dj b/vision.dj new file mode 100644 index 0000000..bbd6e98 --- /dev/null +++ b/vision.dj @@ -0,0 +1,70 @@ +# linji.at — vision + +What this is and why. How: [architecture](architecture.dj). When: +[roadmap](roadmap.dj). + +## What it is + +A place for 修学, built on atproto. Two commitments from day one: + +1. **A home for study.** A community platform where 修学 (自修 + 共修) happens, + with an AI participant that helps rather than lectures. +2. **A better shape for community.** Zulip's topics (the part worth keeping), + Discord's account model (one global identity, many communities), atproto's + ownership (your identity and your words are yours, not the platform's). + +Seeded by one real community (同喜班), invite-only, then opened. + +## Why atproto fits + +- "One account for many servers" is not a feature to build — it is atproto's + native shape. Identity (DID + handle) is global and user-owned; communities + are spaces you join, not accounts you create. The same person can be present + in many without resolving into a single context. +- Data portability: messages as records in user repos means leaving the + platform doesn't erase you. +- Tooling on our chosen stack already exists and is open to contribution: + zat (Zig atproto primitives) and zds (Zig PDS with a permissioned-data + prototype), both by the same author. + +## Data model + +Lexicons under `at.linji.*`. One schema, two publicity modes — see +[architecture](architecture.dj). + +- `community` — definition record, lives in the community's own repo + (a community account run by its moderators). +- `channel` — belongs to a community; name + description. +- `membership` — record in the member's repo: community ref, role, invitedBy. +- `invite` — record in the community repo: code, maxUses, expiry. The + invite-only gate. +- `post` — record in the author's repo: community ref, channel, **topic + (string, Zulip-style)**, text, optional reply ref. Topic-first threading is + the core reading experience. Each user owns their chat messages as records + in their own repo. +- `study.journal` — 心得/自修 notes. Private: permissioned spaces only, + never public repos. + +## AI + +Two distinct roles, kept separate: + +- **Bot participant** (`@…linji.at` account): joins topics when invited, + asks Socratic questions on the text under discussion, summarizes 共修 + threads. Its replies are ordinary records — the AI is a community member, + not a sidebar. +- **Private 自修 companion**: journaling, reflection prompts, RAG over the + community's own course materials. Handles sensitive content; private data + only. + +## Honest constraints + +- atproto group-chat-at-scale is unsolved territory. There is no mature open + lexicon for Discord-shaped group chat. We are building in the open part of + the map. +- Member-only chat rides proposal 0016 (permissioned data), which is + experimental and still moving. Containment: one schema across both + publicity modes, full-state export (`getRepo`), a simulation-tested local + PDS fork. See [architecture](architecture.dj). +- Moderation is a real cost the moment a second community joins. Roles, + labelers, and community-level autonomy need design before phase 002.