From 6d98273c0d0f5443e46f15bca4e588787dfb691e Mon Sep 17 00:00:00 2001 From: Yuto Nishida Date: Fri, 3 Jul 2026 23:05:49 -0700 Subject: [PATCH] [channel-party] Add design doc --- channel-party/DESIGN.md | 514 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 514 insertions(+) create mode 100644 channel-party/DESIGN.md diff --git a/channel-party/DESIGN.md b/channel-party/DESIGN.md new file mode 100644 index 0000000..155c0e7 --- /dev/null +++ b/channel-party/DESIGN.md @@ -0,0 +1,514 @@ +# channel-party — Design + +Status: draft / pre-implementation +Last updated: 2026-07-03 + +A Discord-inspired chat platform whose defining property is **extensibility**: the +number of channel kinds and message kinds is expected to grow without bound, while +the shared infrastructure (`channel-party-core`, `channel-party-frontend`) stays +fixed and simple. + +channel-party is **agnostic to how a user experiences the app**. The core is +headless; a web frontend is merely the first consumer. Everything below is designed +so that adding a new kind of channel or message touches **zero lines of core or +frontend-shell code** — it adds a self-contained *vertical slice* instead. + +--- + +## 1. Guiding thesis: envelope + registry, types as vertical slices + +The tension is: channels and messages are open-ended and infinitely flexible, but +the core and frontend must stay stable. The resolution is to make **core a +mechanism, not a policy**. + +- **Storage is schemaless.** There is no base class and no required fields on a + message, because core never looks inside one. Every channel and every item is an + *envelope*: an id, a type discriminator, a containment edge, and an opaque JSON + payload. Core stores envelopes; it does not interpret payloads. +- **Behavior is pluggable.** A *Kind* (channel-kind or item-kind) is a registry + entry keyed by the type string. Core calls into it through a fixed set of + capability traits; core never `match`es on a concrete type. + +A "type" is therefore not a base class and not a separate service. It is a +**vertical slice** — a Rust crate (+ colocated frontend islands) that implements +only the capabilities it needs. The core and frontend are the *horizontal* +infrastructure; each type is a *vertical* plugin. + +--- + +## 2. The two super-types + +There are exactly **two super-types**, each an open-ended, plugin-extensible family: + +- **`channel-type:*`** — containers. +- **`item-type:*`** — everything that lives in a container. + +**Channels contain items *and* other channels, arbitrarily.** + +Concrete kinds in the initial design: + +| Super-type | Kinds | +| --- | --- | +| `channel-type:` | `basic` · `space` · `discord-compatible/{guild,section,channel,forum}` · `canvas` | +| `item-type:` | `basic` · `discord-compatible/{message,cached-message,cached-reaction,cached-user}` · `canvas-text-box` | + +The word "message" is intentionally *not* a super-type. Reactions, cached external +users, and canvas text boxes are all **items** — "item" is the generic content +object; a chat message is just its most common kind. + +### Users sit outside the taxonomy + +There is exactly **one real principal: the native channel-party `User`.** It is not +a super-type because it is not extensible — there is one native user representation, +and it is the only thing that authenticates, owns, or holds permissions. + +External identities are **not real users**. A Discord user is stored as an +`item-type:discord-compatible/cached-user` and is inert attribution data. A native +`User` carries `linked-users`: references to the cached-user items that represent it +on external platforms. + +Invariant enforced by core: **only a native `User` can be a principal; items are +inert content.** Auth, sessions, ownership, and permission checks resolve +exclusively against `users`. + +This makes authorship cleanly polymorphic (which is fine precisely *because* there +is no base class): + +- `item-type:basic` → author is a native `UserId` (real FK). +- `item-type:discord-compatible/cached-message` → author is a reference to a + `cached-user` item, optionally resolvable up to a native user via a link. +- `item-type:discord-compatible/message` (originates on channel-party, pushed to + Discord via webhook) → author is a native `User`. + +--- + +## 3. Data model + +Three tables — **two super-types, plus the fixed `users` substrate.** + +``` +users (id, handle, auth…, created_at) -- first-class, native principals only +user_external_links (user_id → users.id, item_id → items.id) -- the `linked-users` edge, bidirectional +channels (id, type_id, container?, payload_json) -- container = parent channel (null = root) +items (id, type_id, container?, external_key?, payload_json) +``` + +- **Uniform containment edge.** Both channels and items carry + `container: Option`. A channel's children = *everything (channels or + items) whose `container` is that channel*. Arbitrary mixing of sub-channels and + items falls out for free. `container` is nullable: a root channel or a + guild-scoped `cached-user` may have none. +- **Minimal required fields.** The only truly universal envelope fields are `id` and + `type_id`. Even `container` is optional. Everything else lives in `payload_json` + and is understood only by the owning kind. +- **`external_key`** on items is the dedup/upsert handle for mirrored external + objects. There must be exactly **one** `cached-user` item per Discord user + (`external_key = discord user id`, unique per namespace), referenced by thousands + of cached-messages — never a copy per message, or `linked-users` becomes + meaningless. +- **IDs are ULID / UUIDv7** (time-ordered). Consequences: a channel feed sorts by id + with no mandated `timestamp` field; canvas boxes get a stable creation order for + free; "jump to timestamp T" is just seeking to the ULID whose time prefix is T + (see §5). + +Writes go through core's mutation API, which calls the kind's `validate` before +committing — so no path (including the debug shell) can persist an invalid envelope. + +--- + +## 4. The Kind abstraction and its capabilities + +Two registries — one per super-type. Kinds are trait objects; capabilities are +**opt-in default methods**, so a trivial kind implements two lines and a rich one +implements many. + +```rust +pub struct TypeId(String); // "discord-compatible/channel" + +pub struct Channel { id: ChannelId, type_id: TypeId, container: Option, payload: Json } +pub struct Item { id: ItemId, type_id: TypeId, container: Option, + external_key: Option, payload: Json } + +#[async_trait] +pub trait ChannelKind: Send + Sync { + fn type_id(&self) -> &TypeId; + + fn validate(&self, p: &Json) -> Result<()> { Ok(()) } // §3 + + /// Discover this channel's contents. `query` and the returned value are type-defined. §5 + async fn contents(&self, cx: &StoreCtx, ch: &Channel, query: Json) -> Result; + + fn index(&self, p: &Json) -> Option { None } // inline projection, §6 + fn membership(&self) -> Option<&dyn Membership> { None } // §8 + fn routes(&self) -> Option { None } // extra HTTP routes, mounted /ext/ + fn debug_commands(&self) -> Vec { vec![] } // §8 + fn debug_summary(&self, c: &Channel) -> Option { None } // §8 + // Frontend island is declared out-of-band via a web/ manifest, not in Rust. §9 +} + +#[async_trait] +pub trait ItemKind: Send + Sync { // same shape, no `contents`/`membership` + fn type_id(&self) -> &TypeId; + fn validate(&self, p: &Json) -> Result<()> { Ok(()) } + fn index(&self, p: &Json) -> Option { None } + fn debug_summary(&self, i: &Item) -> Option { None } +} +``` + +Kinds are grouped by **namespace, not by leaf**: `discord-compatible/` is one crate +holding guild + section + channel + forum + message + cached-message + +cached-reaction + cached-user + the shared Discord runtime, because they share a +great deal (one rate-limited client, one search index). This mirrors the +`namespace/leaf` id scheme. + +### Capability coverage by example family + +| Capability | `basic` | `space` | `discord-compatible` | `canvas` | +| --- | :-: | :-: | :-: | :-: | +| `validate` | – | – | ✓ | ✓ | +| `contents` (channel) | list+paginate | name search | fetch-all subtree | viewport bbox | +| `index` (inline) | name → FTS | – | (via RuntimeComponent) | coord → spatial | +| `membership` | ✓ | ✓ | reject / proxy to Discord | – | +| `RuntimeComponent` | – | – | sync (Primary) + semantic index (Derived) | spatial index (Derived) | +| `routes` | – | – | webhook receiver | – | +| island (frontend) | message list | search UI | threaded view | pan/zoom canvas | + +`basic` rides the generic path and implements almost nothing; `canvas` is +frontend- and index-heavy; `discord-compatible` is runtime-heavy. Same plugin +surface, radically different weights. + +--- + +## 5. Contents discovery + +**How a channel discovers its contents is itself a per-channel-type capability**, +not one fixed endpoint. Core provides one generic dispatch plus a small, *closed* +set of store primitives; each kind composes its own discovery strategy on top. Core +gains nothing per type. + +One HTTP route → one trait method: + +``` +POST /api/channels/:id/contents { query } → registry.channel(ch.type_id).contents(...) +``` + +`query` and the response are **opaque to core**; the island for that channel type +knows their shape. `StoreCtx` hands the kind these primitives (this set does not +grow as types are added): + +```rust +cx.children(container, filter{super_type?, type_ids?}, page, order) // one level, cursor-paginated +cx.descendants(root, filter, depth?) // whole subtree (fetch-all) +cx.seek_time(container, timestamp) -> Cursor // ULID ⇒ time-jump is a cursor +cx.search(scope, text, filter, page) // FTS over the index() projection, §6 +``` + +The three worked examples all reduce to these: + +| Channel kind | `contents(query)` | +| --- | --- | +| `basic` | `children(id, {Item, [basic]}, page, TimeDesc)`; `query.at` → `seek_time` first ⇒ jump-to-timestamp is free | +| `space` | `search(descendants(id), query.q, {Channel, [basic]}, page)` → paginated name matches | +| `discord/guild` | `descendants(id, {Channel, [channel, section]})` → whole subtree at once; island builds the tree | +| `discord/section` | same, scoped to itself | + +Discovery is **recursive**: a guild island renders channel *references*; opening one +mounts that child's island, which calls its *own* `contents`. Containers delegate to +children exactly as they delegate item rendering (§9). + +--- + +## 6. Indexing + +Substring name-search (`space`) can't run over opaque JSON efficiently — `name` +lives in `basic`'s payload. So the honest price of schemaless storage + type-specific +discovery is a **kind-declared projection** of searchable/sortable fields. Core still +hardcodes no field; it only calls a kind-supplied function. Two tiers: + +### Tier 1 — inline projection (the 90% case) + +`index(payload) -> IndexEntry` is a pure function applied **transactionally on +write** into core's built-in substrates: + +- **FTS5** (trigram tokenizer → true substring search) for `name` / body text. +- Expression indexes for declared **sort keys**. +- A **2D / R-tree** substrate for coordinates (`canvas-text-box`). + +`IndexEntry { name?, text?, sort_key?, coord? }`. Cheap, deterministic, no I/O, no +task. A kind that needs no indexing returns `None` and costs nothing. + +### Tier 2 — async indexing via `RuntimeComponent` (§7) + +Anything expensive, async, cross-item, or externally-enriched — embeddings for +semantic search, a thread/reaction adjacency graph — is a supervised runtime +component, not an inline function. It may write core substrates or a table the type +owns. + +**Type-owned index tables** are the escape hatch: a crate may ship namespaced +migrations (`discord_*`, `canvas_*`) registered with core's migrator, giving it a +fully self-contained search stack — *own table (schema) + RuntimeComponent (writer) + +`contents` (reader)* — that core never learns the shape of. Cost: those schemas must +be in each crate's `.sqlx` offline cache for compile-time query checking (§13). + +--- + +## 7. `RuntimeComponent` — the one supervised-task abstraction + +Ingesting workers and derived indexers are **the same machine**: a supervised, +long-lived task that does an initial batch pass, then reacts to a stream +("backfill-then-stream"). The Discord sync worker is *initial sync, then poll*; a +semantic indexer is *initial backfill, then react to change events*. One loop, one +trait — there is **no separate `Worker` and `Indexer`.** + +```rust +#[async_trait] +pub trait RuntimeComponent: Send + Sync { + fn name(&self) -> &str; + fn interests(&self) -> Interests { Interests::none() } // a schedule, and/or the change stream (by type) + fn version(&self) -> u32 { 0 } // bump ⇒ reset; component decides what reset means + fn writes(&self) -> WriteScope { WriteScope::Derived } // Primary | Derived — see safety note + async fn run(&self, cx: RuntimeCtx) -> Result<()>; // backfill + steady-state, all in one loop +} +``` + +`RuntimeCtx` provides the store handle (restricted per `writes()`), the change-stream +subscription filtered by `interests`, the scheduler, and shutdown. + +Two facets that are **behavior, not interface**: + +- **Reset semantics differ by input source.** A derived component resets by replaying + *local* primary data — deterministic, cheap. An ingesting component resets by + re-fetching from Discord — rate-limited, non-deterministic, unreconstructable + locally. Same `version()` trigger, different `run` behavior; the interface doesn't + care. +- **Write-scope safety.** `writes()` lets core hand back a restricted store handle: + an ingestor writes `Primary` envelopes (source of truth); a derived indexer is + confined to `Derived` tables, so a bug can't corrupt the source of truth. + +Components are **crate-contributed singletons**, not one-per-channel-instance. A +*single* Discord component manages all bridged guilds behind one shared rate-limited +client and one token bucket — far better than N workers fighting the rate limit. +They compose via the event bus: the sync component writes cached-messages and emits +change events; the semantic-index component consumes them — independent tasks, so a +slow embedding pipeline never blocks ingestion. + +--- + +## 8. Debug shell (`channel-party-core`) + +A REPL that lets developers and LLMs grok the running application. It is a **thin +read wrapper over the database** by default, with an explicitly-gated write surface. + +### Read side + +Direct DB reads (fast, honest view of what is actually stored): show channels, show +a channel's items, show users, inspect an envelope. Payloads print as JSON; the +optional per-kind `debug_summary` adds a one-line human summary when present. + +### Write-mode gate + +- **Off by default, per-session, never persisted** — a fresh shell is always + read-only, so exploration can't mutate anything by accident. +- `enable-write-mode` / `disable-write-mode` flip a session flag. Every mutating + command refuses until it is on (`"read-only; run enable-write-mode first"`). +- The prompt reflects the mode: **`cp[ro]>`** vs **`cp[write]>`**. +- **Writes route through core's mutation API + the kind's `validate`, never raw + SQL** — the one asymmetry with reads. A debug write cannot create an invalid + envelope or bypass an invariant. + +### Write commands are capability-gated + +`add-user-to-channel ` is backed by an optional `Membership` +capability on `ChannelKind`: + +```rust +pub trait Membership: Send + Sync { + async fn add_user(&self, cx: &StoreCtx, ch: &Channel, u: UserId) -> Result<()>; + async fn remove_user(&self, cx: &StoreCtx, ch: &Channel, u: UserId) -> Result<()>; + async fn members(&self, cx: &StoreCtx, ch: &Channel) -> Result>; +} +``` + +Resolution of `add-user-to-channel`: + +1. write-mode off → refuse; +2. `ch.kind().membership()` is `None` → **"channel-type `canvas` does not accept + users"** (this *is* the "accepts users" check — capability presence); +3. `Some` → execute. What "add a user" means is the kind's choice: `basic` writes a + membership edge (core offers a generic `channel_members` substrate); `space` + likewise; `discord-compatible/channel` may reject (membership is owned by Discord) + or proxy an outbound invite. + +### The write surface scales with types + +New types bring their own mutations (`move-box `, +`resync-channel `). The shell is a **generic command dispatcher**: kinds +contribute commands via `debug_commands()`, each flagged read/write so the mode gate +applies uniformly without the shell knowing what they do. Built-in +capability-backed commands (membership) + kind-registered commands, all behind the +one gate. The shell never grows per type. + +--- + +## 9. Frontend (`channel-party-frontend`) + +Astro serves a **static outline** (nav, channel tree, layout — entirely +type-agnostic). The dynamic, type-specific parts are **islands**. + +- A channel page fetches `{ id, type_id }`, then **dynamically imports** the island + bundle for that type from a client-side registry + `Map import(...)>` (dynamic import ⇒ code splitting). It mounts the + island; the island owns its own data fetching (calls `POST .../contents` with its + type-specific query), rendering, and interactions. +- **The registry map is generated at build time** by scanning `kinds/*/web/`, so + adding a type does not edit any central switch. Convention over central + registration — the frontend shell stays frozen. +- **Channel-kind islands render *containers* and delegate item rendering to + item-kind islands** via the same registry. The canvas channel island owns + pan/zoom/placement and delegates drawing each box to the `canvas-text-box` item + island; a new item type that can appear on a canvas is one new item island and the + canvas island never changes. The two axes (channel kinds, item kinds) scale + independently. +- **Live updates**: the frontend server exposes SSE backed by the core event bus; + islands subscribe to changes for their channel. + +### Rust ↔ TS contract + +To stop payloads from drifting, TS types are generated from the Rust payload structs +(`ts-rs`): each kind emits its payload/query/response types into its own `web/` dir, +which the island imports. The vertical slice stays coherent across the +language boundary. + +### HTTP surface (all served by `channel-party-frontend`) + +``` +GET /api/channels/:id -> { id, type_id, container } (generic) +POST /api/channels/:id/contents {q} -> type-defined contents (dispatch, §5) +GET /api/items/:id -> envelope (generic) +GET /api/events?scope=… -> SSE change stream (generic) +/ext//… -> kind-contributed routes (webhooks, etc.) (§4) +``` + +--- + +## 10. Component / crate layout + +``` +channel-party/ +├── flake.nix · Cargo.toml # nix + crane; one cargo workspace +├── crates/ +│ ├── cp-model/ # envelope, ULID ids, Kind + capability traits, Membership, +│ │ # RuntimeComponent, serde, ts-rs exports (the interface crate) +│ ├── cp-core/ # sqlx(sqlite) store · two Kind registries · contents dispatch · +│ │ # index substrates + inline projection · RuntimeComponent supervisor · +│ │ # event bus · migrator · gated debug shell +│ ├── cp-frontend/ # axum server: generic API + kind routes · serves Astro build · SSE +│ └── cp-bin/ # composition root: registers kinds + runtime components, boots core + frontend +├── kinds/ # vertical-slice plugins (Rust src/ + colocated web/) +│ ├── basic/ +│ ├── space/ +│ ├── discord-compatible/ (src/ + web/ + migrations/) # guild/section/channel/forum + items + runtime +│ └── canvas/ (src/ + web/ + migrations/) # canvas + canvas-text-box + spatial index +├── web/ # Astro app shell (outline) + island registry generated from kinds/*/web +└── DESIGN.md +``` + +`cp-core` depends only on `cp-model` (the trait/interface crate). It has **no +dependency on any concrete kind crate.** Concrete kinds are wired in exactly one +place — the composition root: + +```rust +let registry = Registry::builder() + .item(basic::BasicItem).channel(basic::BasicChannel) + .channel(space::Space) + .channels(discord::channels()).items(discord::items()) + .runtime(discord::Sync) // WriteScope::Primary — ingests messages/users/reactions + .runtime(discord::SemanticIndex) // WriteScope::Derived — embeddings, off the change stream + .channel(canvas::Canvas).item(canvas::CanvasTextBox) + .runtime(canvas::SpatialIndex) // WriteScope::Derived + .migrations(discord::MIGRATIONS).migrations(canvas::MIGRATIONS) + .build(); + +let core = cp_core::Core::open(db, registry.clone()).await?; // runs migrations, drives backfills +core.spawn_runtime(); // supervises every RuntimeComponent +cp_frontend::serve(core, registry, addr).await?; +``` + +Explicit registration (over `inventory`-style auto-registration) is chosen for +clarity, testability, and control over ordering. + +--- + +## 11. Build — Nix + crane + +- **Rust workspace via crane.** Shared `cargoArtifacts` across the workspace for + fast incremental caching; the whole workspace builds as one dependency graph. +- **sqlx in offline mode.** Compile-time query checking uses a committed `.sqlx` + query cache (`SQLX_OFFLINE=1`), so builds are hermetic with no live DB — required + under Nix. Type-owned migrations (§6) must be reflected in the offline cache. +- **Astro frontend** built as a separate derivation (`buildNpmPackage` / pnpm). The + island registry is generated from `kinds/*/web` at build time. Final package + combines the server binary with the static Astro output. +- **Devshell**: rust toolchain + node/pnpm + `sqlx-cli`. + +--- + +## 12. Testing strategy + +The whole point of splitting components is that each is independently +integration-testable. + +- **`cp-model`** — unit tests for envelope serde and trait defaults. +- **`cp-core`** — integration tests against a tempfile/in-memory sqlite: store CRUD, + the store primitives (`children`/`descendants`/`seek_time`/`search`), contents + dispatch, index backfill + versioning, and RuntimeComponent supervision. Core's + genericity is *proven* by testing it against a throwaway `test-support` kind — this + also keeps the plugin API honest. +- **Each kind crate** — its own integration tests. `discord-compatible` tests + rate-limit handling and cache/dedup logic against a **mock Discord API** + (`wiremock`), never the real one. +- **`cp-frontend`** — tests hitting the axum server against a seeded DB; island tests + via vitest/Playwright on the JS side. + +--- + +## 13. Capability summary + +Core is a fixed, small set of mechanisms: + +> envelope store · two kind registries · `contents` dispatch · index substrates + +> inline projection · one `RuntimeComponent` supervisor · event bus · migrator · +> gated debug shell + +Every channel/item type is a vertical slice (Rust crate + colocated islands) that +opts into only the capabilities it needs: + +| Capability | Surface | Where it runs | +| --- | --- | --- | +| `validate` | ChannelKind / ItemKind | core write path | +| `contents` | ChannelKind | core (HTTP dispatch) | +| `index` (inline) | ChannelKind / ItemKind | core write path (transactional) | +| `RuntimeComponent` | registry | core runtime (supervised) | +| `membership` | ChannelKind | core / debug shell | +| `debug_commands` | ChannelKind | debug shell | +| `debug_summary` | ChannelKind / ItemKind | debug shell | +| `routes` | ChannelKind | frontend server | +| island | `web/` manifest | frontend (Astro) | + +Adding a type touches none of these mechanisms — only the slice. + +--- + +## 14. Open questions / deferred decisions + +- **Permissions model.** Beyond "only native users are principals," the shape of + per-channel authorization is unspecified. Likely another capability, but deferred. +- **Membership storage.** Whether the generic `channel_members` substrate is + sufficient, or membership-heavy kinds should own their edge tables. +- **`external_key` namespacing.** Exact uniqueness scope for cached objects + (per-guild vs global Discord user identity). +- **Auth / session mechanism** for native users (out of scope for the core data + model; needed before the frontend is usable). +- **Registry ergonomics.** Whether to keep explicit registration or adopt + `inventory`/`linkme` distributed registration as the type count grows. +``` -- 2.51.2