A political conference and discussion platform, in Rust and Dioxus
wiki docs appview-roadmap.md
49 kB
Markdown
at main

AppView roadmap: the new backend #

Goal #

crates/appview becomes the ONLY backend the Dioxus app talks to. NHost auth, Hasura GraphQL, NHost storage, the Postgres schema and the serverless axum sidecar (backend/) all retire at the big-bang cutover (docs/cutover-runbook.md).

Done means all of these hold:

  1. Every read, write and live update the frontend performs today (src/graphql/*, src/nhost.rs, src/backend_api.rs, src/subscription.rs) is served by the AppView, behind the src/model.rs seam, with no request leaving for NHost or Hasura.
  2. Every request is authenticated by an AppView session bound to an atproto DID, and authorized by DID-keyed membership. No placeholder auth remains.
  3. cargo test --workspace in crates/ exits 0, and the browser suite (just test-browser) passes against an AppView instead of NHost.
  4. The migration pipeline (dump, extract, load) carries every interim row the frontend can show, and the runbook's verification gates are green on staging.

The kickoff phase (docs/rewrite-kickoff-plan.md) is finished: the skeleton, the generated schema, the Store seam, the OAuth stores and callback, the firehose consumer, the durable ballot core, the loader and the deploy unit all exist. This document is the phase after it: turning that skeleton into the backend.

Where it stands (2026-09-20) #

Built and tested in crates/appview, milestone by milestone below: sessions on atproto OAuth, the read and write gates, the node tree the frontend routes by, membership and the roster, meetings (speaker lists, the projector, the canvas), voting with blind-signed secret ballots, live topics, files, every endpoint the interim sidecar served, search and the feeds, and appview import, which loads a migrated wiki. docs/appview-api-coverage.md sets every data call the frontend makes against the method that answers it, and none is left without.

The frontend runs on it. Built with the appview feature, the three modules that speak to the interim's backend (graphql, backend_api, nhost) are the AppView's under the same names (src/appview/), so no component changes which function it calls; built without it, which is what ships, nothing is different. scripts/test-browser-appview.nu builds it, starts a dev AppView, and drives it in headless Firefox: the screens draw what was seeded, writing through them lands (a comment, a reaction, a page through the editor, a file and a picture, a place in the speaker list, a cell on the canvas, an open ballot and a secret one, blinded and cast from the browser's own wasm), a deleted comment goes to the bin and comes back, search finds a page by its text, a change made elsewhere arrives without a reload, and signing out ends the session at the AppView too.

The cutover has been rehearsed on a dump of production (docs/cutover-rehearsal.md): the real wiki loads, its files come across whole, every gate of the runbook is green, every account that can be recognized by its address takes its old account over without a failure, and a smoke test reads and writes it over HTTP. nu scripts/rehearse-cutover.nu runs all of that again from a fresh dump.

A real sign-in has been made too, on one machine (scripts/test-real-login.nu): the real appview, a real PDS, the PDS's own sign-in and consent pages in a browser, the token exchange, the profile read and a post written to the PDS.

The interim's own browser suite (test-browser.nu) runs against the AppView too (--appview), with no account: 143 of its 144 checks pass, and the one that does not is the stylesheet's.

What comes after the cutover is a redesign, not a milestone of this list: atproto gained non-public records (spaces, in alpha since 2026-08-20), and the owner asked that the wiki use them. docs/atproto-spaces-redesign.md has the design, the owner's five calls and the order of work (S1 to S7); crates/spaces-spike asked the alpha PDS whether the design's footing holds, and it does. Built so far, and held to that PDS by just test-spaces: the lexicons and the mapping between rows and records, the protocol in Rust, and in the AppView the first stage itself (src/spaces.rs): every page, comment and reaction also written as a record into a space per context, under the organization's account, with the AppView answering the PDS on who gets in. It is off unless configured. None of it touches the cutover, and none of it runs in production before spaces are released.

Findings that shape the plan #

  • The workspace did not build in the devshell. atrium-oauth's default HTTP client pulls reqwest/default-tls, so OpenSSL, against the recorded rustls-only decision (docs/atproto-stack-decisions.md, Rust server libraries). Fixed in M0 by supplying the one rustls client the decision asks for.
  • Writes trusted the caller. xrpc::caller_did read the DID straight out of the Authorization header, so anyone could write as anyone by naming them. Replaced in M1 by sessions.
  • /login is an SSRF primitive unless guarded. It is unauthenticated and fetches whatever host the caller names. The outbound client now refuses non-https and non-public addresses, at resolution time, so DNS rebinding gains nothing. A dev instance (no public URL) stays unrestricted for a local PDS.
  • The engine was five minor versions behind, and it mattered. turso 0.2.2 had no EXISTS, no IN (subquery), no upsert, and ignored the foreign-key pragma, so nothing was referentially checked. 0.7.2 has all four. Turning enforcement on surfaced two real defects at once (the firehose and the loader, both fixed). Recursive CTEs are still missing, which M3's path resolution has to design around.
  • The extractor leaves gaps the loader will now refuse. A context's parent_id is carried over verbatim, but in the interim tree a group or event can sit under a folder, which is a document here, so that key dangles. The node that owns a context without holding a membership row in it (a real case: docs/read-permissions.md) gets no owner row. Interim public contexts are all extracted as private. Extracted polls are never loaded, and are extracted wrong: the question is read from data.question and the state from data.open, neither of which the interim writes (a poll's name is its node's name, and open is mutable), and minVote, maxVote, hidden, the place in the tree and the RESULT are not read at all. Ballots cannot be carried; the outcome of every past vote can, and poll.counts is where it goes. M3 and M9, where all of it is now carried.
  • Nobody would have found their own account. The extractor carries an interim account under its interim id, "until the DID binding runs", and no such binding existed. Everyone with an account, which is everyone who has ever signed in, would have come back after the cutover as a stranger: their seats, owner rights and authorship held by an account that can never sign in, and not even open to a new invitation. A person now takes their old account over by signing in with the address it was registered under, where the interim had verified that address and a trusted PDS confirms it; anyone else is handed a seat at a time by claim link (crates/appview/src/legacy.rs).
  • A claim link the interim had spent would have opened its seat again. The interim keeps the token on the row after the seat is taken. Once a seat held by a carried account can be claimed, that old link, sitting in somebody's inbox, claims it. Spent tokens are left behind, and an owner mints a new one for a seat that needs it.
  • What was deleted would have come back. The interim bins a comment by stamping it, and the extractor took no notice of the stamp, so every comment somebody had deleted was extracted as live. A comment now comes across in the bin it was in. A reaction or a report in the bin is left behind.
  • The first audit of the API read the data layer and not its callers. The frontend deletes a comment through the same bin_node and update_node it uses for a page, from inside the comments component, so the table had a comment as something one posts and reads. It can also show a picture, be deleted, be emptied in place when it has been answered (deleting it would take everyone who answered along), and come back from the bin. None of that existed here, and the extractor dropped the picture without a word. Building it found three more: a move to another group left every ANSWER in the old group, read there and not in the new; a purge deleted the comments on a page and left the answers to them, and every reaction, in the database for good; and an answer was news of nothing, so the feed left answers out and the list of what has gone astray listed every one of them. A comment now knows the document its thread is on (root_id), which is what all three go by.
  • A row shown before the server has it never went away. The components show a comment or a speaker at once under a key they choose, and drop that row when a fetched one carries the same key. The AppView names rows itself, so in a real browser a posted comment sat as "Sending…" beside its own copy for as long as the page was open. No test below a browser could see it: the data layer's answers were right. The layer now hands a row's key back as its component chose it (src/appview/seen.rs).
  • Saving a page re-dated it to midnight. The editor sends a page's day back with every save an owner makes, and a day alone is stored as its midnight, so a page made a minute ago read "19 hours ago" after its first save. The interim does the same. Here the day a node already has leaves its time of day alone. Seen in the browser run, where a page is made and saved as a person would.
  • A public board would have told the world what a closed group's votes came to. The custody memo chose atproto records because everyone can see them, which is the point for a congress and a leak for a board meeting: the counts of a poll are its members' to know, and a hidden tally is its owners'. So a board is published only for a poll opened as public (openPoll.public_board, by default where the context is open to everyone), a hidden tally cannot have one, and every other secret poll keeps what needs no publication: a receipt for every ballot and a signed close-out.
  • Two record names, one a prefix of the other. The mirror told a poll's announcement from its close-out by whether the address contained wiki.radikal.poll, which wiki.radikal.pollCloseOut does too, so every close-out was counted as a second poll. Found by the first end-to-end test of publisher and mirror together; a record's collection is now read as the whole path segment.
  • Looking a ballot up named the voter. getBoardEntry asked that the caller be able to read the poll, so in a closed group a voter could only check their ballot with their session on the request: their name and their token in one call, which is the pairing the blind signature exists to keep from the server. It takes no session now, as a cast does.
  • A lost reply would have cost a vote. The AppView signs a voter's tokens once, and again only for the same blinded tokens. A browser that blinded fresh ones after a dropped answer would have been refused for good. The frontend keeps what it blinded before it asks, and a retry that finds its token already spent asks the board what that ballot said instead of assuming.
  • A change said where, and a feed needs to know what. A change named the page a comment was on and not the comment, so a feed hearing of one had nothing to fetch. A change now also carries the row it made.
  • The clock came off the JWT. Countdowns and cooldowns are reckoned against rows the server stamped, and the interim read the difference between the two clocks out of its token. A session here is no JWT, so every answer carries the server's clock (x-server-time) and the client measures the difference.
  • A report's screenshot was for nobody. It is filed in a context of its sender's, which whoever reads the reports is usually no member of. They may open it now, and a report takes only a screenshot its sender uploaded.
  • The site had no home. The interim's root is a context like any other: its members run the site, its content is the welcome page, and what sits at the top of the tree is made in it. The extractor skipped it as "the root every path starts under", and the AppView had nothing in its place. After a cutover nobody could have started a group at the top level, since a context is made under one the caller owns and at the top there was none to own; nobody could have read a report or put right what had gone astray, which were held to "whoever owns a site", and that meant any blog; and the welcome was gone. The home is a row now (kind = 'home', the empty path), carried with its owners and its welcome, and made at start where a datastore has none.
  • A place said nothing about itself. A group's front page, its cover and a redirect are the context's own data, as a page's are. context had no column for any of it and the extractor dropped it without a line in the report: every group's front page was to be lost. The public list also left out every site, where the interim leaves out the root.
  • Nothing could run a load. The loader was a library with no binary, and what a poll came to, a canvas and the reports live in tables it cannot know. appview import is the load step: one transaction, the same twice, and refused on a datastore somebody has signed in to.
  • There are no schema migrations. The entity tables are plain CREATE TABLE, so a datastore file made by an older binary keeps its old columns. Until migrations exist the file records its schema version and a binary refuses to start on another one, rather than failing a query at a time. Pre-cutover that costs nothing, since the view is rebuilt from the migration pipeline; after it, migrations become real work (M9).
  • The interim serves every member's email to every member. A column permission is per role and an owner is role user too, so one plain member could read 1,467 of the organisation's 2,007 addresses. The AppView can draw the line Hasura could not: listMembers serves an address to an owner of that context and to nobody else, and matches a search against one for an owner only.
  • Reads were ungated. Every read served private content to anyone. Gated in M2 on visibility and membership.
  • Reactions stayed ungated until an audit of the whole API found them. The kickoff had classed reactions as mirrors of public records, so anyone signed in could react to any string with any string, and who had reacted to a closed group's comment was served to the signed out. The interim's reactions are a member's, on content and comments. They are now held to that.
  • active is voting rights, not a read gate. The interim reads by membership alone, and migrations/0024 records what reinterpreting a write model as a read model cost: a context went dark for its own members. The AppView's gate follows the same line.
  • A document could not be addressed by path. The entity schema gave context a slug and document none, while the frontend routes every node by its key path (/a/b/c), so every existing link would have broken at cutover. The extractor dropped the key without a word, and with it index, mutable, attachable, the owner, updatedAt and the bin; it REPORTED a file's fileId and type as gaps rather than carrying them, which would have migrated every attachment as an empty page; and context.kind rejected wiki/site. Reconciled in M3.
  • Two people writing at once, and one of them failed. The engine's default is to refuse a write the moment another connection holds the write lock, and every request here has its own connection. Measured with 16 writers: 371 of 400 transactions refused with "database is locked". A meeting is exactly that load (a room joining a speaker list, 500 ballots in a minute). Every connection now waits its turn, up to ten seconds.
  • Speaker lists were classed as ephemeral and dropped by the extractor, but the speak app persists them and an assembly reads them back. M5 gives them tables; M9 migrates them or records why not.

Milestones #

Each lands with tests, committed on its own. A box is ticked only when the code is merged and its tests pass.

M0: build hygiene #

M1: identity and sessions (replaces NHost auth) #

M2: authorization core #

M3: the node tree the frontend routes by #

The frontend knows one tree of typed nodes, each reachable by the path of keys in its URL; the store keeps typed entities in separate tables. The design that joins them:

  • context and document are the two spines of the tree. Both carry slug (the interim key), a stored path, idx, attachable, owner_did, updated_at and deleted_at; document also mutable and data. A parent may be either kind (a group can sit in a folder), so parent_id is a plain column on both and the write path keeps it honest.
  • path is STORED and maintained on write, as the interim does with a trigger. turso has no recursive CTEs, and a stored path makes resolution one indexed lookup at any depth. A rename or a move rewrites the subtree's prefix. It is unique among live rows only, so a binned node does not hold its URL hostage.
  • The server picks the slug (name, then name-2, ...), because only it can see every sibling; the frontend finds collisions today by attempting inserts.
  • Kinds that are nodes in the frontend but have their own state and a URL of their own (polls, canvases) become document kinds with a side table keyed by the same id, when their milestone arrives. Speaker lists and the projector have no URL, so they are plain tables beside the tree (M5).
  • Who may create what under what is the interim's per-context template (context_permission_objects), as one static table: (kind, role, parents). Reading stays by membership (M2); this is the write model only.

The steps:

M4: membership and roster #

M5: meetings #

Speaker lists and the projector are not migrated, on purpose: a queue or a screen from a meeting that has ended is of no use to the next one, and a list is one click to make again.

M6: voting #

A poll is a node (a document of kind poll, so it has a URL, moves with its motion and goes to the bin with it) and a poll row of the same id (crates/appview/src/poll.rs, over crates/ballot-store).

Decided here without the owner, and cheap to change now:

  • A hidden tally stays hidden after the close, for everyone but the owners of the context, and the board with it. That is the interim's rule, and an election is where it is used: who won is announced, not by how much. The cost is that only the owners can recount such a poll. A voter can still check that their own ballot is there and says what they said.
  • Nobody joins a poll that is already open. Someone given voting rights after it opened votes in the next one. The interim checked at the moment of casting.
  • A poll with nobody to vote in it does not open (NoVoters), which is what a chair sees who forgot to hand out voting rights.

M7: live updates #

M8: blobs and the carried-over endpoints #

The interim sidecar's other endpoints #

Carried over one at a time. Handle typeahead is not among them: the browser asks Bluesky's public API itself. The steps:

M9: frontend swap, migration, cutover rehearsal #

Working rules #

  • cargo test --workspace in crates/ is the check. There is no CI job for it (crates/README.md says why), so run it before every commit.
  • Anything that stands in for real behaviour says so in its doc comment and is listed under "Findings" until it is replaced.
  • The frontend is not touched before M9 except to add to the seam, so the live app keeps shipping from main throughout.