diff --git a/text/0001-appview.md b/text/0001-appview.md new file mode 100644 --- /dev/null +++ b/text/0001-appview.md @@ -0,0 +1,526 @@ +# Summary + +Build starhaven.dev as an [atproto](https://overreacted.io/a-social-filesystem/) +appview with a community forum and mod browser. Users sign in with any atproto +account or sign up for a `*.starhaven.dev` handle via the Star Haven PDS. +Forum threads, replies, and mod listings are atproto records stored in users' +repos. The source lives at `starhaven.dev/appview`. + +In this RFC, "end-users" refers to people who play mods (as opposed to modders +who create them). Modders are also users of the site, but the distinction +matters for the mod browser and patching sections. + +# Status quo + +Currently, Star Haven is used as a central hub for Paper Mario modding, except +for when it is not and breaks off into smaller sub-communities (Modern Paper +Mario Modding, WUA, NicTube and Lily's mods, etc.) or when modders migrate to +larger platforms (Itch, or formerly romhacking.net). + +Star Haven facilitates general chat, private chats, individual mod and game +discussion, general modding assistance, mod release notifications and community +events. + +Some modders have the social capital to promote, release and discuss their mods +in their own bespoke communities and have no need for a centralised modding hub. +However, such modders are the exception rather than the rule. Initially, most +modders bounced around various forums until landing in Star Haven's Discord +server, where they continued to develop and release their mods. + +Star Haven's Discord Server holds the largest repository of modding advice, +through years of Discord chats and real-time assistance from experienced modders. +Star Haven also acts as an aggregation service for modders to publish their +creations to a wider audience with relative ease, providing either a thread or a +bespoke channel dependent on the mod's popularity. On top of that, Star Haven +aggregates and hosts modding tools, as well as similar help channels for said +tools. Star Haven also uses Discord and Itch.io to host regular jam events, +where modders create teams to create themed mods in a certain timeframe. + +Modders use GitHub to collaboratively develop their mods, private Discord +servers to facilitate development chats and voice-calls, and Itch.io to publish +their mods. Offline, modders use a wide variety of tools such as Star Rod, +Mamar, Unsimplifier and more. + +For players, Star Haven provides an avenue to discuss mods, the Paper Mario +games, ask for help on a per-mod or a general basis, or have a general chat. +Additionally, a list of mods is available in the Discord, and the +(currently broken) mod browser on starhaven.dev/mods. + +The community is splintered across multiple technologies and platforms that do +not neatly integrate with one another, limiting most modders' reach to a single +medium-sized Discord server. That, and everyone quite dislikes Discord. + +# Motivation + +We add Star Haven Forums (built atop atproto) as a slower-paced complement to +the Discord server, replace GitHub with Tangled, and add a mod browser at +starhaven.dev. Discord remains the place for real-time chat and voice calls - +the forums are for longer-form discussion, modding guides, and announcements +that benefit from being permanent and searchable. + +Critically, forums get important modding knowledge out of the walled garden +that is Discord and onto the open web, where search engines can index it. +Years of modding advice currently trapped in Discord messages becomes +discoverable by anyone with a web browser - no Discord account required. +The best place for agreed-upon knowledge is docs.starhaven.dev, but updating +that is out of scope for this RFC. The use of open-source, federated +technology also allows other communities to fork the work for their own +purposes. + +A non-Discord mod browser allows for much greater modder control of how their +mods are promoted and advertised, providing descriptions, updates and +promotional material all on the same page. Integration with PM-DX (and further +tools down the line) enhances the developer experience. + +For end-users, a modern, professional mod browser removes the reliance on +Discord for viewing, downloading and playing mods. + +## Why atproto? + +We already run a PDS at pds.starhaven.dev and a knot (git server) at +knot.starhaven.dev. Most community members still use GitHub, and some Star +Haven infrastructure (e.g. papermario-dx CI) still depends on it, but the +aspiration is to self-host community things over time. The question is whether +building on atproto is worth it versus a traditional forum with Discord OAuth +or similar. + +The core benefit is **data ownership**. Forum posts, mod listings, and profiles +are atproto records stored in users' own repos. If starhaven.dev goes down or +the maintainer disappears, users still have their data - they can take it to +another appview or export it. With a traditional forum, all content lives in +the site operator's database. The content may remain accessible, but users have +no independent copy and no way to move it. For a small community maintained by +volunteers, reducing the bus factor matters. + +Beyond that: + +- One identity system - no new accounts to make. A `*.starhaven.dev` handle + works on Bluesky, Tangled, and any future atproto service. +- We don't have to maintain a user/password system. atproto OAuth handles it. +- The PDS can support Discord OAuth via + [tranquil](https://tangled.org/tranquil.farm/tranquil-pds), so Discord users can + sign up without learning what atproto is. They get an atproto identity for + free. +- Other communities can run the same appview software against their own PDS and + get a fully independent instance with zero data migration needed. + +Existing forum software (Discourse, phpBB) doesn't really fit the vibe we want, +and we'd need to write a custom auth plugin for atproto OAuth anyway. + +# Explanation + +## Migration from current site + +starhaven.dev is currently a SvelteKit site on GitHub Pages (star-haven/starhaven.dev). +It has a homepage and a /mods page that renders a JSON file of mods with download +links. The /mods page is broken (500 error). The appview replaces this entirely - +we should not regress from the existing functionality, so the homepage and mod +listing need to be carried over. + +Migration is not about shutting Discord down. Forum-style channels (mod +releases, guides, showcase) move to the appview over time, while real-time +chat stays on Discord. Optionally, a #forums channel on Discord can post messages for new forum +threads via webhook to help with early adoption, though this should be +turned off once the forums have enough activity on their own. + +## How it works + +starhaven.dev is an atproto **appview** - it subscribes to the firehose, +indexes records that match our custom lexicons, and serves a web frontend. + +``` +browser --> starhaven.dev (appview + web server) --> database + | ^ + | | firehose + v | + relay / PDS +``` + +You sign in with atproto OAuth. When you create a forum post, it writes a +record to your PDS repo. The appview picks it up from the firehose and indexes +it. + +## Auth + +### Signing in + +`/login` asks for your handle (e.g. `luigi.starhaven.dev` or +`someone.bsky.social`) and initiates atproto OAuth. Same flow as tangled.sh. + +### Creating an account + +Users who don't have an atproto account can create one on pds.starhaven.dev. +`/signup` collects an email and a captcha, then calls the PDS admin API +(`com.atproto.server.createAccount`) to create the account. The user picks a +handle (`*.starhaven.dev`) and authenticates via passkey or Discord OAuth +(supported by the PDS via +[tranquil](https://tangled.org/tranquil.farm/tranquil-pds)). No passwords. + +## Routes + +``` +/ Homepage (community landing page) +/login atproto OAuth sign-in +/signup Create a *.starhaven.dev account + +/mods Mod browser (browse/search/filter) +/mods/:game Mods for a specific game +/mods/:game/:slug Mod page (description, media, downloads) +/mods/new Upload a new mod listing +/repo.json Homebrew App Store / Universal Updater compatible + endpoint for on-device mod browsers + +/search Search across threads, mods, and profiles + +/forums Forum index (categories/subcategories below) +/forums/:subcategory Thread list for a subcategory, paginated +/forums/:subcategory/new New thread form +/forums/thread/:id Thread view, paginated (?page=n) + +/:handle Actor profile (avatar, posts, mods) +``` + +Mods don't require a source repo - closed-source and legacy mods are welcome. +The mod browser is git forge agnostic; modders can use GitHub, Tangled, or +anything else. If a mod team happens to use Tangled, the appview can provide +tighter integration (linking to repos, showing recent commits), but it is not +required. + +## Lexicons + +Everything under `dev.starhaven.*`. Record types use `key: "tid"` unless +noted. References to other records use at-uri strings; references that need +to pin to an exact version (e.g. to prevent replying to a since-deleted +revision) use `com.atproto.repo.strongRef`. + +``` + ┌──────────────────┐ + │ actor.profile │ + │ key: self │ + └──────────────────┘ + +┌──────────────────┐ ┌──────────────────┐ +│ forum.thread │◄───│ forum.thread │ +│ │ │ .post │ +│ title, tags │ │ │ +│ subcategory │ │ body │ +└──────────────────┘ └──────────────────┘ + +┌──────────────────┐ ┌──────────────────┐ +│ mod.listing │◄───│ mod.release │ +│ │ │ │ +│ has: media[] │ │ has: deliverable│ +│ (PDS blob) │ │ (S3 URL) │ +└──────────────────┘ └──────────────────┘ +``` + +``` +dev.starhaven.actor.profile key: "literal:self" + displayName: string (maxGraphemes: 64) + description: string (maxGraphemes: 256) + pronouns: string (maxGraphemes: 20) + +dev.starhaven.forum.thread + title: string (required, maxGraphemes: 300) + subcategory: string (required, maxLength: 64) + tags: string[] (maxLength: 8, items maxLength: 64) + createdAt: datetime (required) + +dev.starhaven.forum.thread.post + thread: at-uri (required) → dev.starhaven.forum.thread + body: string (required, maxGraphemes: 30000) + createdAt: datetime (required) + + The "new thread" form creates both a thread and its initial post. If + the firehose delivers them out of order, the UI may briefly show a + thread without a body - this is acceptable. + +dev.starhaven.mod.listing + title: string (required, maxGraphemes: 300) + slug: string (required, maxLength: 64) URL-safe, globally unique, e.g. "master-quest" + description: string (required, maxGraphemes: 1000) + details: string (maxGraphemes: 30000) extended description (markdown) + game: string (required, maxLength: 32) fixed set, see below + platform: string (required, maxLength: 32) fixed set, see below + category: string (maxLength: 64) + tags: string[] (maxLength: 8, items maxLength: 64) + license: string (maxLength: 128) + media: #mediaItem[] (maxLength: 16) images and videos, ordered + createdAt: datetime (required) + +#mediaItem + file: blob (required, accept: image/*, video/*, maxSize: 50MB) + alt: string (maxGraphemes: 1000) accessibility description + +dev.starhaven.mod.release + mod: strongRef (required) → dev.starhaven.mod.listing + version: string (required, maxLength: 32) + changelog: string (maxGraphemes: 30000) + deliverable: string (required, format: uri) URL in object store + createdAt: datetime (required) +``` + +Valid `game` values: `64` (Paper Mario 64), `ttyd` (TTYD, GCN), `ttyd-r` +(TTYD remake, Switch), `spm` (Super Paper Mario), `ss` (Sticker Star), `cs` +(Color Splash), `tok` (Origami King). Valid `platform` values: `n64`, `gcn`, +`wii`, `3ds`, `wiiu`, `switch`. + +The appview enforces that only the listing author (or contributors listed on +the listing, if/when that field is added) can create `mod.release` records +pointing at a given `mod.listing`. + +Mod releases are separate records so they appear individually on the firehose +and don't require rewriting the listing on every update. The deliverable URL +points to the object store - the type of deliverable (patch file, zip, etc.) +depends on the game (see Mod browser below). Media blobs (screenshots, videos) +are stored on the PDS like any atproto blob. Mod deliverables are too large +for PDS blob storage, so they go in the S3 object store and are referenced by +URL. + +The `game` and `platform` fields are split so the `/repo.json` endpoint can +group mods by console (which the on-device app stores need) while the web mod +browser can filter by game. The `author` field isn't stored in the record - +the record's DID identifies the author, and the endpoint generator resolves it +to a display name. The Homebrew App Store `name` field (used for update +tracking) is derived from the DID + record TID, which is immutable. Fields +like `license`, `details`, and `changelog` (on releases) map directly to their +Homebrew App Store equivalents. + +## Tech + +- **Language:** Rust. +- **Frontend:** Server-rendered HTML + htmx (like tangled.sh). +- **Database:** SQLite (like tangled.sh). +- **Search:** SeekStorm (see Search section below). +- **Blob storage:** S3-compatible object store, configured via environment + variable (endpoint, bucket, credentials). Used for mod deliverables, media + (screenshots, videos), and any other uploaded files. The appview doesn't + assume a specific provider - any S3-compatible service (MinIO, Tigris, + Cloudflare R2, AWS S3, etc.) works. +- **Hosting:** kuribo, behind Caddy. +- **Repo:** starhaven.dev/appview on tangled. +- **Nix:** `flake.nix` outputs a NixOS module that starhaven.dev/infra + (kuribo) consumes as a flake input, plus a devshell for local development. + +## Forum categories + +Categories and subcategories are hardcoded in the appview source. Changing them +is rare enough that a code change and redeploy is fine. + +Initial structure (subject to change): + +- **About Star Haven** + - Announcements (locked: only admins/moderators can create threads) + - Suggestions +- **Making Mods** + - Help + - Collaboration +- **Mod Showcase** + - In Development + - Released Mods + +Threads are tagged with games (`64`, `ttyd`, etc.) using the same fixed set as +`mod.listing.game`, so any subcategory can be filtered by game. This avoids +duplicating categories per game and scales if new games get modding +communities. The `subcategory` field on `forum.thread` records stores the +subcategory slug, e.g. `making-mods/help`. + +## Visual design + +TBD, but the forum area should feel like a 2000s web forum while meeting modern +accessibility and mobile standards. Discourse is a good reference for how a +modern forum can work - the aesthetic is retro, the UX should not be. + +## Firehose and backfill + +The appview subscribes to the Bluesky relay's Jetstream for live events. + +Jetstream only provides live events and a short replay buffer (a couple of +days at most). A fresh database (new developer setup, or recovery from data +loss) needs to be seeded from PDSs directly using `com.atproto.sync.getRepo`, +which returns every record in a user's repo. + +For comparison, Tangled's appview does not do backfill - it connects to +Jetstream with a saved cursor and replays from there. This works for Tangled +because its knot servers are the source of truth for git data. For Star Haven, +the forum and mod records only exist in user repos, so backfill is necessary. + +On first startup with an empty database, the appview: + +1. Lists all repos on each configured PDS via `com.atproto.sync.listRepos`. + The default config includes pds.starhaven.dev. Scanning all of bsky.social + (millions of repos) is not practical, so bsky.social users need to have + interacted with the appview at least once (e.g. signed in) for their DID to + be known. The appview maintains a table of known DIDs and fetches those + repos individually. +2. For each known repo, calls `com.atproto.sync.getRepo` and processes every + `dev.starhaven.*` record as if it came from the firehose (inserting into + both SQLite and the search index). +3. Saves the current Jetstream cursor. +4. Switches to live firehose consumption from that cursor. + +This backfill runs automatically on first startup. + +## Search + +Search uses [SeekStorm](https://github.com/SeekStorm/SeekStorm), a Rust +full-text search crate that embeds in the appview binary. No separate service. + +The ingester writes to both SQLite and the SeekStorm index on every firehose +event (create, update, delete), so records are searchable immediately. + +Indexed content: +- Forum threads and posts (title, body). Filterable by category, tags. +- Mod listings (title, description, details). Filterable by game, platform, + category. +- Actor profiles (display name, description). + +`/search` is a single search bar across all record types. Facet filters let +you narrow by type (thread, mod, profile), game, platform, or category. +Results ranked by BM25F. + +SeekStorm has built-in typo tolerance (edit distance 1, so "stikcer star" +finds "Sticker Star") and query auto-completion. + +## Mod browser + +The mod browser is the primary way to discover and download mods. Modders sign +in and create a `dev.starhaven.mod.listing` record with a title, description, +media, and one or more versioned releases. The appview indexes these from the +firehose like any other record. Mod deliverables (patches, zips, etc.) are +uploaded through the appview web UI, which stores them in the S3-compatible +object store. For games that need patching, the patching pipeline runs in the +browser. + +### Patching + +Paper Mario mods can't just ship a ready-to-play ROM or ISO - that would +require distributing copyrighted game data. Instead, mods are distributed as +patches or file overlays that are applied to a user-supplied copy of the +original game. The mod browser handles all of this transparently: end-users +provide their base ROM/ISO once (stored in the browser's IndexedDB for reuse) +and then just click "download" on any mod. No separate patching program +required. The patching strategy varies by game: + +**PM64 (N64)** - Mods are distributed as `.bps` patch files targeting the US +ROM (the only supported base for now). The `.bps` patch is fetched, applied to +the base ROM entirely client-side using JavaScript, and the patched ROM is +downloaded to the end-user's machine. No game data ever leaves the browser or +touches the server. + +**TTYD (GCN)** - Patching strategy TBD. Needs input from existing GameCube +modders. + +**SPM (Wii)** - Like PM64, the end-user supplies a base `.iso` or `.wbfs`. +There are multiple regional revisions (3 US, +2 JP, ~1 PAL) but mods ship a single patch - the tooling uses symbol-based +replacement so it handles revision differences automatically. The existing +Flipside Mod Manager uses the C libraries +[wiimms-szs-tools](https://github.com/Wiimm/wiimms-szs-tools) and +[wiimms-iso-tools](https://github.com/Wiimm/wiimms-iso-tools) to apply +patches. Rather than porting these to JavaScript, the patching runs client-side +inside CheerpX (WebVM), a browser-based x86 virtual machine. The output is an +`.iso` (WBFS filesystem) that works with both Dolphin and USB loaders on real +hardware. As with PM64, no game data leaves the browser. + +**Modern PM (3DS, Wii U, Switch)** - Mods are zip files containing replacement +assets that get extracted to console-specific paths (e.g. +`sdmc:/luma/titles//romfs/` on 3DS). These don't require patching +against a base ROM - the zip is the deliverable. Modders upload the zip to +starhaven.dev and end-users download it directly, either through the web mod +browser or through on-device homebrew app stores (Universal Updater on 3DS, +Homebrew App Store on Wii U/Switch) via the `/repo.json` compatibility endpoint. + +### `/repo.json` compatibility endpoint + +The 3DS/Wii U/Switch modding communities already use on-device homebrew app +stores that read from a `repo.json` file listing available mods. starhaven.dev +serves a `/repo.json` endpoint that is compatible with these formats, generated +from the indexed `dev.starhaven.mod.listing` and `dev.starhaven.mod.release` +records. This means existing on-device mod browsers continue to work - they +just point at starhaven.dev instead of a GitHub repo. + +The 3DS community uses Universal Updater, which reads a `.unistore` file (a +JSON format with a different extension). The Wii U and Switch communities use +the Homebrew App Store, which reads `repos.json`. starhaven.dev generates both +formats. + +The mapping from lexicon records to these formats: +- DID + record TID → `name` (Homebrew App Store package identifier, stable) +- `title` → `title` / unistore `info.title` +- DID → resolved to `author` +- `description` → `description`, `details` → `details` +- `license` → `license` +- `category` → `category` / unistore `info.category` +- `platform` → unistore `info.console` +- Latest `mod.release.version` → `version` / unistore `info.version` +- Latest `mod.release.changelog` → `changelog` +- Latest `mod.release.deliverable` → download URL +- `media` → filtered to images only (on-device app stores don't support video) + +For the unistore format, the appview also generates per-region install scripts +(download, extract, cleanup) using a lookup table of title IDs per game per +region (e.g. `00040000000A5E00` for PM:SS USA). This is appview logic, not +stored in the lexicon. + + +## Moderation + +Moderation is appview-level: it controls what the site displays, not what +exists in user repos. Users' atproto data is theirs regardless of moderation +actions. + +Roles: +- **Admin** - full control. Can manage moderators, edit site config, post in + locked categories (e.g. Announcements). +- **Moderator** - can remove content from the index, lock/move threads, ban + handles, pin/feature content. +- **User** - default role for signed-in users. + +Moderation state (bans, removed posts, role assignments) is stored in the +appview database, not in atproto records. A banned user's records still exist +in their repo but are hidden from the site. + +# Drawbacks + +- It's a lot of work compared to just deploying Discourse. +- We'd be maintaining custom forum software forever. +- The PM modding community is small - might be overkill. +- atproto custom lexicons are still early days, things might change. +- atproto is fully public. Modders who want to work in secret (e.g. for mod + jams) cannot use Tangled private repos yet. Until atproto adds permissioned + data, private work should stay on whatever git forge the modder prefers. +- If users delete records from their repos, forum threads get holes. + +# Alternatives + +**Discourse/phpBB with atproto auth plugin** - still need a custom auth +plugin, doesn't give us the retro look without heavy theming, forum data +isn't on atproto. + +**Don't do this** - keep everything on Discord. Works for real-time chat but +important knowledge stays trapped behind a login wall, unsearchable by search +engines and unarchivable. + +# Future Work + +- Integration with tangled, so that mod source repos can be linked from listings. +- Build integration with papermario-dx CI. +- Mod reviews with a `dev.starhaven.mod.review` lexicon. +- Reactions with a `dev.starhaven.feed.reaction` lexicon (applicable to threads, + replies, mods, etc.) +- Multi-region PM64 patch support via a custom patch format for papermario-dx. +- Mod collections and events (jams, curated sets like Master Quest, Paper Dank + Rave, the Holiday Specials). +- Additional PDS auth providers beyond passkeys and Discord (e.g. GitHub OAuth). + +# Unresolved questions + +- Rich text: facets or Markdown? Markdown is familiar, facets are atproto-native. +- Thread ordering: chronological or upvote-based? +- Notifications: atproto has no push for custom lexicons. Email? Bluesky DMs? + Just check the site? +- TTYD (GCN) patching strategy: needs input from existing GameCube modders. +- Lexicon namespacing: before 1.0, ensure lexicons aren't overly specific to + Paper Mario if the platform may open to other modding communities in future. +- Homepage design: what should it look like? The current one isn't great.