diff --git a/README.md b/README.md index e7922fc..4d4fba6 100644 --- a/README.md +++ b/README.md @@ -1,58 +1,13 @@ # Malfestio -Malfestio is a learning OS: flashcards + notes + lectures + articles, designed for daily study. +Malfestio is a social learning platform: flashcards + notes + lectures + articles, designed for daily study. Social layer: publish/share/remix learning artifacts; follow curators; discuss. -## Personas +## Documentation -- **Learner**: studies daily; imports content; wants fast "review queue". -- **Creator**: makes decks/notes; publishes updates; wants feedback + forks. -- **Curator/Teacher**: bundles content into learning paths; annotates lectures/articles. -- **Moderator/Community admin**: handles reports, takedowns, spam. - -## Principles - -- Local-first study experience; offline study must not feel "second-class". -- Shareable artifacts are portable: Lexicon-defined schemas + stable IDs. -- Privacy by design: progress + recall history are private unless explicitly - shared. - -### Data Model - -- Note: markdown + structure + citations + links to sources. -- Card: front/back (+ optional cloze, audio, image, code block). -- Deck: ordered/clustered cards (+ metadata, tags). -- Lecture: external URL + outline + timestamps + linked notes/cards. -- Article: URL + extracted text (readability style heuristics) + highlights + - linked notes/cards. -- Collection/Path: curated bundle of decks + notes + sources. - -## System Architecture - -### Frontend (SolidJS) - -- App shell + router-driven workspaces (Library / Study / Create / Social). -- Signals as primary state primitive; keep study session state in signals/store. - -### Backend (Rust) - -- Axum API gateway: REST/XRPC-ish endpoints, tower middleware, typed extractors. -- Services (logical, not necessarily microservices): - - Identity/Auth service (local + optional ATProto OAuth integration) - - Content service (notes/cards/decks/sources) - - Study service (queue generation + grading + scheduling) - - Social service (follows, feeds, comments, notifications) - - Search service (indexing + query) - - Moderation service (reports, takedowns, rules) - -### Storage - -- Postgres: canonical app DB (users, private study state, cache of published records). -- Object storage: images/audio, extracted article snapshots (if you store them). -- Search index: separate system (Meilisearch/Typesense/ZincSearch-pick one later). - -### Eventing - -- Internal outbox pattern (DB table) for: - - reindex jobs, notification fanout, federation publish steps +- [Personas & Principles](./docs/personas.md) – Target users and design philosophy +- [Architecture](./docs/architecture.md) – System components and data model +- [Information Architecture](./docs/information-architecture.md) – Navigation and URL structure +- [Data Model Mapping](./docs/data-model-mapping.md) – Lexicon to database mapping +- [Roadmap](./docs/todo.md) – Development milestones diff --git a/docs/architecture.md b/docs/architecture.md new file mode 100644 index 0000000..2664fa0 --- /dev/null +++ b/docs/architecture.md @@ -0,0 +1,43 @@ +# System Architecture + +This document describes the technical architecture of Malfestio. + +## Frontend (SolidJS) + +- App shell + router-driven workspaces (Library / Study / Create / Social). +- Signals as primary state primitive; keep study session state in signals/store. + +## Backend (Rust) + +- Axum API gateway: REST/XRPC-ish endpoints, tower middleware, typed extractors. +- Services (logical, not necessarily microservices): + - Identity/Auth service (local + optional ATProto OAuth integration) + - Content service (notes/cards/decks/sources) + - Study service (queue generation + grading + scheduling) + - Social service (follows, feeds, comments, notifications) + - Search service (indexing + query) + - Moderation service (reports, takedowns, rules) + +## Storage + +- Postgres: canonical app DB (users, private study state, cache of published records). +- Object storage: images/audio, extracted article snapshots (if you store them). +- Search index: separate system (Meilisearch/Typesense/ZincSearch-pick one later). + +## Eventing + +- Internal outbox pattern (DB table) for: + - reindex jobs, notification fanout, federation publish steps + +## Data Model + +See [Data Model Mapping](./data-model-mapping.md) for the mapping between public Lexicon records and internal database tables. + +### Core Entities + +- **Note**: markdown + structure + citations + links to sources. +- **Card**: front/back (+ optional cloze, audio, image, code block). +- **Deck**: ordered/clustered cards (+ metadata, tags). +- **Lecture**: external URL + outline + timestamps + linked notes/cards. +- **Article**: URL + extracted text (readability style heuristics) + highlights + linked notes/cards. +- **Collection/Path**: curated bundle of decks + notes + sources. diff --git a/docs/at-notes.md b/docs/at-notes.md index c25d58d..0585a15 100644 --- a/docs/at-notes.md +++ b/docs/at-notes.md @@ -1,5 +1,7 @@ # AT Protocol Research Notes +Reference material for AT Protocol integration. For implementation details, see [todo.md](todo.md). + ## OAuth 2.1 Specification AT Protocol uses a specific profile of OAuth 2.1 for client↔PDS authorization. @@ -44,21 +46,22 @@ Format: `at:////` Example: `at://did:plc:abc123/app.malfestio.deck/3k5abc123` -## Firehose Consumption +## Firehose / Jetstream -For social features (trending, discovery, feeds): +### Raw Firehose -- **WebSocket Connection**: Subscribe to `com.atproto.sync.subscribeRepos` from a Relay -- **CBOR Decoding**: Parse incoming events (or use Jetstream for JSON) +- **WebSocket**: Subscribe to `com.atproto.sync.subscribeRepos` from a Relay +- **CBOR Decoding**: Parse incoming events - **Cursor Management**: Track position for reconnection -## AppView Pattern +### Jetstream (Recommended) -Index network-wide records to power discovery features: +Bluesky's simplified JSON firehose: -- Index `app.malfestio.*` records from firehose -- Implement `getFeedSkeleton` for custom algorithmic feeds -- Hydration service combines skeletons with full content from PDSes +- JSON format (no CBOR decoding) +- Reduced bandwidth (zstd compression) +- Collection/repo filtering at source +- Simpler reconnection with cursors ## Well-Known Endpoints @@ -66,6 +69,46 @@ Index network-wide records to power discovery features: - `/.well-known/oauth-protected-resource` — PDS OAuth metadata - `/.well-known/oauth-authorization-server` — Auth server metadata +## Labelers + +**Architecture:** + +1. Labels = metadata (source DID + subject AT-URI + value string) +2. User Subscription = users subscribe to labelers; clients include in API requests +3. Label Interpretation = per-user config to hide, warn, or ignore content + +**Structure:** + +```json +{ + "src": "did:plc:labeler", + "uri": "at://did:user/app.bsky.feed.post/123", + "val": "spam", + "cts": "2026-01-01T00:00:00Z" +} +``` + +## Feeds + +**Core Flow**: + +1. User requests feed via at-uri of declared feed +2. PDS resolves at-uri → Feed Generator's DID doc +3. PDS sends `getFeedSkeleton` to service endpoint (authenticated by user's JWT) +4. Feed Generator returns skeleton (list of post URIs + cursor) +5. PDS hydrates skeleton with full content (via AppView) +6. Hydrated feed returned to user + +## AppView + +**Responsibilities**: + +1. Record Processing & Indexing - consume firehose, build indices for likes, threads, follows +2. Moderation Enforcement - apply labels from subscribed labelers +3. Query Interface - expose XRPC API (proxied through PDS) +4. Media CDN - fetch/cache blobs from upstream PDSes, generate thumbnails +5. Search & Discovery - full-text search, type-ahead, content ranking + ## Patterns from Real AT Protocol Apps ### plyr.fm (Music) @@ -73,13 +116,11 @@ Index network-wide records to power discovery features: - OAuth 2.1 via `@atproto/oauth-client` library - Records synced to PDS: tracks, likes, playlists - Separate moderation service (Rust labeler) -- Data ownership: "tracks, likes, playlists synced to your PDS as ATProto records" ### leaflet.pub (Writing) - React/Next.js frontend with Supabase + Replicache for sync - Bluesky integration via dedicated `lexicons/` and `appview/` directories -- Publications posted to Bluesky ### wisp.place (Static Sites) @@ -101,3 +142,6 @@ Index network-wide records to power discovery features: - [Repository & XRPC](https://atproto.com/specs/xrpc) - [Feed Generator Starter Kit](https://github.com/bluesky-social/feed-generator) - [atproto TypeScript SDK](https://github.com/bluesky-social/atproto) +- [Ozone Moderation Service](https://github.com/bluesky-social/ozone) +- [Jetstream Firehose](https://docs.bsky.app/blog/jetstream) +- [Labels and Moderation Guide](https://docs.bsky.app/docs/advanced-guides/moderation) diff --git a/docs/personas.md b/docs/personas.md new file mode 100644 index 0000000..c5a5550 --- /dev/null +++ b/docs/personas.md @@ -0,0 +1,14 @@ +# Personas & Principles + +## Personas + +- **Learner**: studies daily; imports content; wants fast "review queue". +- **Creator**: makes decks/notes; publishes updates; wants feedback + forks. +- **Curator/Teacher**: bundles content into learning paths; annotates lectures/articles. +- **Moderator/Community admin**: handles reports, takedowns, spam. + +## Design Principles + +- Local-first study experience; offline study must not feel "second-class". +- Shareable artifacts are portable: Lexicon-defined schemas + stable IDs. +- Privacy by design: progress + recall history are private unless explicitly shared. diff --git a/docs/todo.md b/docs/todo.md index 165bc08..5b792bb 100644 --- a/docs/todo.md +++ b/docs/todo.md @@ -21,7 +21,7 @@ - Lexicon defines record types + XRPC endpoints; JSON-schema-like constraints. - Use "optional fields" heavily; avoid enums that will calcify the product too early. -- Versioning: add fields, don’t rename; never rely on being able to rewrite history. +- Versioning: add fields, don't rename; never rely on being able to rewrite history. ### Schema boundaries (important) @@ -45,64 +45,255 @@ - **(Done) Milestone D**: Identity + Permissions + Publishing Model. - Auth MVP, Permission model (Private/Public/SharedWith), and basic Publishing flow implemented. - Backend API and Frontend Editor updated with tests covering permissions and publishing. +- **(Done) Milestone E**: Internal component library/UI Foundation + Animations. - **(Done) Milestone F**: OAuth + PDS Record Publishing. - OAuth 2.1 client flow (PKCE, DPoP, handle/DID resolution, token refresh). - PDS client for `putRecord`, `deleteRecord`, `uploadBlob`. - TID generation and AT-URI builder in core crate. -- **(Done) Milestone E**: Internal component library/UI Foundation + Animations. -- **(Done) Milestone F**: Content Authoring (Notes + Cards + Deck Builder). -- **(Done) Milestone G**: Study Engine (SRS) + Daily Review UX. +- **(Done) Milestone G**: Content Authoring (Notes + Cards + Deck Builder). +- **(Done) Milestone H**: Study Engine (SRS) + Daily Review UX. - SM-2 spaced repetition scheduler. -- **(Done) Milestone H**: Social Layer v1: Follow graph, Feeds (Follows/Trending), Forking workflow, and Threaded comments. -- **(Done) Milestone I**: Search + Discovery + Taxonomy. +- **(Done) Milestone I**: Social Layer v1: Follow graph, Feeds (Follows/Trending), Forking workflow, and Threaded comments. - Full-text search with pg_trgm/unaccent, visibility filtering, and unified search index. - Tag taxonomy and Discovery page with top tags. -### Milestone J - Moderation + Abuse Resistance +### Milestone J - Static Content & Landing Page #### Deliverables -- Look into [Ozone](https://github.com/bluesky-social/ozone) -- Reporting pipeline + review queue -- Rate limits + spam heuristics -- Takedown/visibility states (shadowed, removed, quarantined) -- Audit logging for moderation actions +**Marketing/Static Site:** + +- [ ] Landing page with hero section, feature highlights, social proof + - Hero should have a graph paper/grid background + - Floating flash cards, notes +- [ ] "How it works" section with study flow visualization +- [ ] About page with team/mission + +**App Vision Content:** + +- [ ] Onboarding flow with persona selection (Learner/Creator/Curator) +- [ ] Empty states with helpful prompts for new users +- [ ] Tutorial/walkthrough for first deck creation +- [ ] Help center or FAQ section -> Should mention that the app is still in development and subject to change. + +**SEO & Meta:** + +- [ ] Open Graph / Twitter Card meta tags + - [ ] Scripted with HTML2Canvas/PNG generation + - Graph paper background + - Floating flash cards, notes +- [ ] Sitemap.xml generation +- [ ] robots.txt configuration #### Acceptance -- You can safely operate an open publishing surface. +- New visitors understand the value proposition within 10 seconds. +- Onboarding flow guides users to create their first deck. + +### Milestone K - AppView Indexing + +#### Deliverables + +**Firehose Enhancement:** + +- [ ] Upgrade firehose consumer to store full record content (not just metadata) +- [ ] Add `indexed_decks`, `indexed_cards`, `indexed_notes` tables for remote content +- [ ] Track latest processed revision per repo; handle deletions + +**Search & Discovery:** + +- [ ] Implement search over indexed remote records (extend `search_items` view) +- [ ] User profile aggregation: follower counts, deck counts from federated sources + +**Export & Interop:** + +- [ ] Export local records as valid Lexicon JSON (`/api/export/:collection`) +- [ ] Read-only "federated library" view showing remote decks + +#### Acceptance + +- A deck published from Malfestio can be discovered via another AT Protocol client. +- Remote decks from followed users appear in search results. + +### Milestone L - ATProto Integration Pass + +#### Deliverables + +**Identity & Auth:** + +- [ ] OAuth login directly to user's PDS (vs. local-only auth) +- [ ] Handle resolution via DNS TXT or `/.well-known/atproto-did` +- [ ] DPoP token binding for secure API calls + +**Sync & Conflict Resolution:** + +- [ ] Bi-directional sync: local drafts → PDS records, PDS records → local cache +- [ ] Conflict resolution strategy for concurrent edits (last-write-wins or merge UI) +- [ ] Offline queue for pending publishes + +**Deep Linking:** + +- [ ] AT-URI deep linking from external clients +- [ ] Handle `at://` URL scheme in app + +#### Acceptance + +- User can log in with their existing Bluesky/PDS identity. +- Local drafts sync correctly after reconnecting. + +#### Implementation Details + +**Considerations:** + +- Scalability: substantial compute; caching, DB optimization, distributed processing +- Lexicon Validation: validate schemas, ignore invalid records gracefully +- Account State: track latest processed revision per repo; handle deletions +- Bluesky's AppView uses PostgreSQL or ScyllaDB + image proxy + AppView core + +**Identity:** + +- Use `did:web` for simplicity, `did:plc` for long-term stability +- ATProto OAuth is the forward path -### Milestone K - Federation / ATProto Integration Pass +### Milestone M - Custom Feed Generator #### Deliverables -- Phase 1 (minimum): - - export Lexicon records - - ingest remote records into a read-only "federated library" -- Phase 2: - - OAuth login to PDS + publish records directly (client or server mediated) - - reconcile local drafts with remote published state +**Infrastructure:** + +- [ ] Feed Generator service with `did:web` identity +- [ ] Publish `app.bsky.feed.generator` declaration record to creator's repo +- [ ] DID document with service endpoint for feed requests + +**Endpoints:** + +- [ ] `app.bsky.feed.getFeedSkeleton` - Return post URIs + cursor for pagination +- [ ] `app.bsky.feed.describeFeedGenerator` - Feed metadata (DID, name, description) +- [ ] JWT authentication for user-personalized feeds + +**Algorithms:** + +- [ ] "Trending Decks" - Top decks by fork/like count in last 7 days +- [ ] "New from Following" - Latest decks from followed creators +- [ ] "Study Streak Leaders" - Decks with highest completion rates (anonymized) + +**Indexing:** + +- [ ] Subscribe to `com.atproto.sync.subscribeRepos` (or Jetstream) for `app.malfestio.*` records +- [ ] Index posts with compound cursor (timestamp::CID) for deterministic pagination +- [ ] Garbage collect indexed data older than 48 hours (except pinned content) #### Acceptance -- A published artifact is portable beyond your app. +- Custom feed appears in Bluesky and other AT Protocol clients. +- Feed surfaces relevant learning content based on engagement signals. +- Pagination works correctly across feed refreshes. + +#### Implementation Details + +**Core Flow:** see [AT Notes](./at-notes.md#feeds) + +**Skeleton Response Format:** -#### Notes +```json +{ + "feed": [ + {"post": "at://did:example/app.bsky.feed.post/1"}, + {"post": "at://did:example/app.bsky.feed.post/2"} + ], + "cursor": "1683654690921::bafyrei..." +} +``` -- ATProto OAuth is the forward path; plan on it. -- XRPC endpoint patterns and legacy session behavior exist, but treat them as transitional. +**Skeleton Metadata Types:** -### Milestone L - Reliability, Observability, Launch +```typescript +type SkeletonItem = { + post: string // post URI + reason?: Reason // optional context (e.g., repost) +} +type ReasonRepost = { + $type: 'app.bsky.feed.defs#skeletonReasonRepost' + repost: string // repost URI +} +``` + +**Considerations:** + +- Validate user JWTs if feed depends on user state (follows, likes) +- Use compound cursor (timestamp::CID) for deterministic pagination +- Most feeds can garbage collect data older than 48 hours +- Reference: [Feed Generator Starter Kit](https://github.com/bluesky-social/feed-generator) + +### Milestone N - Reliability, Observability, Launch #### Deliverables -- Metrics + tracing + structured logs -- Backups + restore drills -- Load test targets (study session + feed + search) -- Beta program + feedback loop + roadmap iteration +**Observability:** + +- [ ] Structured logging with correlation IDs +- [ ] Metrics collection (Prometheus/OpenTelemetry) +- [ ] Distributed tracing for request flows +- [ ] Error tracking (Sentry or similar) + +**Reliability:** + +- [ ] Database backups + restore drills +- [ ] Health check endpoints (`/health`, `/ready`) +- [ ] Graceful shutdown handling +- [ ] Circuit breakers for external dependencies + +**Load Testing:** + +- [ ] Study session throughput targets +- [ ] Feed generation latency benchmarks +- [ ] Search query performance under load + +**Launch Prep:** + +- [ ] Beta program signup flow +- [ ] Feedback collection mechanism +- [ ] Feature flags for gradual rollout + +#### Acceptance + +- System handles 10x expected load without degradation. +- Mean time to recovery < 5 minutes for common failures. + +### Milestone O - Moderation + Abuse Resistance + +#### Deliverables + +**Labeler Infrastructure:** + +- [ ] Dedicated Bluesky account for labeler service +- [ ] Publish `app.bsky.labeler.service` record to make discoverable +- [ ] Self-host Ozone backend + UI (Docker setup in `HOSTING.md`) +- [ ] Configure report types via `goat` CLI + +**Endpoints:** + +- [ ] `com.atproto.label.subscribeLabels` - real-time label stream +- [ ] `com.atproto.label.queryLabels` - query published labels +- [ ] `com.atproto.report.createReport` - accept user reports + +**Moderation Features:** + +- [ ] Reporting pipeline + review queue UI +- [ ] Rate limits + spam heuristics +- [ ] Takedown/visibility states (shadowed, removed, quarantined) +- [ ] Audit logging for moderation actions + +#### Acceptance + +- You can safely operate an open publishing surface. +- Users can subscribe to your labeler and see moderation applied. + +**Reference:** [Ozone Moderation Service](https://github.com/bluesky-social/ozone) -## Open Questions (Parked Decisions) +## Open Question/Parked Decisions -- Local-first mechanics: full offline authoring + later publish, or online-only creation? -- Federation depth: read-only ingest first, or publish-to-PDS in the first public beta? -- Content extraction: store extracted article snapshots (legal/ops implications), or store only metadata + highlights? +- Full offline authoring + later publish +- Federation depth: publish-to-PDS in the first public beta +- Content extraction: store extracted article snapshots locally (browser) + - Persist only metadata + highlights