diff --git a/docs/astro.config.mjs b/docs/astro.config.mjs index 6de2b4b..1a5021b 100644 --- a/docs/astro.config.mjs +++ b/docs/astro.config.mjs @@ -25,6 +25,7 @@ export default defineConfig({ { label: 'Guides', items: [ + { label: 'Architecture', slug: 'architecture' }, { label: 'Self-Hosting', slug: 'deployment' }, { label: 'Monitoring & Metrics', slug: 'monitoring' }, { label: 'Redirects & Rewrites', slug: 'redirects' }, diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md new file mode 100644 index 0000000..acaad8b --- /dev/null +++ b/docs/src/content/docs/architecture.md @@ -0,0 +1,211 @@ +--- +title: Architecture Guide +description: How the hosting service, firehose service, and tiered storage work together +--- + +Wisp.place's serving infrastructure is split into two microservices: the **firehose service** (write path) and the **hosting service** (read path). They communicate through S3-compatible storage and Redis pub/sub. + +## Service Overview + +### Firehose Service + +The firehose service watches the AT Protocol firehose (Jetstream WebSocket) for `place.wisp.fs` and `place.wisp.settings` record changes. When a site is created, updated, or deleted, it: + +1. Downloads all blobs from the user's PDS +2. Decompresses gzipped content +3. Rewrites HTML for subdirectory serving (absolute paths become relative) +4. Writes the processed files to S3 (or disk) +5. Publishes a cache invalidation event to Redis + +The firehose service is **write-only** — it never serves requests to end users. + +**Key configuration:** + +```bash +# Firehose connection +FIREHOSE_URL="wss://jetstream2.us-east.bsky.network/subscribe" + +# S3 storage (recommended for production) +S3_BUCKET="wisp-sites" +S3_REGION="auto" +S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com" +S3_ACCESS_KEY_ID="..." +S3_SECRET_ACCESS_KEY="..." + +# Redis for cache invalidation +REDIS_URL="redis://localhost:6379" + +# Concurrency control +FIREHOSE_CONCURRENCY=5 # Max parallel event processing +``` + +**Backfill mode:** Start with `--backfill` to do a one-time bulk sync of all existing sites from the database into the cache. + +### Hosting Service + +The hosting service is a **read-only** CDN built with Node.js and Hono. It serves static files from a three-tier cache and handles routing for custom domains, wisp subdomains, and direct URLs. + +On each request, the hosting service: + +1. Resolves the site from the request hostname/path +2. Looks up the file in tiered storage (hot → warm → cold) +3. On a cache miss, fetches from the PDS on-demand and populates the cache +4. Applies HTML path rewriting if serving from a subdirectory +5. Processes `_redirects` rules +6. Serves the file with appropriate headers + +The hosting service subscribes to Redis pub/sub for cache invalidation messages from the firehose service. When it receives an invalidation, it evicts the affected entries from its hot and warm tiers so the next request fetches fresh content. + +## Tiered Storage + +The `@wispplace/tiered-storage` package implements a three-tier cascading cache. Data flows **down** on writes and is looked up **upward** on reads. + +``` +Read path: Hot (memory) → Warm (disk) → Cold (S3/disk) +Write path: Hot ← Warm ← Cold (writes cascade down through all tiers) +``` + +### Hot Tier (Memory) + +- **Implementation:** In-memory LRU cache +- **Eviction:** Size-based (bytes) and count-based (max items) +- **Use case:** Frequently accessed files (index.html, CSS, JS) +- **Lost on restart** — repopulated from warm/cold tiers on access + +```bash +HOT_CACHE_SIZE=104857600 # 100 MB (default) +HOT_CACHE_COUNT=500 # Max items +``` + +### Warm Tier (Disk) + +- **Implementation:** Filesystem with human-readable paths +- **Eviction:** Configurable — `lru` (default), `fifo`, or `size` +- **Structure:** `cache/sites/{did}/{sitename}/path/to/file` +- **Survives restarts** — provides fast local reads without network calls + +```bash +WARM_CACHE_SIZE=10737418240 # 10 GB (default) +WARM_EVICTION_POLICY=lru # lru, fifo, or size +CACHE_DIR=./cache/sites +``` + +The warm tier is optional when S3 is configured. Without S3, disk acts as the cold (source of truth) tier. + +### Cold Tier (S3 or Disk) + +- **With S3:** The firehose service writes here; the hosting service reads (read-only wrapper) +- **Without S3:** A disk-based tier serves as both warm and cold +- **Compatible with:** Cloudflare R2, MinIO, AWS S3, or any S3-compatible endpoint + +```bash +S3_BUCKET="wisp-sites" +S3_REGION="auto" +S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com" +S3_ACCESS_KEY_ID="..." +S3_SECRET_ACCESS_KEY="..." +S3_METADATA_BUCKET="wisp-metadata" # Optional, recommended for production +``` + +### Tier Placement Rules + +Not all files are placed on every tier. The hosting service uses placement rules to keep the hot tier efficient: + +| File Pattern | Tiers | Rationale | +|---|---|---| +| `index.html`, `*.css`, `*.js` | Hot, Warm, Cold | Critical for page loads | +| Rewritten HTML (`.rewritten/`) | Hot, Warm, Cold | Pre-processed for fast serving | +| Images, fonts, media (`*.jpg`, `*.woff2`, etc.) | Warm, Cold | Already compressed, large — skip memory | +| Everything else | Warm, Cold | Default placement | + +### Promotion and Bootstrap + +When a file is found in a lower tier but not a higher one, it's **eagerly promoted** upward. For example, a cache miss on hot that hits warm will copy the file into hot for future requests. + +On startup, the hosting service can **bootstrap** tiers: +- Hot bootstraps from warm by loading the most-accessed items +- Warm bootstraps from cold by loading recently written items + +## Cache Invalidation + +The firehose service and hosting service communicate through Redis pub/sub: + +``` +Firehose Service Hosting Service + │ │ + │ ── Redis pub/sub ──────────────→ │ + │ (wisp:revalidate) │ + │ │ + │ Site updated/deleted: │ Receives invalidation: + │ 1. Write new files to S3 │ 1. Evict from hot tier + │ 2. Publish invalidation │ 2. Evict from warm tier + │ │ 3. Next request fetches fresh +``` + +If Redis is not configured, the hosting service still works — it just won't receive real-time invalidation and will rely on TTL-based expiry (default 14 days) and on-demand fetching. + +## On-Demand Cache Population + +When the hosting service receives a request for a site that isn't in any cache tier, it fetches directly from the user's PDS: + +1. Resolves the user's DID to their PDS endpoint +2. Downloads the `place.wisp.fs` record +3. Fetches the requested blob +4. Decompresses and processes the file +5. Stores it in the appropriate tiers based on placement rules +6. Serves the response + +This means the hosting service works even without the firehose service running — it just won't have pre-populated caches. + +## Deployment Scenarios + +### Minimal (Disk Only) + +No S3 or Redis required. The hosting service uses disk as both warm and cold tier. Best for small deployments or development. + +```bash +# Hosting service only +CACHE_DIR=./cache/sites +HOT_CACHE_SIZE=104857600 +``` + +### Production (S3 + Redis) + +The firehose service pre-populates S3 and notifies the hosting service of changes via Redis. Multiple hosting service instances can share the same S3 backend. + +```bash +# Both services +S3_BUCKET=wisp-sites +S3_ENDPOINT=https://account.r2.cloudflarestorage.com +REDIS_URL=redis://localhost:6379 + +# Hosting service +HOT_CACHE_SIZE=104857600 +WARM_CACHE_SIZE=10737418240 +``` + +### Scaled (Multiple Hosting Instances) + +Run multiple hosting service instances behind a load balancer. Each has its own hot and warm tiers, but they share the S3 cold tier and receive the same Redis invalidation events. + +``` + Load Balancer + / | \ + Hosting-1 Hosting-2 Hosting-3 + (hot+warm) (hot+warm) (hot+warm) + \ | / + S3 (cold tier) + | + Firehose Service +``` + +## Observability + +Both services expose internal observability endpoints: + +- `/__internal__/observability/logs` — Recent log entries +- `/__internal__/observability/errors` — Error log entries +- `/__internal__/observability/metrics` — Prometheus-format metrics +- `/__internal__/observability/cache` — Cache tier statistics (hosting service only) + +See [Monitoring & Metrics](/monitoring) for Grafana integration details. diff --git a/docs/src/content/docs/deployment.md b/docs/src/content/docs/deployment.md index 9cd1dec..6e2c00d 100644 --- a/docs/src/content/docs/deployment.md +++ b/docs/src/content/docs/deployment.md @@ -3,35 +3,38 @@ title: Self-Hosting Guide description: Deploy your own Wisp.place instance --- -This guide covers deploying your own Wisp.place instance. Wisp.place consists of two services: the main backend (handles OAuth, uploads, domains) and the hosting service (serves cached sites). +This guide covers deploying your own Wisp.place instance. Wisp.place consists of three services: the main backend (handles OAuth, uploads, domains), the firehose service (watches the AT Protocol firehose and populates the cache), and the hosting service (serves cached sites). See the [Architecture Guide](/architecture) for a detailed breakdown of how these services work together. ## Prerequisites - **PostgreSQL** database (14 or newer) -- **Bun** runtime for the main backend +- **Bun** runtime for the main backend and firehose service - **Node.js** (18+) for the hosting service - **Caddy** (optional, for custom domain TLS) - **Domain name** for your instance +- **S3-compatible storage** (optional, recommended for production — Cloudflare R2, MinIO, etc.) +- **Redis** (optional, for real-time cache invalidation between services) ## Architecture Overview ``` -┌─────────────────────────────────────────┐ ┌─────────────────────────────────────────┐ -│ Main Backend (port 8000) │ │ Hosting Service (port 3001) │ -│ - OAuth authentication │ │ - Firehose listener │ -│ - Site upload/management │ │ - Site caching │ -│ - Domain registration │ │ - Content serving │ -│ - Admin panel │ │ - Redirect handling │ -└─────────────────────────────────────────┘ └─────────────────────────────────────────┘ - │ │ - └─────────────────┬───────────────────────────┘ - ▼ - ┌─────────────────────────────────────────┐ - │ PostgreSQL Database │ - │ - User sessions │ - │ - Domain mappings │ - │ - Site metadata │ - └─────────────────────────────────────────┘ +┌──────────────────────────┐ ┌──────────────────────────┐ ┌──────────────────────────┐ +│ Main Backend (:8000) │ │ Firehose Service │ │ Hosting Service (:3001) │ +│ - OAuth authentication │ │ - Watches AT firehose │ │ - Tiered cache (mem/ │ +│ - Site upload/manage │ │ - Downloads blobs │ │ disk/S3) │ +│ - Domain registration │ │ - Writes to S3/disk │ │ - Content serving │ +│ - Admin panel │ │ - Publishes invalidation │ │ - Redirect handling │ +└──────────────────────────┘ └──────────────────────────┘ └──────────────────────────┘ + │ │ │ │ + │ │ S3/Disk │ Redis pub/sub │ + └────────┬───────────────┘ └─────────────────────┘ + ▼ +┌─────────────────────────────────────────┐ +│ PostgreSQL Database │ +│ - User sessions │ +│ - Domain mappings │ +│ - Site metadata │ +└─────────────────────────────────────────┘ ``` ## Database Setup @@ -106,21 +109,89 @@ bun run scripts/create-admin.ts Admin panel is available at `https://yourdomain.com/admin` +## Firehose Service Setup + +The firehose service watches the AT Protocol firehose for site changes and pre-populates the cache. It is **write-only** — it never serves requests to users. + +### Environment Variables + +```bash +# Required +DATABASE_URL="postgres://user:password@localhost:5432/wisp" + +# S3 storage (recommended for production) +S3_BUCKET="wisp-sites" +S3_REGION="auto" +S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com" +S3_ACCESS_KEY_ID="..." +S3_SECRET_ACCESS_KEY="..." +S3_METADATA_BUCKET="wisp-metadata" # Optional, recommended + +# Redis (for notifying hosting service of changes) +REDIS_URL="redis://localhost:6379" + +# Firehose +FIREHOSE_URL="wss://jetstream2.us-east.bsky.network/subscribe" +FIREHOSE_CONCURRENCY=5 # Max parallel event processing + +# Optional +CACHE_DIR="./cache/sites" # Fallback if S3 not configured +``` + +### Installation + +```bash +cd firehose-service + +# Install dependencies +bun install + +# Production mode +bun run start + +# With backfill (one-time bulk sync of all existing sites) +bun run start -- --backfill +``` + +The firehose service will: +1. Connect to the AT Protocol firehose (Jetstream) +2. Filter for `place.wisp.fs` and `place.wisp.settings` events +3. Download blobs, decompress, and rewrite HTML paths +4. Write files to S3 (or disk) +5. Publish cache invalidation events to Redis + ## Hosting Service Setup -The hosting service is a separate microservice that serves cached sites. +The hosting service is a **read-only** CDN that serves cached sites through a three-tier storage system (memory, disk, S3). ### Environment Variables ```bash # Required DATABASE_URL="postgres://user:password@localhost:5432/wisp" -BASE_HOST="wisp.place" # Same as main backend +BASE_HOST="wisp.place" # Same as main backend + +# Tiered storage +HOT_CACHE_SIZE=104857600 # Hot tier: 100 MB (memory, LRU) +HOT_CACHE_COUNT=500 # Max items in hot tier + +WARM_CACHE_SIZE=10737418240 # Warm tier: 10 GB (disk, LRU) +WARM_EVICTION_POLICY="lru" # lru, fifo, or size +CACHE_DIR="./cache/sites" # Warm tier directory + +# S3 cold tier (same bucket as firehose service, read-only) +S3_BUCKET="wisp-sites" +S3_REGION="auto" +S3_ENDPOINT="https://your-account.r2.cloudflarestorage.com" +S3_ACCESS_KEY_ID="..." +S3_SECRET_ACCESS_KEY="..." +S3_METADATA_BUCKET="wisp-metadata" + +# Redis (receive cache invalidation from firehose service) +REDIS_URL="redis://localhost:6379" # Optional -PORT="3001" # Default: 3001 -CACHE_DIR="./cache/sites" # Site cache directory -CACHE_ONLY_MODE="false" # Set true to disable DB writes +PORT="3001" # Default: 3001 ``` ### Installation @@ -136,23 +207,24 @@ npm run dev # Production mode npm run start - -# With backfill (downloads all sites from DB on startup) -npm run start -- --backfill ``` The hosting service will: -1. Connect to PostgreSQL -2. Start firehose listener (watches for new sites) -3. Create cache directory -4. Serve sites on port 3001 +1. Initialize tiered storage (hot → warm → cold) +2. Subscribe to Redis for cache invalidation events +3. Serve sites on port 3001 + +### Cache Behavior + +Files are cached across three tiers with automatic promotion: + +- **Hot (memory):** Fastest, limited by `HOT_CACHE_SIZE`. Evicted on restart. +- **Warm (disk):** Fast local reads at `CACHE_DIR`. Survives restarts. +- **Cold (S3):** Shared source of truth, populated by firehose service. -### Cache Management +On a cache miss at all tiers, the hosting service fetches directly from the user's PDS and promotes the file into the appropriate tiers. -Sites are cached to disk at `./cache/sites/{did}/{sitename}/`. The cache is automatically populated: -- **On first request**: Downloads from PDS and caches -- **Via firehose**: Updates when sites are deployed -- **Backfill mode**: Downloads all sites from database on startup +**Without S3:** Disk acts as both warm and cold tier. The hosting service still works — it just relies on on-demand fetching instead of pre-populated S3 cache. ## Reverse Proxy Setup @@ -318,11 +390,11 @@ Access observability metrics at `https://yourdomain.com/admin`: ## Scaling Considerations -- **Multiple hosting instances**: Run multiple hosting services behind a load balancer +- **Multiple hosting instances**: Run multiple hosting services behind a load balancer — each has its own hot/warm tiers but shares the S3 cold tier and Redis invalidation - **Separate databases**: Split read/write with replicas - **CDN**: Put Cloudflare or Bunny in front for global caching -- **Cache storage**: Use NFS/S3 for shared cache across instances -- **Redis**: Add Redis for session storage at scale +- **S3 cold tier**: Shared storage across all hosting instances (Cloudflare R2, MinIO, AWS S3) +- **Redis**: Required for real-time cache invalidation between firehose and hosting services at scale ## Security Notes diff --git a/docs/src/content/docs/index.mdx b/docs/src/content/docs/index.mdx index ce7139b..a262250 100644 --- a/docs/src/content/docs/index.mdx +++ b/docs/src/content/docs/index.mdx @@ -57,7 +57,9 @@ https://sites.wisp.place/{your-did}/{site-name} The deployment process starts when you upload your files. Each file is compressed with gzip, base64-encoded, and uploaded as a blob to your PDS. A `place.wisp.fs` record then stores the complete site structure with references to these blobs, creating a verifiable manifest of your site. -Hosting services continuously watch the AT Protocol firehose for new and updated sites. When your site is first accessed or updated, the hosting service downloads the manifest and blobs, caching them locally for optimized delivery. Custom domains work through DNS verification, allowing your site to be served from your own domain while maintaining the cryptographic guarantees of the AT Protocol. +The **firehose service** continuously watches the AT Protocol firehose for new and updated sites. When a site is created or updated, it downloads the manifest and blobs from the PDS, writes them to S3 (or disk), and publishes a cache invalidation event via Redis. The **hosting service** is a read-only CDN that serves files from a three-tier cache (memory, disk, S3). When a file isn't in cache, the hosting service fetches it on-demand from the PDS and promotes it through the tiers. + +Custom domains work through DNS verification, allowing your site to be served from your own domain while maintaining the cryptographic guarantees of the AT Protocol. ## Architecture Overview @@ -67,7 +69,6 @@ Hosting services continuously watch the AT Protocol firehose for new and updated │ (Rust Binary)│ │ Website │ │ │ │ (React UI) │ └──────────────┘ └──────────────┘ - │ │ │ │ ▼ ▼ ┌─────────────────────────────────────────────────────────┐ @@ -83,9 +84,7 @@ Hosting services continuously watch the AT Protocol firehose for new and updated │ │ │ ┌──────────────────────────────────────────────┐ │ │ │ Blobs (gzipped + base64 encoded) │ │ -│ │ - index.html │ │ -│ │ - styles.css │ │ -│ │ - assets/* │ │ +│ │ - index.html, styles.css, assets/* │ │ │ └──────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────┘ │ @@ -97,13 +96,26 @@ Hosting services continuously watch the AT Protocol firehose for new and updated │ ▼ ┌─────────────────────────────────────────────────────────┐ -│ Wisp Hosting Service │ +│ Firehose Service (Write Path) │ +│ │ +│ - Watches firehose for place.wisp.fs changes │ +│ - Downloads blobs from PDS │ +│ - Writes cached files to S3 / disk │ +│ - Publishes cache invalidation via Redis │ +└─────────────────────────────────────────────────────────┘ + │ │ + │ (S3 / Disk) │ (Redis pub/sub) + ▼ ▼ +┌─────────────────────────────────────────────────────────┐ +│ Hosting Service (Read Path) │ │ │ │ ┌──────────────────────────────────────────────┐ │ -│ │ Cache (Disk + In-Memory) │ │ -│ │ - Downloads sites on first access │ │ -│ │ - Auto-updates on firehose events │ │ -│ │ - LRU eviction for memory limits │ │ +│ │ Tiered Storage │ │ +│ │ ┌──────┐ ┌──────┐ ┌──────────────┐ │ │ +│ │ │ Hot │ → │ Warm │ → │ Cold │ │ │ +│ │ │(Mem) │ │(Disk)│ │(S3/Disk) │ │ │ +│ │ └──────┘ └──────┘ └──────────────┘ │ │ +│ │ On miss: fetch from PDS and promote up │ │ │ └──────────────────────────────────────────────┘ │ │ │ │ ┌──────────────────────────────────────────────┐ │ @@ -120,13 +132,17 @@ Hosting services continuously watch the AT Protocol firehose for new and updated └─────────────┘ ``` +For a detailed breakdown of the services and storage system, see the [Architecture Guide](/architecture). + ## Tech Stack - **Backend**: Bun + Elysia + PostgreSQL - **Frontend**: React 19 + Tailwind 4 + Radix UI -- **Hosting**: Node.js + Hono +- **Hosting Service**: Node.js + Hono +- **Firehose Service**: Bun - **CLI**: Rust + Jacquard (AT Protocol library) - **Protocol**: AT Protocol OAuth + custom lexicons +- **Storage**: S3-compatible (Cloudflare R2, MinIO, etc.) + Redis for cache invalidation ## Limits @@ -139,7 +155,8 @@ Files are automatically compressed with gzip before upload, so actual limits may ## Getting Started - [CLI Documentation](/cli) - Deploy sites from the command line -- [Deployment Guide](/deployment) - Configure domains, redirects, and hosting +- [Architecture Guide](/architecture) - How hosting, firehose, and tiered storage work +- [Self-Hosting Guide](/deployment) - Deploy your own instance - [Lexicons](/lexicons) - AT Protocol record schemas and data structures ## Links