diff --git a/docs/plans/PLAN-ATP-Blob-Hype.md b/docs/plans/PLAN-ATP-Blob-Hype.md new file mode 100644 index 0000000..7511a0e --- /dev/null +++ b/docs/plans/PLAN-ATP-Blob-Hype.md @@ -0,0 +1,233 @@ +# PLAN-ATP-Blob-Hype.md — Blob Hype (SKELETON — decisions open) + +**Goal:** a static, client-side web tool that downloads every blob (image) +for an AT Protocol account, sorts them by visual similarity, and assembles +them into a music video (images sequenced and synced to audio). + +Name: **Blob Hype** (working title — naming is OPEN, §10.1). Family: a +companion to [Sifter](https://sifter.psingletary.com) / Altifier / Verifier. +Repo: `git@tangled.org:psingletary.com/blob-hype` (OPEN — confirm). + +> **Status: SKELETON.** Unlike PLAN-sifter.md (fully decided), this plan +> deliberately leaves every consequential choice open in §10. Resolve those +> before pickup; the skeleton below captures what Sifter already proved. + +--- + +## TL;DR + +- **What:** pull all of an account's blobs → cluster/sort by visual similarity + → render a music video (Ken Burns + crossfade, beat-synced) → export video. +- **Why:** the pipeline is the product — "your own photos, as a hype video." +- **Key inheritance from Sifter:** blob retrieval is solved; the extractor, + OAuth, rate limiting, and deploy patterns lift nearly verbatim. +- **Prime invariant:** downloads are read-only and non-destructive — no + fail-closed flag needed (unlike Sifter's deletion). The hard parts are + similarity quality and video export, not safety. + +--- + +## 1. Context — what Sifter taught us (bake these in, do not rediscover) + +Blob retrieval for a full account is a solved problem. Proven this month: + +| Concern | Proven answer | Where | +|---|---|---| +| Auth | `@atproto/oauth-client-browser`; `clientMetadata` in AuthContext + `public/client-metadata.json` must match the deployed domain | Sifter AuthContext | +| OAuth scope | `atproto transition:generic` (read-only `atproto` may suffice for downloads — OPEN, §10.2) | Sifter | +| Held blobs | `com.atproto.sync.listBlobs` (`did`, `limit: 1000`, cursor loop) | Sifter scan.js | +| Records walk | `describeRepo` → `listRecords` per collection (`limit: 100`, cursor loop) | Sifter scan.js | +| **Blob refs are HYDRATED** | @atproto/api converts `ref: {$link}` into multiformats CID objects `{code, version, hash}`; extract via `ref.toString()` — string/`$link`/CID all needed | Sifter extract.js + F6 fixture | +| Raw bytes | `com.atproto.sync.getBlob` returns **Uint8Array** (not Blob) — `new Blob([data])` to download | Sifter BlobGrid/backup.js | +| Cheap previews | `https://cdn.bsky.app/img/feed_thumbnail/plain/{did}/{cid}@jpeg` — no auth, plain `` | atproto.at pattern, Sifter BlobGrid | +| Full-res URL | `{pds}/xrpc/com.atproto.sync.getBlob?did={did}&cid={cid}` | atproto.at "PDS" view | +| Rate budget | headers `ratelimit-*` authoritative; ~1s pacing safe; constants are fallbacks | Sifter rateLimit.js | +| CRA + @atproto | Pin `@atproto/api@0.13.35` + `@atproto/oauth-client-browser@0.3.30` (single lexicon 0.4.14); `"overrides": {"zod": "3.24.3"}`; 0.5x line is ESM-only, do NOT use with CRA | Sifter package.json + skill | +| Deploy | wisp.place `wispctl deploy --path build --site --spa`; Tangled Sites deploy dir `/build` | Sifter | + +**Design consequence:** the novel work is not fetching — it is (a) similarity +sorting, (b) video assembly/export in the browser, (c) audio sync. Those get +the milestone budget; retrieval is a lift. + +--- + +## 2. The pipeline (four stages) + +### Stage A — Fetch (`src/lib/fetch.js`) + +Reuse Sifter's scan engine shape (listBlobs + listRecords + extractor), then: + +1. Enumerate blob CIDs (held set; **do not** dedupe to referenced-only — the + whole point is ALL of them, orphans included). +2. For each CID, fetch the image. Two tiers: + - **Thumbnails first** (CDN URL, no auth) for the similarity pass — + similarity on thumbnails is ~100x cheaper than full-res. + - **Full-res only on demand** (export / selected favorites) via + authenticated getBlob (Uint8Array → Blob). +3. Persist progress to `localStorage` keyed by DID (resume after tab close), + mirroring Sifter's run-state pattern. +4. Rate limiting: same tracker as Sifter; ~4 parallel thumbnails, serial + full-res (downloads are reads — parallel reads are fine within budget). + +### Stage B — Similarity (`src/lib/similarity.js`) — THE OPEN CORE (§10.3) + +Sort N images "by similarity." Candidate approaches, cheapest first: + +1. **Perceptual hash (pHash)** — pure-JS (e.g. `sharp`-less: canvas → + 8×8 DCT/avg → 64-bit hash); sort via nearest-neighbor chain or + Hamiltonian-path over Hamming distances. Zero deps, instant, good enough + for a montage. +2. **Color histogram** — 16-bin RGB or hue histogram, cosine distance. Even + cheaper; good for tonal sequencing (sunset→dark), weak on subject. +3. **CLIP embeddings in-browser** — `@huggingface/transformers` (ONNX/WASM), + e.g. `Xenova/clip-vit-base-patch32`. Best semantic grouping (faces, cars, + landscapes) but ~100–300MB model download and slower on mobile. Requires + WebGPU/WASM runtime fallback (OPEN). +4. **Server-side embeddings** — a tiny API (Tangled-compatible?) or hosted + endpoint. Violates the "no backend" family convention unless justified + (OPEN — likely rejected). + +Sorting shape (OPEN §10.4): nearest-neighbor chain (each image attaches to +its most-similar unvisited image → smooth montage) vs. cluster-then-order +(grouped scenes, jump cuts between clusters). + +### Stage C — Music video assembly (`src/lib/video.js`) + +Browser-native, no backend: + +- **Canvas renderer:** draw images with Ken Burns (slow zoom/pan) + crossfade + between adjacent images; the sorted order IS the edit order. +- **Audio:** user uploads an MP3/WAV (WebAudio `decodeAudioData`), OR + WebAudio-synthesized beat (OPEN §10.5). No music licensing issues. +- **Sync:** beat detection (onset energy via AnalyserNode) or fixed BPM — + images advance on beats (OPEN §10.6). Fallback: time-based cuts. +- **Capture:** `canvas.captureStream(30)` + `MediaRecorder` → WebM. Audio + mixed via `AudioContext.createMediaStreamDestination()`. +- **Export quality:** WebM/VP9 default; `ffmpeg.wasm` for MP4/H.264 or + exact-CLI output is a stretch option (OPEN §10.7 — large WASM download). + +### Stage D — UI (match Sifter's layout language) + +Single page, states mirroring Sifter (logged out → fetching → sorting → +preview/export). Tabs: **Fetch | Sort | Video**. Sticky top action bar +(Fetch all / Re-sort / Preview / Export). Minimal functional CSS, 18px base, +48px targets, dark/light via the lifted ThemeContext. + +Copy honesty: "Your images are processed entirely in your browser. Nothing +is uploaded. Big accounts = big downloads." + +--- + +## 3. Edge cases + +| Case | Handling | +|---|---| +| Very large accounts (10k+ blobs) | Thumbnail-only similarity; chunked fetch with progress; memory = CID lists + hashes, never full-res until export | +| Non-image blobs (video, PDFs, audio) | Filter to image MIME types for the video; list them separately (count only) | +| Orphaned blobs | Included (they're your blobs too); note that full-res getBlob 404 → skip + report | +| CDN 404 on a thumbnail | Fall back to PDS getBlob URL; if that also fails, skip with a count | +| Zero images | Success-state "no images" — same family tone as Sifter | +| Mobile | Video export is heavy; show warning + suggest desktop; fetch/sort work fine | +| Rate limited mid-fetch | Pause, countdown from `ratelimit-reset`, resume (Sifter pattern) | +| Private/self-hosted PDS | getBlob may require auth — OAuth session covers it; note for self-hosters | +| Deactivated account | Surface error clearly (Sifter §6 pattern) | + +--- + +## 4. File layout (skeleton) + +``` +public/ + client-metadata.json +vendor/ # lifted from Sifter (author pre-copies, §10.8) + AuthContext.js + ThemeContext.js + styles/ (variables.css, app.css) +src/ + lib/ + fetch.js # Stage A — blob enumeration + tiered download + similarity.js # Stage B — pHash/histogram/CLIP + sort (§10.3) + video.js # Stage C — canvas Ken Burns + MediaRecorder + audio.js # Stage C — decode, beat detection, sync + rateLimit.js # lifted from Sifter + extract.js # lifted from Sifter (hydrated-CID aware!) + components/ + FetchProgress.js + SortPreview.js + VideoTimeline.js + ExportPanel.js + pages/BlobHype.js + __tests__/ + __fixtures__/ # CID fixtures (F6 pattern from Sifter) + fetch.test.js + similarity.test.js # known-similar image pair fixture + video.test.js # mocked canvas/MediaRecorder +README.md PLAN-ATP-Blob-Hype.md LICENSE +``` + +--- + +## 5. Milestones (draft — re-cut after §10 decisions) + +- **M1 — Scaffold + Fetch.** CRA (same pins as Sifter), OAuth login, fetch + all blobs with progress + resume. Thumbnails render (CDN). Validated on + author account. +- **M2 — Similarity.** pHash + nearest-neighbor sort; preview strip shows + the ordering. Ship read-only (like Sifter M2) if quality is good enough. + > HARD STOP — author reviews ordering quality before video work. +- **M3 — Video v1.** Ken Burns + crossfade + uploaded audio, time-based cuts. + Export WebM. (No beat sync yet.) +- **M4 — Beat sync.** Onset detection / BPM; images advance on beats. + Author approves sync feel. +- **M5 — Polish + export.** ffmpeg.wasm MP4 (if chosen), edge cases, + README, LICENSE (MIT). + +### Acceptance criteria (draft) + +- Automated: fetch of a mocked account returns all CIDs (fixtures F1–F6). +- Automated: similarity sorts a known-similar fixture pair adjacently. +- Automated: video export produces a non-empty WebM blob (mocked canvas). +- Manual (author): full pipeline on real account → watchable music video. +- No backend, no API keys, no analytics. Works from static hosting. + +--- + +## 6. Out of scope (draft) + +- Account analytics / charts. Record deletion (that's Sifter). Social sharing. +- Server-side rendering or transcoding (unless §10.7 flips). +- Video editing UI (trim/captions) — export-then-edit elsewhere. + +--- + +## 7. Conventions to reuse verbatim (family consistency) + +- Stack: CRA, JavaScript, Jest, `react-scripts test` — **do not migrate to + Vite** (Sifter guardrail). +- Auth/layout/styles lifted from Sifter's vendor/, never fetched from web. +- Deploy: wisp.place + Tangled Sites, same commands as Sifter. +- No emojis in code/docs (user convention). + +--- + +## 8. Open questions — RESOLVE BEFORE PICKUP (this is the skeleton's point) + +| # | Question | Options | Default lean | +|---|---|---|---| +| 1 | Name | Blob Hype / Hype Reel / montage / something else | Blob Hype | +| 2 | OAuth scope | `atproto` (read-only) vs `transition:generic` | read-only if it works; else generic | +| 3 | Similarity method | pHash / histogram / in-browser CLIP / server embeddings | pHash for M2, CLIP as stretch | +| 4 | Sort shape | nearest-neighbor chain vs cluster-then-order | nearest-neighbor | +| 5 | Audio | user-uploaded vs WebAudio-synth vs both | both (upload primary) | +| 6 | Beat sync | onset detection vs fixed BPM vs time-based | onset detection, BPM fallback | +| 7 | Export | WebM only vs + ffmpeg.wasm MP4 | WebM first, MP4 stretch | +| 8 | vendor/ source | lift from Sifter (author pre-copies) vs Verifier | Sifter (same auth stack) | +| 9 | Repo name/path | blob-hype vs hype vs other | blob-hype | + +--- + +## 9. References + +- PLAN-sifter.md (archive) — the proven blob-retrieval pattern. +- atproto-development skill (§CRA/webpack pitfalls, OAuth, Wisp/Tangled deploy). +- atproto.at blob page — the rendering reference (CDN thumbs + PDS getBlob).