From 25c3f3baed77492a72dc894bdd86f4348752d2e9 Mon Sep 17 00:00:00 2001 From: Boris Mann Date: Thu, 02 Jul 2026 09:17:44 +0000 Subject: [PATCH] Add README and documentation: usage, development guide, session summary, and next steps --- README | 5 ----- README.md | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ docs/development.md | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ docs/next-steps.md | 43 +++++++++++++++++++++++++++++++++++++++++++ docs/session-summary.md | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 5 file(s) changed, 275 insertion(s)(+), 5 deletion(s)(-) diff --git a/README b/README deleted file mode 100644 --- a/README +++ /dev/null @@ -1,5 +0,0 @@ -# GhostOff - -Exporting content using the Ghost API, and uploading it to Offprint and other atproto / Standard Site formats - -https://ghostoff.offprint.app diff --git a/README.md b/README.md new file mode 100644 --- /dev/null +++ b/README.md @@ -0,0 +1,91 @@ +# GhostOff + +Export public posts from a [Ghost](https://ghost.org/) CMS and upload them to [Offprint](https://offprint.app/) via the AT Protocol / [Standard.site](https://standard.site/) lexicons. + +The live test publication is at ****. + +## What it does + +- Paginates through Ghost's public Content API for posts. +- Downloads each post's images, resizes them to fit atproto's blob limits, and uploads them as blobs to a PDS. +- Converts Ghost HTML bodies into Offprint's block-based format (`app.offprint.content`). +- Writes a `site.standard.document` record per post and an `app.offprint.document.article` record that points back to it. +- Tracks progress in a local state file so reruns update existing records instead of duplicating them. + +## Quick start + +Copy `.env.example` to `.env` and fill in the secrets: + +```bash +cp .env.example .env +``` + +```env +GHOST_URL=https://your-ghost-site.com +GHOST_API_KEY=your-ghost-content-api-key +ATP_IDENTIFIER=your-handle-or-did +ATP_APP_PASSWORD=your-app-password +ATP_SERVICE=https://selfhosted.social +ATPUBLICATION_AT_URI=at://did:plc:.../site.standard.publication/... +``` + +Run a dry first pass to inspect what would be uploaded: + +```bash +npm install +npm run dev -- --dry-run +``` + +When you're ready, run the real migration: + +```bash +npm run dev +``` + +Rerunning the same command will `putRecord` existing records instead of creating duplicates. + +## CLI options + +| Flag | Environment variable | Description | +|------|------------------------|-------------| +| `--ghost-url` | `GHOST_URL` | Ghost publication URL | +| `--ghost-api-key` | `GHOST_API_KEY` | Ghost Content API key | +| `--atproto-identifier` | `ATP_IDENTIFIER` | atproto handle or DID | +| `--atproto-app-password` | `ATP_APP_PASSWORD` | atproto app password | +| `--atproto-service` | `ATP_SERVICE` | PDS/service URL | +| `--publication-at-uri` | `ATPUBLICATION_AT_URI` | Existing `site.standard.publication` AT-URI | +| `--dry-run` | — | Build records without uploading | +| `--export-dir` | — | Local asset cache directory (default: `ghostoff-export`) | +| `--state-file` | — | Idempotency state file (default: `ghostoff-state.json`) | +| `--verbose` | — | Verbose logging | + +## Scripts + +```bash +npm run dev # run the migration +npm run build # compile TypeScript to dist/ +npm run typecheck # type-check without emitting +``` + +## Project layout + +``` +src/ +├── index.ts # CLI orchestration +├── config.ts # env + argv loading +├── ghost.ts # Ghost Content API client +├── assets.ts # image download, cache, resize, blob upload +├── html-to-offprint.ts # Ghost HTML -> Offprint blocks +├── facets.ts # inline formatting -> richtext facets +├── atproto.ts # auth, record create/put, validation +├── state.ts # idempotency state file +├── rate-limit.ts # retry + rate-limit helpers +└── types.ts # shared type definitions +``` + +## Learning more + +- `PLAN.md` — the original migration plan. +- `docs/session-summary.md` — a narrative of how the first version was built. +- `docs/development.md` — how the codebase is organized. +- `docs/next-steps.md` — ideas for what to add or improve next. diff --git a/docs/development.md b/docs/development.md new file mode 100644 --- /dev/null +++ b/docs/development.md @@ -0,0 +1,66 @@ +# Working with the GhostOff codebase + +This doc describes the architecture of the migration tool and how the pieces fit together. + +## Entry point + +`src/index.ts` is the CLI entry point. It: + +1. Loads configuration from `.env` / CLI flags (`src/config.ts`). +2. Loads the previous run state (`src/state.ts`). +3. Authentices with the atproto PDS (`src/atproto.ts`). +4. Fetches all public posts from Ghost (`src/ghost.ts`). +5. For each post, downloads images, builds blocks, validates the document, writes records, and saves state. + +## Configuration + +`src/config.ts` uses `commander` and `dotenv`. Any CLI flag overrides its corresponding environment variable. Required values are validated eagerly; missing values throw before any network calls are made. + +## Ghost client + +`src/ghost.ts` is a thin wrapper around the Ghost Content API. It fetches `/settings` and paginates `/posts` with `include=authors,tags&formats=html,plaintext&limit=100`. + +## Asset pipeline + +`src/assets.ts`: + +- Resolves relative image URLs against the Ghost base URL. +- Downloads images to `ghostoff-export/assets/` with a SHA-256 hash filename, skipping already-cached files. +- Uses `sharp` to resize/compress images over ~900 KB. +- Uploads processed images via the raw `com.atproto.repo.uploadBlob` XRPC endpoint. +- Blob references are cached in the state file so reruns don't re-upload identical images. + +## HTML transformation + +`src/html-to-offprint.ts` parses each post's HTML with `linkedom` and converts the body into an `app.offprint.content` item array. Because `linkedom`'s `parseHTML()` treats fragments oddly, the HTML is wrapped in `...` before parsing. + +`src/facets.ts` handles inline formatting. It recursively walks inline nodes, builds plaintext segments, and computes UTF-8 byte-offset facets for bold, italic, links, code, etc. + +## atproto layer + +`src/atproto.ts` wraps: + +- PasswordSession login/resume. +- Blob upload (raw `fetch` because atcute's client is JSON-oriented). +- `com.atproto.repo.createRecord` and `com.atproto.repo.putRecord` for documents and articles. +- Runtime validation of `site.standard.document` against `@atcute/standard-site`. + +## State and idempotency + +`src/state.ts` reads/writes `ghostoff-state.json`. It maps each Ghost post `uuid` to the created document/article AT-URIs and CIDs. On reruns, existing records are updated via `putRecord` with `swapRecord` set to the old CID. + +The state file also caches uploaded image blob references so images are only uploaded once. + +## Rate limiting + +`src/rate-limit.ts` provides small helpers for parsing `RateLimit-*` and `Retry-After` headers, plus exponential backoff. XRPC calls use `@atcute/client`'s `retryFetchHandler`. Blob uploads have a manual retry loop. + +## Type checking and conventions + +```bash +npm run typecheck +``` + +The project uses strict TypeScript, ESM, and Node.js's native `fetch`. The `DOM` lib is included because the tool manipulates HTML elements via `linkedom`. + +When adding new block conversions, keep the type definitions in `src/html-to-offprint.ts` close to the conversion logic, and make sure the emitted object has a `$type` discriminator matching the real Offprint lexicon. diff --git a/docs/next-steps.md b/docs/next-steps.md new file mode 100644 --- /dev/null +++ b/docs/next-steps.md @@ -0,0 +1,43 @@ +# What's next for GhostOff + +This is a first working version. The items below are roughly ordered from smaller polish to larger additions. + +## Content conversion + +- **Better embed support.** Bluesky embeds are already converted to `block.webEmbed`, but other embeds (YouTube iframes, GitHub cards, generic oEmbeds) are currently skipped or fall back to text. Extract `href` from more iframe/src patterns and consider fetching OpenGraph metadata to populate `webBookmark` blocks. +- **Picture and srcset handling.** Ghost can serve responsive images via `` and `srcset`. We currently only use the first ``. Prefer the largest available source for upload. +- **Unsupported image formats.** `.ico` files and some SVGs fail in `sharp`. Either skip them cleanly, convert SVGs to PNG, or detect icons and ignore them entirely. +- **Code block language detection.** Ghost stores the language in the `` attribute. We extract it, but Ghost sometimes uses different class names; broaden the regex. +- **Reduce warning noise.** Block-level elements that contain mixed inline content sometimes produce `unsupported element` warnings. Some of these are inside paragraphs and should be handled inline rather than as block fallbacks. + +## CLI and UX + +- **Selective migration.** Add flags to filter by tag, date range, or slug, and to limit the number of posts processed. +- **Progress output.** Add a progress bar or summary report at the end. +- **JSON/NDJSON export.** Allow exporting records to disk without uploading, for manual inspection or external publishing. +- **Resume after failure.** If the run stops midway, restarting it should pick up from where it left off based on the state file. +- **Configurable image limits.** Expose the blob size target and resize dimensions as CLI flags. + +## Reliability + +- **Full retry policy for Ghost.** The Ghost API can also rate-limit; apply the same retry/backoff logic to `src/ghost.ts`. +- **Retry failed uploads with state awareness.** If a blob upload fails temporarily, don't mark the image as processed in the state file. +- **Validation of Offprint blocks.** The atcute validator only validates `site.standard.document` because it doesn't know about `app.offprint` schemas. Pull the Offprint schemas from Lexicon Garden and validate `app.offprint.content` blocks, or at least check byte offsets and required fields in tests. + +## Tests + +- Add unit tests for the block converter using small HTML fixtures. +- Test facet generation for nested inline tags and multi-byte UTF-8 characters. +- Test the asset pipeline with mocked `fetch` responses. +- Test idempotency: a second run should call `putRecord`, not `createRecord`. + +## Auth and deployment + +- **OAuth support.** App passwords work for one-off migrations, but OAuth would be safer for interactive use and for users on custom PDSs. +- **GitHub Action.** Provide a workflow that runs the migration on a schedule so a Ghost publication can stay in sync with Offprint. +- **Docker image.** Make the tool runnable without a local Node installation. + +## Other platforms + +- **Standard.site beyond Offprint.** The records already use Standard.site, so they should be readable by Leaflet, pckt, and other readers. Add a mode that skips the Offprint article wrapper for users who only want Standard.site documents. +- **Export to other formats.** Use the same converter to emit Markdown, plain HTML, or static-site-friendly JSON. diff --git a/docs/session-summary.md b/docs/session-summary.md new file mode 100644 --- /dev/null +++ b/docs/session-summary.md @@ -0,0 +1,75 @@ +# Building GhostOff: from Ghost CMS to Offprint in one session + +GhostOff is a small TypeScript tool that exports public posts from a Ghost CMS and uploads them to Offprint using the AT Protocol and Standard.site lexicons. This post walks through the decisions and gotchas from the first build session. + +## Starting point + +The goal was straightforward: take content that lives in Ghost and make it available on Offprint, an atproto-based long-form publishing platform. Ghost exposes a read-only Content API, and Offprint's underlying data format is a set of atproto records (`site.standard.document` plus `app.offprint.document.article`). So the task reduces to: + +1. Fetch posts from Ghost. +2. Download and re-host images as atproto blobs. +3. Convert Ghost HTML into Offprint blocks. +4. Write the records to a PDS. +5. Keep track of what was uploaded so reruns are idempotent. + +## Stack choices + +We picked TypeScript because the atproto ecosystem has first-class tooling there. For the atproto side we used the atcute packages: `@atcute/client` for XRPC, `@atcute/password-session` for password-based auth, and `@atcute/standard-site` plus `@atcute/lexicons` for schemas and runtime validation. + +For DOM parsing we used `linkedom`, which is light and Node-friendly. For image resizing we used `sharp`. For the CLI surface we used `commander` and `dotenv`. + +## Mapping the two formats + +Ghost posts come back as HTML with a JSON representation of the Ghost editor's cards: headings, paragraphs, galleries, embeds, code blocks, and custom HTML. Offprint expects a block array: `app.offprint.block.text`, `heading`, `bulletList`, `orderedList`, `codeBlock`, `image`, `imageGrid`, `webEmbed`, `blockquote`, and so on. + +The converter walks the body of the post and maps top-level elements: + +- `

` -> `block.text` +- `

`-`

` -> `block.heading` +- `