diff --git a/docs/superpowers/plans/2026-07-06-markpub-content.md b/docs/superpowers/plans/2026-07-06-markpub-content.md new file mode 100644 index 0000000..024a964 --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-markpub-content.md @@ -0,0 +1,534 @@ +# Markpub Content Member Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Every `site.standard.document` record ghoststandard writes gains a `content` union member of type `at.markpub.markdown`, derived from the Ghost post's HTML. + +**Architecture:** A new `src/lib/content-shape.ts` module owns the union member: a module-level turndown (HTML→Markdown) converter with the GFM plugin and one Ghost-specific rule for captioned image figures, plus `buildMarkpubContent(html)` which returns the record-ready member or `null`. `ghostPostToRecord` in `src/lib/transform.ts` attaches the member alongside the other optional fields. Tests run on Node's built-in `node:test` through the already-present `tsx`, validating the emitted shape against markpub lexicon JSONs vendored as fixtures. + +**Tech Stack:** TypeScript (strict, ESM, `lib: ES2022` — **no DOM types**), `turndown` + `turndown-plugin-gfm` (runtime), `@atproto/lexicon` (dev, validation only), `node:test` + `tsx` (test runner). + +**Spec:** `docs/superpowers/specs/2026-07-06-markpub-content-design.md` + +## Global Constraints + +- Code style: single quotes, 2-space indent, aligned colons in record-field blocks (see `src/lib/transform.ts:47-63` for the house style). +- ESM imports use `.js` extensions for local files (`from './transform.js'`), even from `.ts` sources. +- tsconfig `lib` is `["ES2022"]` — you may NOT reference `HTMLElement`, `Node`, or any DOM lib type in source files. The turndown ambient shim defines its own minimal node type instead. +- No new dependencies beyond: `turndown@^7.2.4`, `turndown-plugin-gfm@^1.0.2` (runtime); `@atproto/lexicon@^0.7.5` (dev). No `@types/turndown`, no vitest/jest. +- Emitted member fields are exactly `$type`, `text.markdown`, `flavor: 'gfm'` — `renderingRules` is deliberately omitted (derived markdown; nothing renders our pages from it). +- A conversion failure must never fail a sync: `buildMarkpubContent` returns `null` instead of throwing, always. +- Gates before every commit: `npm run typecheck` and `npm test`. +- Commit messages end with: + `Co-Authored-By: Claude Fable 5 ` + +--- + +### Task 1: Test harness + vendored markpub lexicon fixtures + +Introduces the repo's first test setup (`node:test` via `tsx`), vendors the two markpub lexicon JSONs from the sibling gifthood.org checkout, and splits the build config so test files never land in `dist/`. + +**Files:** +- Create: `src/lexicons/markpub/markdown.json` (copied) +- Create: `src/lexicons/markpub/text.json` (copied) +- Create: `src/lexicons/markpub/README.md` (copied) +- Create: `tsconfig.build.json` +- Create: `src/lib/content-shape.test.ts` (fixture-loading tests only; Task 2 extends it) +- Modify: `package.json` (scripts `build`/`test`, devDependency) + +**Interfaces:** +- Consumes: nothing from other tasks. +- Produces: `npm test` (runs `src/lib/**/*.test.ts`), fixtures at `src/lexicons/markpub/{markdown,text}.json`, and the `lexicons` loading pattern Task 2's tests reuse. `npm run build` compiles to `dist/` without test files. + +- [ ] **Step 1: Install the validation dev dependency** + +```bash +cd /Users/timtrautmann/Development/ghoststandard +npm install --save-dev @atproto/lexicon@^0.7.5 +``` + +Expected: `package.json` devDependencies gains `"@atproto/lexicon": "^0.7.5"`. + +- [ ] **Step 2: Vendor the lexicon fixtures from gifthood.org** + +```bash +mkdir -p src/lexicons/markpub +cp /Users/timtrautmann/Development/gifthood.org/src/lexicons/markpub/markdown.json src/lexicons/markpub/ +cp /Users/timtrautmann/Development/gifthood.org/src/lexicons/markpub/text.json src/lexicons/markpub/ +cp /Users/timtrautmann/Development/gifthood.org/src/lexicons/markpub/README.md src/lexicons/markpub/ +diff -r /Users/timtrautmann/Development/gifthood.org/src/lexicons/markpub src/lexicons/markpub && echo IDENTICAL +``` + +Expected: `IDENTICAL`. The README already records the upstream source (AramZS/markpub.at, MIT, commit `9b53a3a8f93d`, fetched 2026-07-06) and contains nothing gifthood-specific — copy verbatim, do not edit. + +- [ ] **Step 3: Add the test script and the build/test tsconfig split** + +Create `tsconfig.build.json`: + +```json +{ + "extends": "./tsconfig.json", + "exclude": ["node_modules", "dist", "src/**/*.test.ts"] +} +``` + +In `package.json`, change the `build` script and add `test` (keep `typecheck` as-is — it must keep checking test files): + +```json + "scripts": { + "dev": "tsx watch --clear-screen=false src/server.ts", + "cli": "tsx src/cli.ts", + "build": "tsc -p tsconfig.build.json", + "start": "node dist/server.js", + "test": "node --import tsx --test \"src/lib/**/*.test.ts\"", + "typecheck": "tsc --noEmit" + }, +``` + +- [ ] **Step 4: Write the initial test file (fixtures load + negative control)** + +Create `src/lib/content-shape.test.ts`: + +```ts +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { Lexicons } from '@atproto/lexicon'; + +// Vendored fixtures are loaded via fs (not import) so no tsconfig JSON-module +// setting is load-bearing for the test suite. +const lexDoc = (name: string) => + JSON.parse(readFileSync(new URL(`../lexicons/markpub/${name}`, import.meta.url), 'utf8')); +const lexicons = new Lexicons([lexDoc('markdown.json'), lexDoc('text.json')]); + +test('the vendored lexicon genuinely rejects invalid shapes (negative control)', () => { + const result = lexicons.validate('at.markpub.markdown', { + $type: 'at.markpub.markdown', + flavor: 'gfm', + }); + assert.ok(!result.success); // required `text` field missing — validation must fail +}); +``` + +- [ ] **Step 5: Run the tests** + +```bash +npm test +``` + +Expected: PASS — `tests 1`, `pass 1`, `fail 0`. + +- [ ] **Step 6: Verify the build stays clean of test files** + +```bash +rm -rf dist && npm run build && find dist -name "*.test.*" | wc -l && ls dist/lib +``` + +Expected: `0` from the `find`, and `ls dist/lib` shows the existing modules (`atproto.js`, `transform.js`, …) with no `content-shape.test.*`. Also confirm no `dist/lexicons` directory exists (fixtures are fs-read at test time, never compiled). + +- [ ] **Step 7: Typecheck** + +```bash +npm run typecheck +``` + +Expected: exit 0, no output. + +- [ ] **Step 8: Commit** + +```bash +git add package.json package-lock.json tsconfig.build.json src/lexicons/markpub src/lib/content-shape.test.ts +git commit -m "test: node:test harness + vendored markpub lexicon fixtures + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 2: `content-shape.ts` — HTML → at.markpub.markdown (TDD) + +**Files:** +- Create: `src/lib/content-shape.ts` +- Modify: `src/types.d.ts` (ambient declarations for `turndown` and `turndown-plugin-gfm`) +- Modify: `src/lib/content-shape.test.ts` (extend with the full suite) +- Modify: `package.json` (runtime dependencies) + +**Interfaces:** +- Consumes: fixtures + `lexicons` pattern from Task 1. +- Produces: `MARKPUB_MARKDOWN_TYPE: string` (`'at.markpub.markdown'`) and `buildMarkpubContent(html: string): Record | null` from `src/lib/content-shape.js` — Task 3 imports both. + +- [ ] **Step 1: Install the runtime dependencies** + +```bash +npm install turndown@^7.2.4 turndown-plugin-gfm@^1.0.2 +``` + +- [ ] **Step 2: Add ambient declarations** + +Append to `src/types.d.ts` (both packages ship no usable types for this repo — `@types/turndown` leans on DOM lib types and tsconfig `lib` is ES2022-only, so we shim exactly the surface we touch): + +```ts +declare module 'turndown' { + // Minimal stand-in for the DOM node turndown hands to rules (it uses a + // bundled DOM implementation in Node) — tsconfig lib is ES2022-only, so + // real DOM types are unavailable here. + export interface TurndownNode { + nodeName: string; + textContent: string | null; + getAttribute(name: string): string | null; + querySelector(selector: string): TurndownNode | null; + } + export interface TurndownRule { + filter: string[] | ((node: TurndownNode) => boolean); + replacement: (content: string, node: TurndownNode) => string; + } + export default class TurndownService { + constructor(options?: Record); + use(plugin: (service: TurndownService) => void): this; + addRule(key: string, rule: TurndownRule): this; + turndown(html: string): string; + } +} + +declare module 'turndown-plugin-gfm' { + import type TurndownService from 'turndown'; + export const gfm: (service: TurndownService) => void; +} +``` + +- [ ] **Step 3: Extend the test file with the full failing suite** + +Replace the entire contents of `src/lib/content-shape.test.ts` with: + +```ts +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync } from 'node:fs'; +import { Lexicons } from '@atproto/lexicon'; + +import { buildMarkpubContent, MARKPUB_MARKDOWN_TYPE } from './content-shape.js'; + +// Vendored fixtures are loaded via fs (not import) so no tsconfig JSON-module +// setting is load-bearing for the test suite. +const lexDoc = (name: string) => + JSON.parse(readFileSync(new URL(`../lexicons/markpub/${name}`, import.meta.url), 'utf8')); +const lexicons = new Lexicons([lexDoc('markdown.json'), lexDoc('text.json')]); + +test('buildMarkpubContent output validates against the vendored at.markpub.markdown lexicon', () => { + const content = buildMarkpubContent('

Hello

Body with a link.

'); + assert.ok(content, 'expected a content member, got null'); + const result = lexicons.validate(MARKPUB_MARKDOWN_TYPE, content); + assert.ok(result.success, result.success ? '' : String(result.error)); +}); + +test('the vendored lexicon genuinely rejects invalid shapes (negative control)', () => { + const result = lexicons.validate(MARKPUB_MARKDOWN_TYPE, { + $type: MARKPUB_MARKDOWN_TYPE, + flavor: 'gfm', + }); + assert.ok(!result.success); // required `text` field missing — validation must fail +}); + +test('buildMarkpubContent carries exactly the declared fields', () => { + const content = buildMarkpubContent('

body

') as Record & { text: { markdown: string } }; + assert.equal(content.$type, MARKPUB_MARKDOWN_TYPE); + assert.equal(content.text.markdown, 'body'); + assert.equal(content.flavor, 'gfm'); + // renderingRules deliberately absent (derived markdown — spec "The wrinkle"), + // and nothing extra sneaks into records: textBlob/facets/lenses/frontMatter + // stay omitted (spec decision 4). + assert.deepEqual(Object.keys(content).sort(), ['$type', 'flavor', 'text']); +}); + +test('prose, headings, and links convert exactly (atx headings)', () => { + const md = buildMarkpubContent( + '

Heading

Some bold text and a link.

', + ) as { text: { markdown: string } }; + assert.equal(md.text.markdown, '## Heading\n\nSome **bold** text and a [link](https://example.com/).'); +}); + +test('Ghost kg-image-card figures become image + caption lines', () => { + const md = buildMarkpubContent( + '
' + + 'A pic' + + '
The caption
' + + '
', + ) as { text: { markdown: string } }; + assert.equal(md.text.markdown, '![A pic](https://blog.example.com/content/images/pic.jpg)\n\n*The caption*'); +}); + +test('code cards become fenced blocks with the language tag', () => { + const md = buildMarkpubContent( + '
const x = 1;\n
', + ) as { text: { markdown: string } }; + assert.match(md.text.markdown, /^```javascript\nconst x = 1;\n```$/); +}); + +test('tables convert via the GFM plugin', () => { + const md = buildMarkpubContent( + '' + + '
ab
12
', + ) as { text: { markdown: string } }; + assert.match(md.text.markdown, /\| a \| b \|/); + assert.match(md.text.markdown, /\| 1 \| 2 \|/); +}); + +test('embed iframes drop out of the markdown', () => { + const md = buildMarkpubContent( + '

Watch this:

' + + '
', + ) as { text: { markdown: string } }; + assert.match(md.text.markdown, /Watch this:/); + assert.doesNotMatch(md.text.markdown, /youtube|iframe/); +}); + +test('an embed-only post converts to nothing → null', () => { + assert.equal( + buildMarkpubContent( + '
', + ), + null, + ); +}); + +test('empty and whitespace-only HTML → null', () => { + assert.equal(buildMarkpubContent(''), null); + assert.equal(buildMarkpubContent(' \n\t '), null); +}); +``` + +- [ ] **Step 4: Run the tests to verify they fail** + +```bash +npm test +``` + +Expected: FAIL — the suite cannot load `./content-shape.js` (module does not exist yet). + +- [ ] **Step 5: Write the implementation** + +Create `src/lib/content-shape.ts`: + +```ts +// The document `content` union member: Ghost post HTML → at.markpub.markdown. +// site.standard.document's content union is open (`closed: false`), so the +// member needs no lexicon registration — but a shared $type is a shared +// contract: at.markpub.markdown (markpub.at, AramZS) is a community lexicon +// for markdown-in-a-union that standard.site viewers understand. The markdown +// here is *derived* from Ghost's HTML (Ghost emits html/lexical/plaintext, +// never markdown), which is why `renderingRules` is omitted: nothing ever +// renders our pages from it. +// Design: docs/superpowers/specs/2026-07-06-markpub-content-design.md + +import TurndownService from 'turndown'; +import { gfm } from 'turndown-plugin-gfm'; + +export const MARKPUB_MARKDOWN_TYPE = 'at.markpub.markdown'; + +const turndown = new TurndownService({ + headingStyle: 'atx', + codeBlockStyle: 'fenced', +}); +turndown.use(gfm); + +// Ghost wraps every editor card in
. Only +// captioned images need a rule (the defaults would smear the figcaption into +// the surrounding text); every other card type degrades through the generic +// rules — bookmark cards to links, embed iframes to nothing. +turndown.addRule('kgImageCard', { + filter: (node) => + node.nodeName === 'FIGURE' && + /\bkg-image-card\b/.test(node.getAttribute('class') ?? ''), + replacement: (_content, node) => { + const img = node.querySelector('img'); + if (!img) return ''; + const image = `![${img.getAttribute('alt') ?? ''}](${img.getAttribute('src') ?? ''})`; + const caption = node.querySelector('figcaption')?.textContent?.trim(); + return caption ? `\n\n${image}\n\n*${caption}*\n\n` : `\n\n${image}\n\n`; + }, +}); + +/** + * Record-ready `content` union member for a post body, or null when there is + * nothing to say: empty HTML, HTML that converts to empty markdown (e.g. an + * embed-only post), or a converter throw. Null means the record simply omits + * `content` — a conversion bug must never fail a sync. + */ +export function buildMarkpubContent(html: string): Record | null { + if (!html.trim()) return null; + + let markdown: string; + try { + markdown = turndown.turndown(html); + } catch (err) { + console.error(' markdown conversion failed, omitting content:', (err as Error).message); + return null; + } + if (!markdown.trim()) return null; + + return { + $type: MARKPUB_MARKDOWN_TYPE, + text: { markdown }, + flavor: 'gfm', + }; +} +``` + +- [ ] **Step 6: Run the tests to verify they pass** + +```bash +npm test +``` + +Expected: PASS — `tests 10`, `fail 0`. + +If (and only if) one of the two *exact-match* conversion assertions fails on incidental whitespace or escaping produced by turndown: print the actual markdown, confirm it is semantically identical (same heading/image/caption content), and adjust the expected string in the test to the real output. Any other mismatch (missing caption, image URL mangled, wrong fields) is an implementation bug — fix `content-shape.ts`, not the test. + +- [ ] **Step 7: Typecheck** + +```bash +npm run typecheck +``` + +Expected: exit 0. + +- [ ] **Step 8: Commit** + +```bash +git add package.json package-lock.json src/types.d.ts src/lib/content-shape.ts src/lib/content-shape.test.ts +git commit -m "feat: build at.markpub.markdown content from Ghost post HTML + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +### Task 3: Wire the member into the document record + README note + +**Files:** +- Create: `src/lib/transform.test.ts` +- Modify: `src/lib/transform.ts:47-63` (import + one optional-field line) +- Modify: `README.md` (line 15, the `post.published` bullet) + +**Interfaces:** +- Consumes: `buildMarkpubContent(html: string): Record | null` and `MARKPUB_MARKDOWN_TYPE` from `./content-shape.js` (Task 2). +- Produces: `ghostPostToRecord` output gains an optional `content` key. No signature change — callers (`src/lib/atproto.ts:225`) need no edits. + +- [ ] **Step 1: Write the failing test** + +Create `src/lib/transform.test.ts`: + +```ts +import { test } from 'node:test'; +import assert from 'node:assert/strict'; + +import { ghostPostToRecord, type GhostPostLike } from './transform.js'; +import { MARKPUB_MARKDOWN_TYPE } from './content-shape.js'; + +const PUBLICATION_URI = 'at://did:plc:example/site.standard.publication/self'; + +const basePost: GhostPostLike = { + id: '64f0000000000000000000ab', + title: 'Hello', + slug: 'hello', + status: 'published', + published_at: '2026-07-06T10:00:00.000Z', + html: '

Hi

Body text.

', +}; + +test('record carries the markpub content member when the post has HTML', () => { + const record = ghostPostToRecord(basePost, PUBLICATION_URI, null); + const content = record.content as { $type: string; text: { markdown: string } }; + assert.equal(content.$type, MARKPUB_MARKDOWN_TYPE); + assert.match(content.text.markdown, /## Hi/); + // textContent is untouched by this feature — both representations coexist. + assert.match(String(record.textContent), /Body text\./); +}); + +test('record omits content when the post has no HTML', () => { + const record = ghostPostToRecord({ ...basePost, html: null }, PUBLICATION_URI, null); + assert.ok(!('content' in record)); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +npm test +``` + +Expected: FAIL — `record carries the markpub content member…` fails because `record.content` is `undefined` (`transform.ts` does not build it yet). The Task 2 suite and the omit-case test pass. + +- [ ] **Step 3: Wire the member into `ghostPostToRecord`** + +In `src/lib/transform.ts`, add the import after the existing ones (line 8): + +```ts +import { buildMarkpubContent } from './content-shape.js'; +``` + +Then extend the optional-field block (currently lines 55-63) — add the two `content` lines after the `coverImageBlob` line, keeping the aligned style: + +```ts + const description = post.custom_excerpt || post.excerpt; + if (description) record.description = description.slice(0, 3000); + if (tags.length) record.tags = tags; + if (textContent) record.textContent = textContent; + if (post.updated_at && post.updated_at !== post.published_at) { + record.updatedAt = post.updated_at; + } + if (coverImageBlob) record.coverImage = coverImageBlob; + + // Post body as at.markpub.markdown — the document content union is open + // (`closed: false`), so no lexicon registration is needed; the shared $type + // is what markpub-aware readers key on. + const content = buildMarkpubContent(post.html ?? ''); + if (content) record.content = content; +``` + +Also update the stale file-header comment (line 5, `// Real implementation TBD — porting from the parent bridge.` is long obsolete — delete that line). + +- [ ] **Step 4: Run tests and typecheck** + +```bash +npm test && npm run typecheck +``` + +Expected: PASS — `tests 12`, `fail 0`; typecheck exit 0. + +- [ ] **Step 5: README note** + +In `README.md` line 15, extend the webhook bullet. Old text: + +```markdown +- On `post.published` / `post.published.edited`: fetches the post, uploads its cover image as a blob, writes a `site.standard.document` record, and updates the post's per-post `codeinjection_head` so its `` carries the link tag — all without touching Ghost's editor UI +``` + +New text: + +```markdown +- On `post.published` / `post.published.edited`: fetches the post, uploads its cover image as a blob, writes a `site.standard.document` record (the post body travels as [`at.markpub.markdown`](https://github.com/AramZS/markpub.at) content, converted from Ghost's HTML), and updates the post's per-post `codeinjection_head` so its `` carries the link tag — all without touching Ghost's editor UI +``` + +- [ ] **Step 6: Commit** + +```bash +git add src/lib/transform.ts src/lib/transform.test.ts README.md +git commit -m "feat: attach markpub content member to document records + +Co-Authored-By: Claude Fable 5 " +``` + +--- + +## Post-merge migration (operational, not a code task) + +After the next deploy, run once per site to rewrite existing records in place +(`putRecord` with stable rkeys — the Ghost post id): + +```bash +docker compose exec app gs backfill +``` + +Spot-check one record afterward: `content.$type` is `at.markpub.markdown`, +`content.text.markdown` reads sensibly, `textContent` unchanged. diff --git a/docs/superpowers/specs/2026-07-06-markpub-content-design.md b/docs/superpowers/specs/2026-07-06-markpub-content-design.md index 53bf058..c589f86 100644 --- a/docs/superpowers/specs/2026-07-06-markpub-content-design.md +++ b/docs/superpowers/specs/2026-07-06-markpub-content-design.md @@ -47,7 +47,9 @@ plugin. Consequences accepted during brainstorming: directly was checked and ruled out; see above.) 2. **Tests: mirror gifthood.** Vendor the two markpub lexicon JSONs as fixtures and validate the emitted member with `@atproto/lexicon`. This - introduces the repo's first test setup (vitest). + introduces the repo's first test setup — Node's built-in `node:test` + runner driven through the already-present `tsx`, exactly how gifthood's + suite runs. Zero new framework dependencies. 3. **One targeted kg-card rule**, everything else turndown defaults. Ghost wraps every editor card in `
`; a single rule for captioned image figures covers the common case without a @@ -109,12 +111,18 @@ search/preview consumers; the markdown serves renderers). ## 3. Dependencies and typing - Runtime deps: `turndown`, `turndown-plugin-gfm`. -- Dev deps: `@types/turndown`, `vitest`, `@atproto/lexicon` (versions pinned - from `npm view version` at implementation time). -- `turndown-plugin-gfm` ships no types: add an ambient declaration to the - existing `src/types.d.ts` (same loose-shim pattern as `@tryghost/admin-api` - and `html-to-text`). -- New package script: `"test": "vitest run"` — the repo's first. +- Dev dep: `@atproto/lexicon` only (versions pinned from + `npm view version` at implementation time). +- Ambient declarations for **both** `turndown` and `turndown-plugin-gfm` in + the existing `src/types.d.ts` (same shim pattern as `@tryghost/admin-api` + and `html-to-text`). This deliberately skips `@types/turndown`: its surface + leans on DOM lib types, and this repo's tsconfig `lib` is ES2022-only. +- New package script — the repo's first test runner: + `"test": "node --import tsx --test \"src/lib/**/*.test.ts\""`. +- Build hygiene: a new `tsconfig.build.json` (extends `tsconfig.json`, + excludes `src/**/*.test.ts`) keeps test files out of `dist/` and the Docker + runtime image; the `build` script points at it while `typecheck` keeps + checking everything including tests. ## 4. Vendored lexicons + tests