diff --git a/docs/early-plan.md b/docs/early-plan.md new file mode 100644 index 0000000..2a95904 --- /dev/null +++ b/docs/early-plan.md @@ -0,0 +1,970 @@ +# Plan: Wikitext Parser — Event-Driven AST Library + +## TL;DR + +Build `@okikio/wikitext`, an event-stream-first wikitext source parser for +Deno/npm. Syntax-first deterministic parse: correct structural model for all +documented wikitext constructs, with strong recovery (never throw). MediaWiki +behavioral quirk-matching is a future profile, not an MVP goal. Events are the +fundamental interchange format — AST ("wikist") is built on top. Architecture +supports sync pull, async chunked streaming, progressive block output, +incremental reparsing, multiple output modes (tokens → events → AST → direct +compilation), and extensibility hooks — all without changing the public API. +All modules accept `TextSource` (plain `string` satisfies the interface). +Range-first events carry offset ranges, not extracted strings. +Flat file layout; every module exports reusable utilities. + +## Architecture: Events as the Interchange Layer + +The key architectural insight (from micromark, pulldown-cmark, lol-html): +**events, not AST, are the fundamental output**. Everything else is a consumer. + +``` +TextSource ──► Tokenizer ──► Event Stream ──► [Consumer] + │ │ + │ charCodeAt scanner ├─► buildTree() → WikistRoot + │ Generator ├─► compileHtml() → string + │ ├─► filterEvents() → events + │ ├─► directConsumer → user callback + │ └─► treeBuilder → incremental update + │ + └─► also exposes raw token stream for lowest-cost consumers +``` + +The pipeline accepts any `TextSource` (plain `string` satisfies the interface). +Events are range-first: text events carry offset ranges into the source, not +extracted strings. Consumers call `slice(source, evt)` to resolve text on demand. + +### Event types + +Events are enter/exit pairs with position data, mirroring SAX/StAX. Events +are **range-first**: text events carry offset ranges, not extracted strings. + +- `enter(nodeType, props)` — opening a node (e.g., `enter("heading", {level:2})`) +- `exit(nodeType)` — closing a node +- `text(startOffset, endOffset, position)` — literal text content (resolve via + `slice(source, startOffset, endOffset)`) +- `token(tokenType, start, end, position)` — raw token (lowest level) + +This gives callers a choice: +1. **Token stream only** — cheapest, no structure, good for search/grep +2. **Event stream** — structured enter/exit pairs, no allocation, good for + streaming transforms and direct compilation +3. **AST** — full tree, good for complex transforms, filtering, round-trip +4. **Direct compilation** — events → HTML without building a tree + +## Parser Contracts + +Four invariants that every API path must satisfy. Defined early because they +constrain every implementation decision that follows. + +### 1. Event well-formedness (stack discipline) + +Every `enter(X)` event has a matching `exit(X)`. Events form a properly nested +stack — no interleaving, no orphaned exits. This holds for all event modes +(outline, full, progressive). Consumers can rely on this for correct tree +construction, streaming HTML emission, and event filtering. + +Formally: at any point in the event stream, the sequence of `enter`/`exit` +events forms a valid prefix of a well-parenthesized string. + +### 2. Position semantics (UTF-16 code unit offsets) + +`position.offset` is a **UTF-16 code unit index** into the original JS input +string. This matches `string.charCodeAt(i)`, `string[i]`, and `string.slice()`. +Not UTF-8 bytes, not Unicode code points. + +- `line` is 1-indexed (first line = 1). +- `column` is 1-indexed, counting UTF-16 code units from start of line. +- `offset` is 0-indexed, counting UTF-16 code units from start of input. + +If a consumer needs UTF-8 byte offsets (e.g., for interfacing with native +tooling), expose them opt-in via `node.data?.utf8Offset` (computed lazily). + +This aligns with LSP position semantics (`utf-16` encoding mode) and avoids +subtle bugs with non-ASCII content (CJK articles, diacritics, emoji). + +### 3. Never-throw guarantee + +The parser never throws on any input. Malformed wikitext produces a valid +wikist tree (with error recovery). Optionally, recovery points emit +`{ type: "error", message, position }` events that consumers can log or ignore. + +This is a continuous invariant enforced from Phase 2 onward — not a Phase 8 +hardening pass. + +### 4. Determinism + +Same input + same config → same events, same tree. No randomness, no +dependency on global state, no Date.now() in output. This enables snapshot +testing, round-trip invariants, and reproducible corpus regression. + +## Streaming Modes + +Three named modes, all built on the same event core. Each preserves event +well-formedness within its scope. + +### Mode A: Outline events (block-only) + +`outlineEvents(source: TextSource): Generator` + +Emits only block-level structure: `Heading`, `List`, `Table`, `Paragraph` +boundaries. Inline content stays as opaque text ranges (raw text events). +No inline parsing cost. + +Use case: table of contents, section index, document structure extraction. + +``` +for (const evt of outlineEvents(input)) { + if (evt.type === "enter" && evt.nodeType === "heading") toc.push(evt) +} +``` + +### Mode B: Full events + +`events(source: TextSource): Generator` + +Complete event stream — block structure plus inline enrichment (bold, italic, +links, templates, etc.). All nested enter/exit pairs. This is the default. + +### Mode C: Progressive blocks + +`parseChunked(chunks: AsyncIterable): AsyncGenerator` + +Yields *completed block-level nodes* as soon as their closing boundary arrives. +In-order within the document, but progressive over time. Each yielded node is +a fully-parsed subtree (block + inline). + +Use case: streaming rendering, progressive wiki preview, LLM streaming output. + +``` +for await (const node of parseChunked(stream)) { + renderNode(node) // heading arrives when closing "==" is seen +} +``` + +### Sync pull (primary, MVP) + +``` +function* events(source: TextSource): Generator +function* outlineEvents(source: TextSource): Generator +function parse(source: TextSource): WikistRoot +``` + +### Async pull (chunked input — Phase 6) + +``` +async function* asyncEvents(chunks): AsyncGenerator +async function* parseChunked(chunks): AsyncGenerator +``` + +Tokenizer maintains a small carry buffer across chunk boundaries. Block parser +flushes complete blocks immediately. + +### Push (SAX-style callbacks — Phase 6) + +``` +const parser = createParser({ onEvent(evt) { ... } }) +parser.write(chunk) +parser.end() +``` + +Thin adapter over async events — inverts control flow. + +### Deferred inline parsing (lazy tree mode — Phase 6) + +`buildTree(events, { inlineMode: "lazy" })` — inline content stays as `Text` +nodes. `resolveInlines(node)` parses on-demand. "Give me all headings" never +pays for bold/italic in paragraphs. Text nodes store their source range; +resolveInlines re-runs the inline parser on that range. + +### Incremental parsing (edit resilience — Phase 7) + +Goal: reparse only the affected region after an edit, reuse everything else. + +**How it works:** +1. During full parse, record a compact **state snapshot** at each block + boundary: `{ inNowiki, inPre, openTagStack, openTemplateDepth, inTable }`. +2. On edit, caller provides `(oldTree, editRange, newText)`. +3. Find affected blocks: any block whose source range overlaps `editRange`. +4. Walk backward from the first affected block to the nearest boundary whose + state snapshot is "neutral" (all stacks empty, no open spans). This is the + actual reparse start — it catches cases where a template or tag spans + multiple blocks (e.g., `{{ ... \n ... }}`). +5. Re-tokenize from the neutral boundary through the dirty range. +6. Re-parse affected blocks. Compare new block-start sequence to old one; + expand dirty range if structural boundaries shifted. +7. Splice new blocks into old tree, reuse untouched subtrees by reference. +8. Return new root. + +**Why state snapshots matter:** block boundaries alone are not sufficient. +Templates can span lines (`{{ ... \n ... }}`). Nowiki/pre/ref blocks suppress +parsing across line boundaries. Tables leave the parser in a non-neutral state +at line boundaries. Without state snapshots, an edit that closes a template +or opens a nowiki block can silently change interpretation of later blocks. + +**File:** `incremental.ts` — `reparseIncremental(oldTree, edit) → WikistRoot` + +## Key Decisions + +- **Spec name**: "wikist" (Wiki Syntax Tree), following mdast/hast/xast pattern +- **File layout**: Flat files at root alongside mod.ts — no src/ folder. All + functions exported as utilities. +- **Scope**: Syntax-first deterministic parse for all documented wikitext + constructs. Exceeds mwparserfromhell structurally (proper lists, tables, + semantic formatting, image/category/definition-list). Source parser only — no + template expansion or rendering. MediaWiki behavioral quirk-matching (exact + apostrophe heuristics, edge case rendering parity) deferred to a future + "mediawiki" profile. +- **Events-first**: Events are the interchange layer. AST is a consumer. Three + named event modes: `outlineEvents()` (block-only), `events()` (full), + `parseChunked()` (progressive completed blocks). All preserve well-formedness. +- **Range-first events**: Text/token events carry `startOffset`/`endOffset` into + the `TextSource`, not extracted `value` strings. `slice(source, evt)` resolves + on demand. Avoids per-event string allocation. +- **TextSource abstraction**: Minimal interface (`length`, `slice`, `charCodeAt`, + optional `iterSlices`) abstracts the backing text store. Plain `string` + satisfies it. Rope trees, CRDTs, and append buffers can implement it for + zero-copy access. Defined in Phase 1; all modules accept `TextSource`. +- **Positions**: UTF-16 code unit offsets (matching JS string indexing). Line + and column are 1-indexed. UTF-8 byte offsets opt-in via `data` slots. +- **Token representation**: Tokens carry start/end offsets into the source, not + `value` strings. A `slice(source, token)` helper resolves strings on demand. + Avoids per-token allocation and V8 sliced-string retention hazard. +- **Session API**: Stateful wrapper (`createSession(source)`) built on stateless + pipeline. Phase 5: basic (events/outline/parse). Phase 6: streaming + (write/drain). Phase 7: incremental (applyChanges/PositionMap). +- **Stability frontier**: During streaming, UTF-16 offset up to which events are + guaranteed stable. Stable prefix grows monotonically as input appends. +- **PositionMap**: Old-offset → new-offset mapping returned by `applyChanges()`. + Covers ~80% of anchor use cases without a dedicated Anchor API. +- **Conflict node**: Reserved `type: "conflict"` in wikist spec. Not produced by + core parser — intended for collab/merge tooling (jj-inspired). +- **Hybrid editing design constraint**: Text is truth, structure is overlay. + Local markup hiding, bounded reflow. We are not building an editor — we are + providing the structural API an editor needs. +- **CST vs AST**: AST with position info for source mapping. Full CST possible + later — events already carry the information. +- **Research and docs**: Live in `docs/` folder. +- **Perf target**: Best-in-class JS throughput on real Wikipedia corpora, + predictable latency, correctness. Architecture stays open so a WASM backend + could replace a hot path without changing the public API. +- **Profiles**: Named presets for parser config. `syntax` (MVP default) uses + deterministic rules. `mediawiki` (Phase 8) matches MediaWiki's exact quirks. + Profiles are sugar over feature gates — not a separate pipeline. +- **Extension model**: Construction-time specialization — feature gates produce + a specialized parser instance. Extensions run as event enrichment passes, not + in the tokenizer hot loop. +- **Hardening early**: Fuzz from Phase 3, corpus regression from Phase 5. + "Never throw" is a continuous invariant, not a Phase 8 bolt-on. + +## Capabilities Matrix + +| Capability | MVP (Phases 0-5) | Advanced (Phases 6-8) | +|------------|-------------------|-----------------------| +| Sync pull parse | Yes | — | +| Full AST (wikist) | Yes | — | +| Token stream (offset-based) | Yes | — | +| Full event stream (`events()`) | Yes | — | +| Outline event stream (`outlineEvents()`) | Yes | — | +| Range-first events (no value strings) | Yes | — | +| `TextSource` abstraction | Yes | — | +| Filter/visit API | Yes | — | +| Round-trip stringify | Yes | Minimal-diff mode | +| Error recovery (never-throw) | Yes | — | +| Source positions (UTF-16 offset + line/col) | Yes | — | +| unist compatibility | Yes | — | +| Fuzz testing (never-throw invariant) | Phase 3 onward | Scaled up | +| Corpus regression snapshots | Phase 5 onward | Scaled up | +| `createSession()` (basic) | Phase 5 | — | +| Async chunked streaming | — | Phase 6 | +| Push/SAX-style API | — | Phase 6 | +| Progressive blocks (`parseChunked()`) | — | Phase 6 | +| Deferred inline parsing (lazy tree) | — | Phase 6 | +| `session.write()` + stability frontier | — | Phase 6 | +| Incremental reparsing (state snapshots) | — | Phase 7 | +| `session.applyChanges()` + `PositionMap` | — | Phase 7 | +| Edit coalescing | — | Phase 7 | +| Direct HTML compilation | — | Phase 7 | +| Selective serialization (min-diff) | — | Phase 7 | +| Extension hooks (construction-time) | — | Phase 8 | +| Profiles (`syntax`, `mediawiki`) | `syntax` (default) | `mediawiki` (Phase 8) | +| Template resolver interface | — | Phase 8 | +| unified plugin pair | — | Phase 8 | +| `Conflict` node type | — | Phase 8 (optional) | +| Anchor API | — | Phase 8 or separate pkg | + +## mwparserfromhell Gap Analysis (structural advantages) + +mwparserfromhell has 11 node types: Text, Heading, Template, Argument, Wikilink, +ExternalLink, HTMLEntity, Comment, Tag, Attribute, Parameter. + +**Structural gaps we close (syntax-first scope):** +- Lists: mwparserfromhell parses `# item` as flat Tag+Text. We model + `List > ListItem` with proper nesting and definition lists. +- Tables: mwparserfromhell has Tag-like table nodes but no first-class table + model (rows/cells/caption). We model full + `Table > TableCaption / TableRow > TableCell` hierarchy. +- Bold/Italic: mwparserfromhell parses `''`/`'''` into style tags by default + (skip_style_tags option exists) but does not model them as a semantic + emphasis layer. We have `Bold`, `Italic`, `BoldItalic` parent nodes. +- Image/Category: mwparserfromhell treats `[[File:...]]` and `[[Category:...]]` + as generic Wikilink with no namespace dispatch. We have distinct `ImageLink` + and `CategoryLink`, with leading-colon escape (`[[:Category:Foo]]` → + Wikilink). +- Redirect: No distinct node in mwparserfromhell. We have `Redirect`. +- Behavior switches: `__TOC__`, `__NOTOC__`, etc. — not modeled in + mwparserfromhell. We have `BehaviorSwitch`. +- Parser functions: `{{#if:...}}` (identified by `#` prefix) treated as + Template in mwparserfromhell. We have `ParserFunction`. Variable-style + magic words (`{{PAGENAME}}`) parse as `Template` by default; profiles + reclassify. +- Streaming/incremental: mwparserfromhell is batch-only. We stream. + +--- + +## Phase 0 — Rewrite Copilot Instructions + +Swap undent-specific content for wikitext-parser guidance. Keep the structural +patterns (tables, checklists, style) already in place. + +**What changes:** +- `copilot-instructions.md`: project description, commands, architecture + overview (event layer, streaming modes), breaking changes checklist +- `typescript.instructions.md`: add wikist type naming conventions + (`WikistNode`, `WikitextToken`, `WikitextEvent`, `WikistRoot`, etc.) +- `testing.instructions.md`: replace undent edge cases with wikitext edge cases + (unclosed tags, apostrophe runs, nested templates, mixed list markers, + malformed tables, round-trip invariants, event-stream assertions) +- `benchmarking.instructions.md`: update competitor list (wtf_wikipedia, + wikiparser-node as JS benchmarks) and benchmark modes (token-only, + events-only, full-AST, round-trip) +- Leave `changelog-commits`, `pull-requests`, `code-review`, + `markdown-writing`, `ascii-diagrams` untouched — they're generic + +**Also in this phase:** +- `changelog.md` — reset for new project +- `readme.md` — stub rewrite (expand later) +- `scripts/build_npm.ts` — update package name/description + +**Files to modify:** +- `.github/copilot-instructions.md` +- `.github/instructions/typescript.instructions.md` +- `.github/instructions/testing.instructions.md` +- `.github/instructions/benchmarking.instructions.md` +- `changelog.md` +- `readme.md` +- `scripts/build_npm.ts` + +## Phase 1 — AST Specification ("wikist") + Event Types + TextSource + +Define wikist node types (extending unist), the event interface that +produces them, and the `TextSource` abstraction. The event types must be +defined first — they are the contract between tokenizer/parsers and all +consumers. Events are **range-first**: text/token events carry offset +ranges, not extracted strings. + +### TextSource interface + +``` +interface TextSource { + readonly length: number; + charCodeAt(index: number): number; + slice(start: number, end: number): string; + iterSlices?(start: number, end: number): Iterable; // optional +} +``` + +Plain `string` satisfies this interface. Rope trees, CRDTs (Yjs `Y.Text`), +and append buffers can implement it too. All tokenizer/parser modules accept +`TextSource` instead of `string`. + +### Event interface + +``` +WikitextEvent = + | { type: "enter"; nodeType: string; props: Record; + position: Position } + | { type: "exit"; nodeType: string; position: Position } + | { type: "text"; startOffset: number; endOffset: number; + position: Position } + | { type: "token"; tokenType: TokenType; start: number; end: number; + position: Position } +``` + +Text and token events carry offset ranges into the `TextSource` — not +extracted `value` strings. Consumers call `slice(source, evt.startOffset, +evt.endOffset)` to resolve text on demand. + +### Node types (26+) + +| Category | Nodes | Parent? | +|----------|-------|---------| +| Root | `Root` | Parent | +| Block | `Heading`, `Paragraph`, `ThematicBreak`, `Preformatted` | Parent (ThematicBreak is leaf) | +| List | `List`, `ListItem`, `DefinitionList`, `DefinitionTerm`, `DefinitionDescription` | Parent | +| Table | `Table`, `TableCaption`, `TableRow`, `TableCell` | Parent | +| Inline formatting | `Bold`, `Italic`, `BoldItalic` | Parent | +| Links | `Wikilink`, `ExternalLink`, `ImageLink`, `CategoryLink` | Parent | +| Templates | `Template`, `TemplateArgument`, `Argument` | Parent | +| HTML | `HtmlTag`, `HtmlEntity` | HtmlTag=Parent, HtmlEntity=Literal | +| Literal content | `Text`, `Nowiki`, `Comment` | Literal | +| Special | `Redirect`, `Signature`, `MagicWord`, `BehaviorSwitch`, `ParserFunction`, `Break`, `Gallery`, `Reference` | varies | + +### unist compatibility + +Every node has `type: string` and optional `position: { start: Point, end: Point }`. +Parent nodes have `children: WikistNode[]`. Literal nodes have `value: string`. +Discriminated union on `type` field enables exhaustive pattern matching. + +### Type guards and builders + +Export type guard functions (`isHeading()`, `isTemplate()`, etc.) and builder +helpers (`heading(level, children)`, `text(value)`, etc.) for ergonomic tree +construction and filtering. + +**Files:** +- `text_source.ts` — `TextSource` interface, `slice(source, start, end)` helper +- `ast.ts` — all type definitions, discriminated union, type guards, builders. + `Conflict` type reserved in union (no guards/builders in MVP). +- `events.ts` — `WikitextEvent` union type, range-first event constructors, + event type guards. Shared by tokenizer, parsers, and all consumers. + +## Phase 2 — Tokenizer (Iterator Core) + +Character-level scanner yielding typed tokens via `Generator`. +charCodeAt-based scanning over `TextSource` for performance. The tokenizer is +the lowest layer; it feeds the event emitter. + +### Token categories + +- **Structural (line-start):** HEADING_MARKER, HR, TABLE_START, TABLE_END, + TABLE_ROW, TABLE_CELL, TABLE_HEADER_CELL, TABLE_CAPTION, + LIST_BULLET, LIST_NUMBER, LIST_INDENT, LIST_DEFINITION_TERM, + LIST_DEFINITION_DESC +- **Inline delimiters:** APOSTROPHE_RUN, LINK_OPEN, LINK_CLOSE, PIPE, + EXT_LINK_OPEN, EXT_LINK_CLOSE, TEMPLATE_OPEN, TEMPLATE_CLOSE, + ARGUMENT_OPEN, ARGUMENT_CLOSE, EQUALS +- **Content:** TEXT, WHITESPACE, NEWLINE, EOF +- **Special:** COMMENT, NOWIKI_OPEN, NOWIKI_CLOSE, HTML_TAG_OPEN, + HTML_TAG_CLOSE, HTML_SELF_CLOSE, MAGIC_WORD, BEHAVIOR_SWITCH, + REDIRECT, SIGNATURE + +### Design choices + +- `function* tokenize(source: TextSource): Generator` — main entry +- Track line/column/offset for every token's start and end position +- Pre-scan comment and nowiki regions to protect their content from tokenization +- Apostrophe runs emitted as single APOSTROPHE_RUN(length) token — + disambiguation deferred to inline parser +- Table/list markers only recognized at line start (after optional whitespace) + +### Performance discipline (applied here, influences all phases) + +- **charCodeAt, not charAt** — avoid string allocation per character +- **Offset-based tokens, not string values** — tokens carry `start` and `end` + offsets into the `TextSource`, not a `value` substring. A `slice(source, token)` + helper resolves the string on demand. This avoids per-token allocation and + sidesteps V8's sliced-string retention hazard (a small slice can pin the + entire parent string in memory). +- **Range-first events** — text events carry `startOffset`/`endOffset`, not + `value` strings. Same benefits as offset-based tokens, applied to events. +- **No object reuse across yields** — each yielded token is a fresh, immutable + object. Object reuse in generators is a footgun: consumers retain references + and see mutated data. Fresh small objects are cheap with modern GC; the real + win is avoiding strings, not objects. +- **Single pass** — the tokenizer never backtracks more than a bounded lookahead + (max: length of longest possible marker, which is `{{{` = 3 chars for + argument open, or `