diff --git a/docs/api-reference.md b/docs/api-reference.md index 6062706..d9fc3f1 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -40,7 +40,10 @@ This page is for lookup, not for the full design explanation. |----------|-------------| | `parse(input)` | Parse wikitext into a `WikistRoot` AST using the cheapest default tree lane. | | `parseWithDiagnostics(input)` | Parse to `{ tree, diagnostics }` with the default tolerant materialization and preserved diagnostics. | +| `parseWithRecovery(input)` | Parse to `{ tree, diagnostics, recovered }` with the same default tree as `parseWithDiagnostics()` plus an explicit recovery summary. | | `parseStrictWithDiagnostics(input)` | Parse to `{ tree, diagnostics }` with the conservative source-strict materialization. | +| `analyze(input, options?)` | Analyze source into replayable findings `{ source, events, diagnostics, recovery? }` without materializing a tree. | +| `materialize(findings, options?)` | Build a tree from previously analyzed findings under the requested `TreeMaterializationPolicy`. | | `stringify(tree)` | Serialize a wikist tree back to wikitext. Not yet implemented. | | `events(input)` | Full event stream (block + inline), with source-backed text ranges and optional diagnostics. | | `outlineEvents(input)` | Block-only event stream over the same source-backed structure. | diff --git a/docs/architecture.md b/docs/architecture.md index 9dbd6eb..5aa5d30 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -10,9 +10,10 @@ readers do not have to parse one very large note to find one specific idea. ## Start here -- [docs/architecture/choosing-a-tree.md](./architecture/choosing-a-tree.md) - Explains the real caller-facing choice between the cheap tree, the default - tolerant diagnostics lane, and the conservative diagnostics lane. +- [docs/architecture/choosing-a-parser-result.md](./architecture/choosing-a-parser-result.md) + Explains the real caller-facing choice between the default tree family, the + conservative tree lane, the planned `analyze()` findings lane, and the + still-exploratory policy lane. - [docs/architecture/pipeline.md](./architecture/pipeline.md) Explains the tokenizer -> block parser -> inline parser -> consumer flow. - [docs/architecture/malformed-input.md](./architecture/malformed-input.md) @@ -41,19 +42,25 @@ layer: - `events()` for the full event stream - `parse()` and related helpers for tree materialization -The tree APIs are easiest to understand as two separate choices: +The result APIs are easiest to understand as a sequence of choices: ```text -1. do you want diagnostics preserved? -2. if yes, which final tree policy do you want? +1. do you want a tree immediately, or `analyze()` first? +2. if you start with `analyze()`, do you later want a package-owned or caller-owned + materialization policy? +3. if you want a tree immediately, do you want the default recovery-applying + tree family, or the conservative no-applied-recovery tree? +4. if you want the default tree family, do you also want diagnostics preserved? ``` -That is why the current public lanes are: +That is why the current tree-first lanes are: - `parse()` - `parseWithDiagnostics()` -- `parseStrict()` - `parseWithRecovery()` +- `parseStrictWithDiagnostics()` + +And the planned findings-first lane is `analyze()`. ## Why the split docs exist @@ -369,7 +376,7 @@ This is the practical contract to carry into downstream code. 3. `parse()` and `parseWithDiagnostics()` keep the default HTML-like materialization. They are the right tree lanes when a caller wants a stable structural overlay that still tolerates malformed committed constructs. -4. `parseStrict()` is intentionally more conservative. It may collapse a +4. `parseStrictWithDiagnostics()` is intentionally more conservative. It may collapse a malformed region back to source-backed text, including outer wrappers that only existed because tolerant recovery inferred them. It is not the lane to use as the canonical structural overlay. @@ -405,7 +412,7 @@ session.events(); // full event stream session.outline(); // block-only events (cheap structural overlay) session.parse(); // full AST session.parseWithDiagnostics(); // default AST + diagnostics -session.parseStrict(); // conservative AST + diagnostics +session.parseStrictWithDiagnostics(); // conservative AST + diagnostics // Streaming session.write(chunk); // append-only streaming input @@ -420,7 +427,7 @@ Session is built *on top of* the pipeline, not replacing it. Each tier adds capability without retroactive redesign: - **Core**: `createSession(source)` with `.events()`, `.outline()`, - `.parse()`, `.parseWithDiagnostics()`, and `.parseStrict()`. Caches the last + `.parse()`, `.parseWithDiagnostics()`, and `.parseStrictWithDiagnostics()`. Caches the last parse result. - **Streaming**: `.write(chunk)` for append-only streaming, `.drainStableEvents()` for stable prefix consumption. diff --git a/docs/corpus-matrix.md b/docs/corpus-matrix.md index 868d13a..0a3f337 100644 --- a/docs/corpus-matrix.md +++ b/docs/corpus-matrix.md @@ -115,8 +115,8 @@ Each category should primarily test one public contract. | Token boundaries and source slicing | `tokens()` | | Block structure stability | `outlineEvents()`, `events()`, `parse()` | | Inline nesting and commitment | `events()` | -| Diagnostics presence and codes | `events({ include_diagnostics: true })`, `parseWithDiagnostics()` | -| Materialization policy differences | `parseWithDiagnostics()`, `parseStrict()`, `parseWithRecovery()` | +| Diagnostics presence and codes | `events({ diagnostics: true })`, `parseWithDiagnostics()` | +| Materialization policy differences | `parseWithDiagnostics()`, `parseStrictWithDiagnostics()`, `parseWithRecovery()` | | Session cache equivalence | `createSession()` APIs | | Never-throw invariant | property tests and reduced fuzz fixtures | diff --git a/docs/diagnostics-first-redesign.md b/docs/diagnostics-first-redesign.md index add43fc..7a509cf 100644 --- a/docs/diagnostics-first-redesign.md +++ b/docs/diagnostics-first-redesign.md @@ -116,42 +116,49 @@ default and `parseStrictWithDiagnostics()` names the conservative lane more hone still makes materialization feel like a parser-owned truth instead of a consumer policy layered over the same syntax findings. -### The tree builder currently owns recovery-shape policy +### The tree builder is mid-transition toward explicit materialization policy -[tree_builder.ts](tree_builder.ts) currently exposes `TreeBuildMode` as -`'strict' | 'loose'` and documents those modes as recovery-shape policies. +[tree_builder.ts](tree_builder.ts) now exposes +`TreeMaterializationPolicy.DEFAULT_HTML_LIKE` and +`TreeMaterializationPolicy.SOURCE_STRICT` as the public materialization names. +The older build-mode names are gone; the tree builder takes the +materialization policy directly. -That creates two problems: +That is a real improvement, but one tension remains: -- tree materialization policy is presented as if it were part of parsing truth -- diagnostics and recovered tree shape are coupled in one API family +- tree materialization policy is now named more honestly +- the public API is still mostly tree-first instead of findings-first -The addition of `buildTreeWithLooseDiagnostics()` helps operationally because -it separates `recovered` from `diagnostics`, but it still keeps the public -story centered on parser-owned recovery shape. +So the problem has narrowed. The public story is no longer centered on strict +versus loose parser truth, but the next design step is still to make findings a +first-class surface rather than only something tree wrappers consume. -### The current docs teach the drift as architecture +### The docs are now closer, but the long-term direction still matters -[docs/architecture.md](docs/architecture.md) and [readme.md](readme.md) -currently describe `strict` and `loose` as top-level parser choices and teach -the tree lanes as the main way to think about malformed input. +[docs/architecture.md](docs/architecture.md), [readme.md](readme.md), and the +parser-result notes now describe the default tree family, the conservative +tree lane, and the planned findings-first direction more accurately. -That is the opposite of the intended mental model. It teaches readers that the -parser owns recovery semantics, when the intended design is that the parser -owns diagnostics and continuation, and consumers own recovery responses. +That fixes the biggest naming drift. What is still useful in this redesign note +is the argument that even a well-named tree-first API is not the end state. The +intended mental model is still that the parser owns diagnostics and +continuation, while consumers ultimately own how recovery findings should be +materialized or surfaced. -### Tests currently lock in parser-owned recovery semantics +### Tests now anchor the shared-findings model more directly [parse_test.ts](parse_test.ts) and [session_test.ts](session_test.ts) assert that: -- `parseWithDiagnostics()` and `parseWithRecovery()` are separate wrappers even - though they now share the same default tree shape -- strict versus loose tree shape is a parser-level distinction -- recovery-specific wrapper nodes are part of the parser's promised result +- `parseWithDiagnostics()` and `parseWithRecovery()` share the same default tree + and diagnostics +- `parseStrictWithDiagnostics()` shares the same diagnostics as the default + diagnostics lane and only changes final materialization +- inline recovered regions can collapse to text without erasing surrounding + block structure -Those tests are useful because they show the exact behavioral drift. They will -need to move as the API is realigned. +Those assertions are closer to the intended architecture. They keep the result +model honest while leaving room for a future findings-first public lane. ## Design Distinction To Restore diff --git a/docs/examples.md b/docs/examples.md index 25e0d06..ff7c3df 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -55,6 +55,25 @@ console.log(result.diagnostics.map((diagnostic) => diagnostic.code)); This is the best current fit for the "recover on my behalf, but tell me what you did" style of usage. +## Keep the same default tree but branch on recovery explicitly + +Use `parseWithRecovery()` when you want the same default tree and diagnostics +as `parseWithDiagnostics()`, but you also want a cheap boolean for control +flow. + +```ts +import { parseWithRecovery } from '@okikio/wikitext'; + +const result = parseWithRecovery('Paragraph with note'); + +if (result.recovered) { + console.log(result.diagnostics.map((diagnostic) => diagnostic.code)); +} +``` + +This is useful when your tool wants the parser's default tolerant tree but does +not want to re-derive recovery status from `diagnostics.length > 0` each time. + ## Ask for the conservative tree instead Use `parseStrictWithDiagnostics()` when diagnostics matter and you want the final tree to be @@ -72,28 +91,84 @@ console.log(result.diagnostics.length > 0); This is useful for inspection or linting flows that do not want the final tree to keep as much repaired structure. +## Analyze once, materialize many + +Use `analyze()` + `materialize()` when you want to inspect parser findings +before deciding how (or whether) to build a tree, or when you want to build +more than one tree from the same parse. + +```ts +import { analyze, materialize, TreeMaterializationPolicy } from '@okikio/wikitext'; + +const findings = analyze('Paragraph with note'); + +// Inspect without materializing a tree. +for (const entry of findings.recovery ?? []) { + console.log(entry.kind, entry.code, entry.policies); +} + +// Build the default tree once. +const tolerant = materialize(findings); + +// Build a conservative tree from the same findings without reparsing. +const strict = materialize(findings, { + policy: TreeMaterializationPolicy.SOURCE_STRICT, +}); + +console.log(tolerant.tree.children[0]?.type); // 'paragraph' +console.log(strict.tree.children[0]?.type); // 'paragraph' with text-only children +``` + +The `findings.recovery` array lists the parser's structural decisions with a +narrow taxonomy (`missing-close`, `unterminated-opener`, `unclosed-table`, +`mismatched-exit`, `orphan-exit`, `eof-autoclose`). Each entry also lists the +materialization policies that can change the final shape, so tooling can +decide when policy choice is meaningful. + +## Skip the recovery list when you only want events + +Pass `{ recovery: false }` to `analyze()` to drop the recovery derivation when +you only care about events and diagnostics. + +```ts +import { analyze } from '@okikio/wikitext'; + +const findings = analyze('{|\n| Cell', { recovery: false }); + +console.log(findings.diagnostics.length > 0); +console.log(findings.recovery); // undefined +``` + ## Compare the current tree lanes on one malformed input The easiest way to understand the tree differences is to run the same input through more than one lane. ```ts -import { parse, parseStrictWithDiagnostics, parseWithDiagnostics } from '@okikio/wikitext'; +import { + parse, + parseStrictWithDiagnostics, + parseWithDiagnostics, + parseWithRecovery, +} from '@okikio/wikitext'; const input = 'Paragraph with note'; -const fast_tree = parse(input); -const recovery_tree = parseWithDiagnostics(input); +const default_tree = parse(input); +const default_diagnostics = parseWithDiagnostics(input); +const default_recovery = parseWithRecovery(input); const conservative_tree = parseStrictWithDiagnostics(input); -console.log(fast_tree.children[0]?.type); -console.log(recovery_tree.tree.children[0]?.type); +console.log(default_tree.children[0]?.type); +console.log(default_diagnostics.tree.children[0]?.type); +console.log(default_recovery.recovered); console.log(conservative_tree.tree.children[0]?.type); -console.log(recovery_tree.diagnostics.map((diagnostic) => diagnostic.code)); +console.log(default_diagnostics.diagnostics.map((diagnostic) => diagnostic.code)); ``` This kind of comparison is useful when you are deciding whether your tool wants -speed first, tolerant structure first, or conservative diagnostics first. +the cheapest default tree, the default tree with diagnostics, the same default +tree with an explicit recovery summary, or the conservative source-strict tree. ## Resolve a diagnostic back to the tree