diff --git a/.agents/memory/ACTIVE/PLAN.md b/.agents/memory/ACTIVE/PLAN.md index db0ce79..81fabeb 100644 --- a/.agents/memory/ACTIVE/PLAN.md +++ b/.agents/memory/ACTIVE/PLAN.md @@ -31,6 +31,22 @@ but no streaming or unist compatibility. - HTML rendering (direct compilation is a convenience, not a goal) - MediaWiki behavioral quirk-matching (deferred to "mediawiki" profile) +## Scope discipline + +- Use wikitext as the proving ground for parser primitives before expanding + into a broader profile-driven document engine. +- Treat the parser as one simple workflow step in the larger future system: + its job is to produce correct primitives that downstream transforms, + renderers, editors, and session-based tools can consume. +- The longer-term direction includes other markup and rich-text profiles, + structured CMS blocks, local-first collaboration, offline or local sync, + and LLM-oriented workflows. +- Unified ecosystem support remains desirable during that transition, but as + optional adapters over unist-compatible exports rather than as the core + runtime architecture. +- Do not let that broader direction dilute current work on the wikitext parser + itself. + ## Approach See `docs/architecture.md` for the full pipeline design. In summary: diff --git a/.agents/memory/ACTIVE/PROGRESS.md b/.agents/memory/ACTIVE/PROGRESS.md index 57e1fe1..fbb869a 100644 --- a/.agents/memory/ACTIVE/PROGRESS.md +++ b/.agents/memory/ACTIVE/PROGRESS.md @@ -12,9 +12,15 @@ - Test imports use deno-lint-ignore comments with inline jsr:/npm: specifiers (no deno.json import map) - Benchmarks GC-annotated for allocation-heavy tokenizer paths - State snapshot recording deferred to Phase 7 (TODO comment in block_parser.ts) -- Current task: T06 (Inline parser implementation, Phase 4) — not started +- Current task: Phase 4 complete; Phase 5 orchestration not started +- Longer-term direction: broader profile-driven document engine is recorded, + but current focus remains validating parser primitives through the wikitext + parser first +- Ecosystem direction: unified compatibility stays desirable, but through + optional adapters at the edge while the native runtime model remains the + long-term target - Blockers: none -- Total tests: 448 across 8 test files, 0 failures, 0 compile errors +- Total tests: 479, 0 failures, 0 compile errors ## Completed @@ -116,6 +122,23 @@ custom TextSource impl, property-based round-trips - `deno task test` passes (448 total tests) +- [x] T06: Implement inline parser (Phase 4) + - `inline_parser.ts`: offset-driven inline enrichment generator + - Covers apostrophe emphasis, wikilinks, image/category namespace dispatch, + bracketed and bare external links, templates, parser functions, + triple-brace arguments, comments, behavior switches, signatures, + HTML entities, `
`, ``, ``, and generic HTML tags + - Adjacent block-parser text spans are merged before enrichment so inline + constructs can span token-sized text events + - Position recovery uses line-start tables instead of per-character `Point` + allocation in the hot path + - `inline_parser_test.ts`: focused examples, behavior-switch edge cases, and + property-based invariants + - `mod.ts`: re-exports `inlineEvents()` + - `mod_bench.ts`: now benchmarks token-only, block-events, and full inline + enrichment paths on representative inputs + - `deno task test` passes (479 total tests) + ## Notes for the next agent - Conventions: @@ -149,7 +172,6 @@ - State snapshot recording is deferred to Phase 7 (TODO comment in place) - blockEvents() accepts (source: TextSource, tokens: Iterable) - What's NOT implemented yet: - - inline_parser.ts - parse.ts, tree_builder.ts, stringify.ts, filter.ts - session.ts - Verification commands: diff --git a/.agents/memory/ACTIVE/RISKS.md b/.agents/memory/ACTIVE/RISKS.md index cc842e2..0b55d59 100644 --- a/.agents/memory/ACTIVE/RISKS.md +++ b/.agents/memory/ACTIVE/RISKS.md @@ -41,6 +41,10 @@ `{{SomeTemplate}}` (template). Default: parse all non-`#`-prefixed as Template. Risk: profiles may need a word list, adding configuration burden. +- **Ecosystem adapter pressure**: Reusing unified ecosystem plugins is useful, + but letting unified-style assumptions leak into the hot path could distort + the event-stream-first runtime and incremental design. Mitigation: keep + unified support at the adapter boundary over unist-compatible exports. - **Heading close marker token mismatch (discovered, resolved)**: The tokenizer emits `EQUALS` (not `HEADING_MARKER_CLOSE`) for trailing `==` in headings. The block parser originally checked for `HEADING_MARKER_CLOSE` diff --git a/.agents/memory/ACTIVE/TASKS.md b/.agents/memory/ACTIVE/TASKS.md index 91c288d..1fb228e 100644 --- a/.agents/memory/ACTIVE/TASKS.md +++ b/.agents/memory/ACTIVE/TASKS.md @@ -90,22 +90,22 @@ Rules: - [x] `deno task test` passes (448 total tests) - [x] Test imports use deno-lint-ignore comments (no deno.json import map) -- [ ] T06: Implement inline parser (Phase 4) +- [x] T06: Implement inline parser (Phase 4) - Why: Inline markup enrichment is the next layer above block parsing - Done when: - - [ ] `inline_parser.ts` exports inline event enrichment generator - - [ ] Bold/italic: apostrophe run disambiguation (2=italic, 3=bold, 5=bold+italic) - - [ ] Wikilinks: `[[target|display]]` with namespace dispatch - - [ ] External links: `[url text]` and bare URLs - - [ ] Templates: `{{name|args}}` with named/positional arguments - - [ ] Template arguments: `{{{param}}}` as Argument nodes - - [ ] Parser functions: `{{#if:...|...}}` classified by `#` prefix - - [ ] HTML tags: inline ``, ``, etc. - - [ ] Event well-formedness: every enter has matching exit - - [ ] Never-throw: any input produces valid events - - [ ] Property-based fuzz tests with fast-check - - [ ] `deno task test` passes - - [ ] `mod.ts` re-exports inline parser + - [x] `inline_parser.ts` exports inline event enrichment generator + - [x] Bold/italic: apostrophe run disambiguation (2=italic, 3=bold, 5=bold+italic) + - [x] Wikilinks: `[[target|display]]` with namespace dispatch + - [x] External links: `[url text]` and bare URLs + - [x] Templates: `{{name|args}}` with named/positional arguments + - [x] Template arguments: `{{{param}}}` as Argument nodes + - [x] Parser functions: `{{#if:...|...}}` classified by `#` prefix + - [x] HTML tags: inline ``, ``, etc. + - [x] Event well-formedness: every enter has matching exit + - [x] Never-throw: any input produces valid events + - [x] Property-based fuzz tests with fast-check + - [x] `deno task test` passes + - [x] `mod.ts` re-exports inline parser ## Parking lot diff --git a/.agents/memory/CONVENTIONS.md b/.agents/memory/CONVENTIONS.md index 76b83ca..54ccbbf 100644 --- a/.agents/memory/CONVENTIONS.md +++ b/.agents/memory/CONVENTIONS.md @@ -21,6 +21,10 @@ survives context resets. - Update `ACTIVE/PROGRESS.md` after meaningful progress. - Mark tasks done only when acceptance checks pass. - Promote architectural decisions to ADRs in `DECISIONS/`. +- Keep the active scope on the wikitext parser even when broader future + platform ideas are discussed. +- Treat unified ecosystem support as an optional adapter boundary, not the + core runtime model. - Run `deno task test`, `deno task bench`, and `deno doc --lint mod.ts` before marking any API-touching task complete. - The parser never throws: enforce the never-throw invariant in every change. diff --git a/.agents/memory/GLOSSARY.md b/.agents/memory/GLOSSARY.md index 6e2e058..443cce2 100644 --- a/.agents/memory/GLOSSARY.md +++ b/.agents/memory/GLOSSARY.md @@ -43,6 +43,19 @@ `variants: WikistNode[][]`). Represents structurally divergent interpretations of the same source range (jj-inspired). Not produced by the core parser; intended for collab/merge tooling. +- **Profile**: A named syntax and behavior configuration layered on the core + parser primitives. A profile can tune classification, parsing rules, + recovery behavior, and later editing or rendering semantics for a markup + family. +- **Native runtime**: The project's own execution model built around tokens, + events, trees, sessions, streaming, and profile-driven behavior. This is + distinct from unified and remains the intended long-term center of gravity. +- **Unified adapter**: Optional compatibility layer that exposes wikist or + related trees through unified parser/compiler or bridge plugins so existing + ecosystem plugins can be reused without making unified the core runtime. +- **Workflow step**: A reminder that the parser is one stage in a larger + document pipeline. It should stay focused on producing correct primitives + that later consumers, editors, sync layers, and transforms build on. ## Wikitext constructs diff --git a/.agents/memory/INDEX.md b/.agents/memory/INDEX.md index 583b786..690f2df 100644 --- a/.agents/memory/INDEX.md +++ b/.agents/memory/INDEX.md @@ -2,7 +2,7 @@ ## Core references -- [PROJECT](PROJECT.md) +- [PROJECT](PROJECT.md): current parser scope, stable architecture, longer-term platform direction - [CONVENTIONS](CONVENTIONS.md) - [GLOSSARY](GLOSSARY.md) diff --git a/.agents/memory/PROJECT.md b/.agents/memory/PROJECT.md index bc79e0b..7e5e21a 100644 --- a/.agents/memory/PROJECT.md +++ b/.agents/memory/PROJECT.md @@ -14,6 +14,22 @@ interchange format. - Flat file layout at root; `mod.ts` re-exports all public APIs - Source parser only: no template expansion or HTML rendering +## Longer-term direction + +- Near-term focus stays on finishing the wikitext parser and validating the + tokenizer, event stream, tree, and future session primitives there first. +- The parser is intentionally only one workflow step in the larger system: + source text enters the parser, and downstream consumers operate on tokens, + events, trees, and later session state. +- Longer-term, those primitives may expand into a profile-driven structured + document engine for other markup and rich-text families, CMS-style blocks, + local-first collaboration, offline or local sync, and LLM-friendly flows. +- Until a native runtime ecosystem exists for those broader goals, the project + should still be able to take advantage of unified ecosystem plugins through + optional adapters built on unist-compatible exports. +- Treat that as future platform direction, not a reason to broaden current + parser scope prematurely. + ## Key modules Implemented: @@ -24,9 +40,11 @@ Implemented: - `tokenizer.ts`: charCodeAt generator-based scanner over TextSource - `block_parser.ts`: block-level event generator (headings, paragraphs, lists, definition lists, tables, thematic breaks, preformatted blocks) +- `inline_parser.ts`: inline event enrichment (emphasis, links, templates, + arguments, comments, behavior switches, signatures, HTML entities, + `
`, ``, ``, and generic HTML tags) Not yet implemented: -- `inline_parser.ts`: inline event enrichment - `parse.ts`: orchestration (tokenizer -> block -> inline -> tree) - `tree_builder.ts`: `buildTree(events) -> WikistRoot` - `stringify.ts`: AST -> wikitext (round-trip) @@ -40,6 +58,7 @@ Available now: - `TokenType`, `Token`, `isToken()` (token.ts) - `tokenize()` (tokenizer.ts) - `blockEvents()` (block_parser.ts) +- `inlineEvents()` (inline_parser.ts) - `WikitextEvent`, `EnterEvent`, `ExitEvent`, ... + constructors + guards (events.ts) - `WikistNode`, `WikistRoot`, 37 node types + type guards + builders (ast.ts) diff --git a/.agents/memory/README.md b/.agents/memory/README.md index a79799b..f0e4d27 100644 --- a/.agents/memory/README.md +++ b/.agents/memory/README.md @@ -16,6 +16,11 @@ live under `SESSIONS` and are gitignored. Use `ACTIVE` as the control panel for ongoing work, `DECISIONS` for long-lived architectural choices, and `CHECKLISTS` for repeatable quality gates. +Keep current parser milestones and the larger platform vision separate. The +parser remains the active proving ground; broader future direction belongs in +`PROJECT`, `DECISIONS`, and tightly scoped notes in `ACTIVE`, not in inflated +task lists. + ## Edge cases If `TASKS` grows too large, split into multiple files or an epic folder. If a