Codebase Patterns: Wikitext Parser #

Reference for the key architecture, data flow, and internal patterns. Read this before making any non-trivial change to core modules.

Pipeline architecture #

Events, not AST, are the fundamental output. Everything else is a consumer. The pipeline accepts any TextSource (plain string satisfies the interface).

TextSource ─► Tokenizer ─► Block Parser ─► Inline Parser ─► [Consumer]
                 │               │               │                │
                 │  charCodeAt   │  line-start   │  recursive     ├─► buildTree()
                 │  scanner      │  dispatch     │  descent       ├─► compileHtml()
                 │               │               │  with stack    ├─► filterEvents()
                 │               │               │                └─► direct callback
                 │               │               │
                 │               │               └─► text events enriched with
                 │               │                   inline enter/exit pairs
                 │               │
                 │               └─► block enter/exit events
                 │                   + state snapshots at boundaries
                 │
                 └─► Generator<Token> (offset-based, no value strings)

Three streaming modes, all event-well-formed (stack discipline):

Mode	API	Cost	Use case
Outline	outlineEvents(input)	Block only	TOC, section index
Full	events(input)	Block + inline	Default, tree building
Progressive	parseChunked(chunks)	Async block nodes	Streaming render

All event modes produce range-first events: text/token events carry start_offset/end_offset into the TextSource rather than an extracted value string. Consumers call slice(source, evt) to resolve text on demand.

TextSource (text_source.ts) #

Minimal interface that abstracts the backing text store:

interface TextSource {
  readonly length: number;
  charCodeAt(index: number): number;
  slice(start: number, end: number): string;
  iterSlices?(start: number, end: number): Iterable<string>; // optional
}

Plain string satisfies this interface. Rope trees, CRDTs (Yjs Y.Text), and append buffers can implement it for zero-copy access.

The tokenizer and all downstream modules accept TextSource, not bare string. This is the collaboration-readiness seam: swapping the backing store does not require changing any parser module.

Tokenizer (tokenizer.ts) #

Generator-based scanner using charCodeAt on the TextSource. Yields Token objects with offset ranges, never value strings.

Token design (offset-based) #

interface Token {
  type: TokenType;
  start: number;   // UTF-16 code unit offset into TextSource
  end: number;     // exclusive end offset
  // No `value` field: use slice(source, token) to resolve on demand
}

Why offsets instead of value strings:

• Avoids per-token string allocation
• Sidesteps V8's sliced-string retention risk (small slice can pin entire parent string; behavior is heuristic-driven, not unconditional)
• Consumer calls slice(source, token) only when it needs the text
• Works identically with TextSource implementations that aren't string

Character code constants #

Hot scanning loops use integer character codes:

Constant	Hex	Character	Wikitext role
CC_EQUALS	0x3d	=	Heading marker, template arg
CC_APOSTROPHE	0x27	'	Bold/italic delimiter
CC_OPEN_BRACKET	0x5b	[	Link open
CC_CLOSE_BRACKET	0x5d	]	Link close
CC_OPEN_BRACE	0x7b	{	Template/table open
CC_CLOSE_BRACE	0x7d	}	Template/table close
CC_PIPE	0x7c	|	Table cell, template arg
CC_BANG	0x21	!	Table header cell
CC_ASTERISK	0x2a	*	Bullet list marker
CC_HASH	0x23	#	Ordered list marker
CC_COLON	0x3a	:	Definition description / indent
CC_SEMICOLON	0x3b	;	Definition term
CC_DASH	0x2d	-	Thematic break (----)
CC_TILDE	0x7e	~	Signature (~~~+)
CC_UNDERSCORE	0x5f	_	Behavior switch (__TOC__)
CC_LT	0x3c	<	HTML/extension tag, comment
CC_GT	0x3e	>	HTML tag close
CC_AMP	0x26	&	HTML entity
CC_LF	0x0a	\n	Line ending
CC_CR	0x0d	\r	Carriage return
CC_SPACE	0x20		Whitespace, preformatted line
CC_TAB	0x09	\t	Whitespace

Scanning rules #

• Single pass: never backtracks more than bounded lookahead (max 4 chars: <!-- for comment open)
• Line-start markers only recognized at start of line (after optional whitespace): =, *, #, :, ;, {|, |-, |}, ----,
• Apostrophe runs emitted as single APOSTROPHE_RUN(length) token; disambiguation deferred to inline parser
• Comment/nowiki regions pre-scanned to protect content from tokenization
• Fresh objects per yield: no object reuse across generator yields

Block parser (block_parser.ts) #

Consumes token stream, emits block-level enter/exit/text events. Line-oriented dispatch:

First token(s) of line → block type:
  =          → Heading (count determines level)
  * # : ;    → List/ListItem/DefinitionList
  {|         → Table
  ----       → ThematicBreak
  (space)    → Preformatted
  (other)    → Paragraph

Block nesting rules:

• *** = 3-deep bullet list → nested enter(list) → enter(listItem) chain
• ; term : desc → DefinitionList with DT/DD events
• Tables: {| opens, |} closes, |- = row, ! = header, | = data, |+ = caption
• Explicit stack for nested structures (lists in lists, tables in tables)

State snapshots #

At each block boundary, record: { inNowiki, inPre, openTagStack, openTemplateDepth, inTable }

Enables incremental reparsing (Phase 7) to find the nearest "neutral boundary" when determining where to start reparsing after an edit.

Neutral boundary: a position where all tracked parser state is at its default (empty/zero) value: openTemplateDepth = 0, openLinkDepth = 0, inTable = false, openTagStack = [], inNowiki = false, quote state clean. Parsing can resume from scratch at a neutral boundary without inheriting state from prior content.

Inline parser (inline_parser.ts) #

Enriches block events with inline markup. Recursive descent with explicit stack for nesting.

Key algorithms #

Apostrophe run disambiguation:

• 2 = italic, 3 = bold, 5 = bold+italic
• 4 = one literal apostrophe + bold
• Unclosed runs → close at end of line (MediaWiki behavior)

Link detection:

• [[target|display]] → Wikilink
• [[File:...]] / [[Image:...]] → ImageLink
• [[Category:...]] → CategoryLink
• [[:Category:Foo]] / [[:File:Foo.png]] → Wikilink (leading-colon escape: visible link, not assignment/embed)
• [url text] and bare URLs → ExternalLink
• Nested [[ inside template args handled by bracket counting

Template parsing:

• {{name|arg1|key=value}} → Template + TemplateArgument events
• {{#if:...}} → ParserFunction (identified by # prefix)
• {{PAGENAME}} → Template by default (variable-style magic words are a configured set; profiles or consumers reclassify known names)
• {{{param|default}}} → Argument (triple braces)
• Cross-block spans possible ({{ ... \n ... }})

HTML tags:

• Self-closing, void elements, matching open/close pairs
• Extension tags (<ref>, <nowiki>, <pre>, etc.): content protected
• Tag recovery boundary is the opener's closing >: malformed-but-closed openers stay structurally real, but an opener that never reaches > stays plain text with an inline recovery diagnostic

Special constructs:

• ~~~/~~~~/~~~~~ → Signature
• __TOC__, __NOTOC__ → BehaviorSwitch
• #REDIRECT [[...]] → Redirect
• & → HtmlEntity

Tree builder (tree_builder.ts) #

buildTree(events) → WikistRoot

Consumes the event stream, produces a wikist tree (unist-compatible AST). Every enter pushes onto a stack, every exit pops and attaches to parent. text events become Text leaf nodes.

Supports { inlineMode: "lazy" }: inline content stays as Text nodes until resolveInlines(node) is called on demand.

Position semantics #

All positions use UTF-16 code unit offsets (matching JS string indexing):

• line: 1-indexed (first line = 1)
• column: 1-indexed, UTF-16 code units from start of line
• offset: 0-indexed, UTF-16 code units from start of input
• input.slice(start.offset, end.offset) returns the source text

Error recovery #

The parser never throws on any input:

Malformed input	Recovery
Unclosed bold/italic	Close at end of line
Unclosed [[ or {{	Treat opening as literal text
Malformed tables	Best-effort row/cell, fall back to text
Unknown extension tags	Opaque HtmlTag with raw content
Mixed/nested list markers	Build deepest sensible nesting
Chunked input mid-token	Carry buffer handles partial tokens

Optionally emit { type: "error", message, position } events for recovery points that consumers can log or ignore.

Performance discipline #

• charCodeAt, not charAt: avoid string allocation per character
• Offset-based tokens, not value strings: avoid per-token allocation
• Range-first events: no value strings on text events
• Single-pass scanning with bounded lookahead
• Fresh immutable objects per yield: no object reuse across generator yields
• JIT-friendly hot loops: no megamorphic call sites, no closures per character
• No closures allocated in inner loops

Session API (session.ts) #

Stateful wrapper around the stateless pipeline. Surface area grows across phases:

Phase	Surface
5	createSession(source) → .events(), .outline(), .parse()
6	.write(chunk), .drainStableEvents()
7	.applyChanges(edits) → PositionMap, edit coalescing

Session delegates to pipeline modules and caches parse state. It is not a God object; each phase adds minimal API surface.

Stability frontier #

During streaming (Phase 6), the Session tracks a stabilityOffset: the UTF-16 offset up to which emitted events are guaranteed stable. Events before the frontier won't change as more input arrives. Events after are provisional.

[───── stable prefix ──────][── provisional tail ──]
                              ▲
                       stabilityOffset

PositionMap #

Returned by session.applyChanges(edits). Maps old-offset → new-offset, enabling callers (editors, annotation layers) to translate positions across edits without re-parsing the full tree.

File layout #

Flat files at root alongside mod.ts. No src/ folder.

File	Role
ast.ts	Wikist node types, type guards, builders
events.ts	WikitextEvent union, constructors
token.ts	Token interface, TokenType const-object
text_source.ts	TextSource interface
tokenizer.ts	Generator-based scanner
block_parser.ts	Block-level event emitter
inline_parser.ts	Inline event enrichment
parse.ts	Orchestration (tokenize → block → inline → tree)
tree_builder.ts	buildTree(events) → WikistRoot
session.ts	Stateful Session wrapper (Phases 5–7)
stringify.ts	AST → wikitext (round-trip)
filter.ts	Filter/visit for tree and event streams
mod.ts	Re-exports all public APIs

For the current full public API, run deno doc mod.ts or read the exports at the top of mod.ts directly. Duplicating that list here would only drift.