Working on tokenling #

This document is the maintainer contract for humans and coding agents changing the repository. tokenling is a virtual pet whose brain is a full fine-tune of GPT-2 (the original 124M "small" model). The page, the pet's state and model weights live in the visitor's browser in local mode. Remote mode is an explicit choice: the optional Rust host stores private state and runs the body and brain. Owners can transfer either way without losing the pet. The browser keeps a recovery checkpoint. Both modes run the same rules: one Rust crate (core/), linked natively by the host and compiled to WebAssembly for the browser.

Start here #

Read the area document before editing a subsystem.

Area	Documentation	Primary source
System overview, data flow	docs/architecture.md	web/main.tsx, web/app/, web/brain.ts
Shared rules: prompt, body, care, turns, saves	core/README.md	core/src/, core/wasm/, web/core.ts
Rust host, ownership, private API, native inference	server/README.md	server/src/, web/app/hosting.ts
Prompt header, tool grammar, lookups	docs/prompt-format.md	train/world.py, core/src/world.rs
Pet simulation, storage, UI	docs/pet-simulation.md	web/app/, web/components/, web/scene.ts, web/sprites.ts
The page and embedding the app	docs/host-page.md	site/index.html, web/components/
Teacher data, training, evals	docs/training.md	train/gen_data.py, train/mix_data.py, train/train.py, train/eval_pet.py
ONNX export, quantization, runtime	docs/export-and-runtime.md	train/export.sh, train/quantize.py, web/brain.ts
Publishing, server requirements, GPU etiquette	docs/deployment.md	train/publish.sh, train/vendor.sh
Tests and browser checks	docs/testing.md	test/, train/eval_pet.py, train/quant_eval.py
Adding a tool, emote, food or stage	docs/extending.md	all of the above
Hard-won lessons	docs/gotchas.md

Load-bearing invariants #

The prompt format is a contract with the weights #

The model only understands what it saw in training. train/world.py and core/src/world.rs render the same state header, event lines and lookup results, and node test/parity.mjs (the browser's WebAssembly build) and node server/tests/parity.mjs (the host's native build) must report 0 mismatches. Any change to the header text, a word list, a level threshold, an event line or a tool name is a retraining event: change both files, regenerate or rewrite the data, train a new model id and publish it. Never "fix" the format in only one place.

Moving execution does not authorize a format change or an implicit brain upgrade. A host runs one model for all its pets and refuses to start while a hosted pet selects another brain.

Shared rules live in core, once #

Every rule the browser and the host both apply lives in core/: the prompt contract, stop and context-fit rules, the lookup loop and text filters, the body, care and offers, the brain's tag effects, the day log and diary, turn scheduling and save parsing. server/ links it natively; core/wasm/ exports it to web/core.ts for the page and its worker. Do not reimplement a rule in TypeScript or in host code; add it to core/ and call it from both. Runtimes own only their logistics: storage, networking, locks, rendering, inference and when to call the engine. server/tests/parity.mjs replays one engine scenario in both builds (two time zones) and requires identical results. Calendar time is the one platform difference: the host converts with the save's time zone, the browser uses its device clock.

Tags are a closed grammar #

The pet may only emit the tags in docs/prompt-format.md, with values from the fixed lists in world.py and core/src/world.rs. The data generator rejects anything else, and the app ignores anything outside the lists. Do not add parsing that tolerates invented tags; fix the data instead.

Training data must match the current app #

Every row is re-checked against the current rules (gen_data.recheck) before training: train.py refuses data with stale rows unless given --drop-stale. Stored text must be canonical and loss spans must match the parser's trained character coverage, excluding owner/event lines, headers and lookup results. recheck refuses corrupt masks; explicit migration rebuilds them. Training-only untrained history is added after re-checking. When the app's behavior or a rule changes, add the rule to check_body in train/gen_data.py so rows written for the old behavior are caught. Never switch the check off to get a run through.

Lookups are written by the app, never the model #

For [clock], [look], [recall:…] and [yesterday] the model stops after the tag, the app appends {result}, and the model continues the same line. Loss is masked on the result in training. The model must never be trusted to invent a result, and recall must stay a deterministic word-overlap search so the training data's results match the app's.

Prompts end at Name: with no trailing space #

GPT-2 merges the space into the next token. A bare trailing space derails sampling (every line started with "ick"). core/src/turn.rs (both runtimes) and train/eval_pet.py build the prefix as Name: plus any text so far.

No repetition penalty #

transformers.js applies it across the prompt, which penalizes tag names and brackets and corrupts tool calls. Leave it off in web/brain.ts.

Published model and runtime directories are immutable #

Model directories (<base>/models/<id>/) and runtime directories (<base>/runtime/<name>/) are meant to be served with a one-year immutable cache header. Never overwrite a file in place. Ship a new model id (tokenling-v4) or runtime directory instead. train/publish.sh refuses to replace an existing model for this reason.

The brain is the pet #

Decisions belong to the model: what the pet says, where it walks ([walk:left|right]), how it reacts to pats, whether it accepts food, a bath, medicine or a game ([yes]/[no] after an offer event), when it naps ([sleep]), what it wants and what it writes in its diary. The app runs only the body (stat decay, poop, sickness, collapsing when exhausted, waking when rested or by disturbance rolls when the owner interacts with it), the room, the lookups, and the physics of what the brain decided (walking to the spot, the eating animation, the minigame). There are no canned replies and no random wandering. Do not add deterministic behavior that makes a decision for the pet; add a tag and train it instead.

Do not hide or disable care offers based on needs just to enforce the answer the model is expected to give. Medicine stays offerable when healthy: the brain learns to decline it, and medicine has no health effect in that state. Availability checks for a missing brain, an open offer or an active game are logistics.

Without its brain the pet dozes: the egg only hatches once the brain has loaded, and a pet whose brain is missing sleeps with the care buttons off until it is downloaded again. The loader retries a stalled threaded runtime on one thread, then reports an error.

Sleep and dreams #

Sleeping lines contain one [dream:X] and at most four mumbled words. The only exception is an offer: a dream plus [no] stays asleep; a short [yes] line wakes it after the answer. Offers are the brain's decision, never a body roll. Messages, pats and lights on use the shared energy/persistence wake formula. Keep Python's sampled outcomes and the core formula in parity. Ignore other effects while asleep; do not add canned dream content.

Offers and silent lines #

An offer event line (* Sam offered Mochi a cookie *, a bath, medicine, asked … to play catch) must be answered by the pet's next line with exactly one [yes] or [no]; the app applies the offer only on [yes] and treats a missing answer as no. A pet line made only of tags is a "silent" line (the pet acting on its own). Training data allows at most two in a row and the app keeps at most the newest two consecutive ones in the chat, so idling never pushes the conversation out of the context.

Saves are never overwritten blindly #

load() migrates older saves step by step (VERSION, MIGRATIONS in core/src/save.rs) and shows the "bad egg" for anything it cannot read, including a save from a newer version; nothing is saved until the owner downloads it, reloads or starts anew (which moves it to tokenling.bad). Only the tab holding the Web Lock runs and saves the pet. Changing a stored shape means a version bump and a migration, tested against a real older save.

Visitors choose when to upgrade the brain #

A new model goes in site/release.json; the previous one moves to older if the prompt format is unchanged. Visitors who have an older brain keep using it until they accept the upgrade dialog, and anything only newer brains can do is gated by a named feature (docs/extending.md). After a prompt-format change, older must be empty.

Local WebAssembly and hosted native inference #

The local brain runs on ONNX Runtime Web's WebAssembly backend. WebGPU was removed in v2: for a model this small, per-token dispatch overhead outweighs the GPU, and one CPU path is simpler to keep correct. The published file stores the body 4-bit and unpacks it to fp32 with constant-foldable ops at load (train/quantize.py), because ORT Web's 4-bit MatMulNBits kernel is about 5x slower than fp32 in WebAssembly. Keep the tied embedding / LM head at 8 bits.

Hosted inference uses native ONNX Runtime with the same graph and prompt protocol. Each worker owns one mutable session. The host serializes turns per pet and commits only matching epochs/context revisions. Prepare inactive state before remote activation; park the host before local resumption. Offline recovery is an explicit branch from the last checkpoint, not proof that an unreachable host stopped. Never silently overwrite newer local progress with its eventual response.

Keep it cute #

Pet text is filtered through NOT_CUTE in core/src/world.rs before display, because GPT-2's base weights can produce explicit text. Keep the filter and the retry. Do not hardcode real people's names or details in training data, fixtures or docs; the sample owner and pet names are generic.

Keeping the docs current #

Documentation is part of the change, not a follow-up. A change that alters behavior, numbers, commands or file layout is not done until the docs that describe it are updated in the same commit:

If you change…	Update
prompt header, tags, word lists, stop rules	docs/prompt-format.md, and docs/extending.md if the recipe changes
shared rules or their boundary (core/, web/core.ts)	core/README.md, and the doc for the rule's area
simulation rates, stages, storage fields, UI behavior	docs/pet-simulation.md
data generation, validation, training settings, evals	docs/training.md (including the dataset stats if you regenerate)
export, quantization, runtime version, brain loading or fallbacks	docs/export-and-runtime.md
element ids, config keys or CSS classes the app uses, or site/index.html	docs/host-page.md
site/build.sh, site/release.json, site/sw.js or server requirements	docs/deployment.md (and docs/pet-simulation.md for offline and alerts behavior)
publish steps, server requirements, GPU handling	docs/deployment.md
test tooling or recipes	docs/testing.md
a new module, script or directory	the layout tables in README.md and docs/architecture.md
an invariant or a rule agents must follow	this file
a new surprise that cost you time	docs/gotchas.md

For a new model, record its run receipt (data, training, eval and quantization numbers, dates) in docs/training.md, and update the numbers quoted in the other docs.

Keep interactive markup and event handlers in Preact components; use refs for canvas/input integration. Do not reintroduce DOM-building controllers. Keep the state and worker protocol typed, and preserve window.tokenling debug controls for browser checks. Animation frames update the canvas, not the component tree.

Before committing, run npm run format:check, npm test (it builds the WebAssembly rules first, then types, parity, contract, Node tests, data and script rules), re-read the docs you touched against the code, and check that relative links and core/, web/, train/, test/ and docs/ paths in the docs still exist. For core/ changes also run cargo fmt --check --manifest-path core/Cargo.toml (and core/wasm/Cargo.toml) and cargo test --locked --manifest-path core/Cargo.toml. State only what you measured: mark anything untested as untested, and put estimates and choices made by reasoning under that label, as docs/training.md does for the two-instance teacher setup.

For host or core/ changes, also run cargo fmt --check --manifest-path server/Cargo.toml, cargo test --locked --manifest-path server/Cargo.toml, build the tokenling-contract binary and run node server/tests/parity.mjs (Python parity plus the native/WebAssembly engine replay). Exercise both transfer directions and lost responses with the real CPU brain; record CUDA as untested until the native CUDA worker has been checked separately.

Keep these docs host-neutral. They describe the project, not any particular machine or deployment: no host names, domains, hardware inventories beyond the GPU model used, service names, absolute paths on a specific machine, or operator procedures. Use placeholders ($DATA, $VENV, <base>, HOST). Deployment-specific notes belong in your own operations documentation.

Keep the code portable too. Scripts and code must not embed machine-specific defaults: no absolute paths, host or service names, model names or ports that belong to one deployment. Required inputs are arguments or environment variables that fail with a clear message when unset; defaults must be generic and safe. Document every variable in the script's header and in the relevant doc, and test a new script against a stand-in before relying on it.

Everyday commands #

npm ci                                  # install pinned frontend dependencies (Node.js 24+)
npm run core                            # build the shared rules to WebAssembly (core/wasm/pkg)
npm test                                # core build, types, parity, UI contract, engine/save, data and script rules
node test/parity.mjs                      # the core's WebAssembly build and Python prompts agree
$VENV/bin/python train/train.py --help    # training environment (torch + CUDA)
$VENV/bin/python train/eval_pet.py CHECKPOINT 40
python3 train/mix_data.py OLD.jsonl NEW.jsonl OUT.jsonl   # reuse older rows the format still accepts
train/export.sh CHECKPOINT OUT_DIR        # needs the separate export environment ($EXP)
train/publish.sh OUT_DIR MODEL_ID RUNTIME_DIR RUNTIME_NAME
BASE=/ site/build.sh OUT_DIR              # the static site from site/release.json
node test/contract.mjs                    # app, reference page and docs agree on element ids
MODEL=… RUNTIME=… test/dev-site.sh /tmp/dev-site   # local preview with the isolation headers
train/teachers.sh start|wait|stop         # teacher servers for data generation (TEACHER_MODEL=…)

Datasets, checkpoints, ONNX exports and logs ($DATA) stay out of Git. The browser app uses Preact with strict TypeScript. Run npm ci (Node.js 24+) once before building; building also needs Rust 1.88+ with the wasm32-unknown-unknown target and wasm-bindgen-cli at the version core/wasm/Cargo.lock pins. site/build.sh builds the shared rules to WebAssembly, checks types, bundles the app and worker with esbuild, fills the host page template and copies assets. Generated core/wasm/pkg/ stays out of Git. The repository must build and deploy on its own, with no host-specific tooling.

Working agreements #

• Match the surrounding code's style. The browser app uses typed Preact components and focused controllers; the train scripts are small standalone Python files.
• Test in a real browser with playwright-cli (see docs/testing.md). Capture the LCD with playwright-cli screenshot "#pet-lcd" when changing art or animation, and check the mobile layout at 360 px for horizontal overflow.
• Say which machine and thread count a speed came from. Phones were not measured unless the receipt says so.
• GPU jobs on a shared machine: look first, free the GPUs, and hand them back as soon as the GPU work is done (docs/deployment.md).
• Keep the page's explanation and build details readable without JavaScript; the interactive app is hidden until the module runs.
• Local mode never sends or stores pet/owner data outside the visitor's browser. Explicitly selected remote mode sends private data only to the chosen host. Keep access keys out of exports, logs, analytics and Git; protect the host's private SQLite state and hash its access keys. The optional, deployment-configured Cloudflare Web Analytics beacon may report page traffic and performance only; never include pet or owner fields, conversations, diaries, saves or gameplay events. Keep its site token in deployment configuration and disclose analytics on the page. Builds without a configured token have no analytics beacon.