Working on tokenling #

This document is the maintainer contract for humans and coding agents changing the repository. tokenling is a browser-only virtual pet whose brain is a full fine-tune of GPT-2 (the original 124M "small" model). The page, the pet's state and the model weights all live in the visitor's browser. No server or API is in the loop.

Start here #

Read the area document before editing a subsystem.

Area	Documentation	Primary source
System overview, data flow	docs/architecture.md	web/main.tsx, web/app/, web/brain.ts
Prompt header, tool grammar, lookups	docs/prompt-format.md	train/world.py, web/world.ts
Pet simulation, storage, UI	docs/pet-simulation.md	web/app/, web/components/, web/scene.ts, web/sprites.ts
The page and embedding the app	docs/host-page.md	site/index.html, web/components/
Teacher data, training, evals	docs/training.md	train/gen_data.py, train/mix_data.py, train/train.py, train/eval_pet.py
ONNX export, quantization, runtime	docs/export-and-runtime.md	train/export.sh, train/quantize.py, web/brain.ts
Publishing, server requirements, GPU etiquette	docs/deployment.md	train/publish.sh, train/vendor.sh
Tests and browser checks	docs/testing.md	test/, train/eval_pet.py, train/quant_eval.py
Adding a tool, emote, food or stage	docs/extending.md	all of the above
Hard-won lessons	docs/gotchas.md

Load-bearing invariants #

The prompt format is a contract with the weights #

The model only understands what it saw in training. train/world.py and web/world.ts render the same state header, event lines and lookup results, and node test/parity.mjs must report 0 mismatches. Any change to the header text, a word list, a level threshold, an event line or a tool name is a retraining event: change both files, regenerate or rewrite the data, train a new model id and publish it. Never "fix" the format in only one place.

Tags are a closed grammar #

The pet may only emit the tags in docs/prompt-format.md, with values from the fixed lists in world.py/world.ts. The data generator rejects anything else, and the app ignores anything outside the lists. Do not add parsing that tolerates invented tags; fix the data instead.

Training data must match the current app #

Every row is re-checked against the current rules (gen_data.recheck) before training: train.py refuses data with stale rows unless given --drop-stale. Stored text must be canonical and loss spans must match the parser's trained character coverage, excluding owner/event lines, headers and lookup results. recheck refuses corrupt masks; explicit migration rebuilds them. Training-only untrained history is added after re-checking. When the app's behavior or a rule changes, add the rule to check_body in train/gen_data.py so rows written for the old behavior are caught. Never switch the check off to get a run through.

Lookups are written by the app, never the model #

For [clock], [look], [recall:…] and [yesterday] the model stops after the tag, the app appends {result}, and the model continues the same line. Loss is masked on the result in training. The model must never be trusted to invent a result, and recall must stay a deterministic word-overlap search so the training data's results match the app's.

Prompts end at Name: with no trailing space #

GPT-2 merges the space into the next token. A bare trailing space derails sampling (every line started with "ick"). web/app/conversation.ts and train/eval_pet.py both build the prefix as Name: plus any text so far.

No repetition penalty #

transformers.js applies it across the prompt, which penalizes tag names and brackets and corrupts tool calls. Leave it off in web/brain.ts.

Published model and runtime directories are immutable #

Model directories (<base>/models/<id>/) and runtime directories (<base>/runtime/<name>/) are meant to be served with a one-year immutable cache header. Never overwrite a file in place. Ship a new model id (tokenling-v4) or runtime directory instead. train/publish.sh refuses to replace an existing model for this reason.

The brain is the pet #

Decisions belong to the model: what the pet says, where it walks ([walk:left|right]), how it reacts to pats, whether it accepts food, a bath, medicine or a game ([yes]/[no] after an offer event), when it naps ([sleep]), what it wants and what it writes in its diary. The app runs only the body (stat decay, poop, sickness, collapsing when exhausted, waking when rested or by disturbance rolls when the owner interacts with it), the room, the lookups, and the physics of what the brain decided (walking to the spot, the eating animation, the minigame). There are no canned replies and no random wandering. Do not add deterministic behavior that makes a decision for the pet; add a tag and train it instead.

Do not hide or disable care offers based on needs just to enforce the answer the model is expected to give. Medicine stays offerable when healthy: the brain learns to decline it, and medicine has no health effect in that state. Availability checks for a missing brain, an open offer or an active game are logistics.

Without its brain the pet dozes: the egg only hatches once the brain has loaded, and a pet whose brain is missing sleeps with the care buttons off until it is downloaded again. The loader retries a stalled threaded runtime on one thread, then reports an error.

Sleep and dreams #

Sleeping lines contain one [dream:X] and at most four mumbled words. The only exception is an offer: a dream plus [no] stays asleep; a short [yes] line wakes it after the answer. Offers are the brain's decision, never a body roll. Messages, pats and lights on use the shared energy/persistence wake formula. Keep Python's sampled outcomes and the browser formula in parity. Ignore other effects while asleep; do not add canned dream content.

Offers and silent lines #

An offer event line (* Sam offered Mochi a cookie *, a bath, medicine, asked … to play catch) must be answered by the pet's next line with exactly one [yes] or [no]; the app applies the offer only on [yes] and treats a missing answer as no. A pet line made only of tags is a "silent" line (the pet acting on its own). Training data allows at most two in a row and the app keeps at most the newest two consecutive ones in S.chat, so idling never pushes the conversation out of the context.

Saves are never overwritten blindly #

load() migrates older saves step by step (VERSION, MIGRATIONS in web/app/save-format.ts) and shows the "bad egg" for anything it cannot read, including a save from a newer version; nothing is saved until the owner downloads it, reloads or starts anew (which moves it to tokenling.bad). Only the tab holding the Web Lock runs and saves the pet. Changing a stored shape means a version bump and a migration, tested against a real older save.

Visitors choose when to upgrade the brain #

A new model goes in site/release.json; the previous one moves to older if the prompt format is unchanged. Visitors who have an older brain keep using it until they accept the upgrade dialog, and anything only newer brains can do is gated by a named feature (docs/extending.md). After a prompt-format change, older must be empty.

WebAssembly only, 4-bit packed #

The brain runs on ONNX Runtime Web's WebAssembly backend. WebGPU was removed in v2: for a model this small, per-token dispatch overhead outweighs the GPU, and one CPU path is simpler to keep correct. The published file stores the body 4-bit and unpacks it to fp32 with constant-foldable ops at load (train/quantize.py), because ORT Web's 4-bit MatMulNBits kernel is about 5x slower than fp32 in WebAssembly. Keep the tied embedding / LM head at 8 bits.

Keep it cute #

Pet text is filtered through NOT_CUTE in web/app/text.ts before display, because GPT-2's base weights can produce explicit text. Keep the filter and the retry. Do not hardcode real people's names or details in training data, fixtures or docs; the sample owner and pet names are generic.

Keeping the docs current #

Documentation is part of the change, not a follow-up. A change that alters behavior, numbers, commands or file layout is not done until the docs that describe it are updated in the same commit:

If you change…	Update
prompt header, tags, word lists, stop rules	docs/prompt-format.md, and docs/extending.md if the recipe changes
simulation rates, stages, storage fields, UI behavior	docs/pet-simulation.md
data generation, validation, training settings, evals	docs/training.md (including the dataset stats if you regenerate)
export, quantization, runtime version, brain loading or fallbacks	docs/export-and-runtime.md
element ids, config keys or CSS classes the app uses, or site/index.html	docs/host-page.md
site/build.sh, site/release.json, site/sw.js or server requirements	docs/deployment.md (and docs/pet-simulation.md for offline and alerts behavior)
publish steps, server requirements, GPU handling	docs/deployment.md
test tooling or recipes	docs/testing.md
a new module, script or directory	the layout tables in README.md and docs/architecture.md
an invariant or a rule agents must follow	this file
a new surprise that cost you time	docs/gotchas.md

For a new model, record its run receipt (data, training, eval and quantization numbers, dates) in docs/training.md, and update the numbers quoted in the other docs.

Keep interactive markup and event handlers in Preact components; use refs for canvas/input integration. Do not reintroduce DOM-building controllers. Keep the state and worker protocol typed, and preserve window.tokenling debug controls for browser checks. Animation frames update the canvas, not the component tree.

Before committing, run npm run format:check, npm run typecheck and node --test test/*.test.mjs, re-read the docs you touched against the code, run node test/parity.mjs and node test/contract.mjs and python3 test/data_rules.py, and check that relative links and web/, train/, test/ and docs/ paths in the docs still exist. State only what you measured: mark anything untested as untested, and put estimates and choices made by reasoning under that label, as docs/training.md does for the two-instance teacher setup.

Keep these docs host-neutral. They describe the project, not any particular machine or deployment: no host names, domains, hardware inventories beyond the GPU model used, service names, absolute paths on a specific machine, or operator procedures. Use placeholders ($DATA, $VENV, <base>, HOST). Deployment-specific notes belong in your own operations documentation.

Keep the code portable too. Scripts and code must not embed machine-specific defaults: no absolute paths, host or service names, model names or ports that belong to one deployment. Required inputs are arguments or environment variables that fail with a clear message when unset; defaults must be generic and safe. Document every variable in the script's header and in the relevant doc, and test a new script against a stand-in before relying on it.

Everyday commands #

npm ci                                  # install pinned frontend dependencies (Node.js 24+)
npm test                                # types, parity, UI contract, body/save, data and script rules
node test/parity.mjs                      # JS and Python prompts agree
$VENV/bin/python train/train.py --help    # training environment (torch + CUDA)
$VENV/bin/python train/eval_pet.py CHECKPOINT 40
python3 train/mix_data.py OLD.jsonl NEW.jsonl OUT.jsonl   # reuse older rows the format still accepts
train/export.sh CHECKPOINT OUT_DIR        # needs the separate export environment ($EXP)
train/publish.sh OUT_DIR MODEL_ID RUNTIME_DIR RUNTIME_NAME
BASE=/ site/build.sh OUT_DIR              # the static site from site/release.json
node test/contract.mjs                    # app, reference page and docs agree on element ids
MODEL=… RUNTIME=… test/dev-site.sh /tmp/dev-site   # local preview with the isolation headers
train/teachers.sh start|wait|stop         # teacher servers for data generation (TEACHER_MODEL=…)

Datasets, checkpoints, ONNX exports and logs ($DATA) stay out of Git. The browser app uses Preact with strict TypeScript. Run npm ci (Node.js 24+) once before building. site/build.sh checks types, bundles the app and worker with esbuild, fills the host page template and copies assets. The repository must build and deploy on its own, with no host-specific tooling.

Working agreements #

• Match the surrounding code's style. The browser app uses typed Preact components and focused controllers; the train scripts are small standalone Python files.
• Test in a real browser with playwright-cli (see docs/testing.md). Capture the LCD with playwright-cli screenshot "#pet-lcd" when changing art or animation, and check the mobile layout at 360 px for horizontal overflow.
• Say which machine and thread count a speed came from. Phones were not measured unless the receipt says so.
• GPU jobs on a shared machine: look first, free the GPUs, and hand them back as soon as the GPU work is done (docs/deployment.md).
• Keep the page's explanation and build details readable without JavaScript; the interactive app is hidden until the module runs.
• Do not store anything about the pet or its owner outside the visitor's browser. The optional, deployment-configured Cloudflare Web Analytics beacon may report page traffic and performance only; never include pet or owner fields, conversations, diaries, saves or gameplay events. Keep its site token in deployment configuration and disclose analytics on the page. Builds without a configured token have no analytics beacon.