# Working on tokenling

This document is the maintainer contract for humans and coding agents changing
the repository. tokenling is a virtual pet whose brain is a full
fine-tune of GPT-2 (the original 124M "small" model). The page, the pet's
state and model weights live in the visitor's browser in local mode. Remote
mode is an explicit choice: the optional Rust host stores private state and
runs the body and brain. Owners can transfer either way without losing the
pet. The browser keeps a recovery checkpoint. Both modes run the same rules:
one Rust crate (`core/`), linked natively by the host and compiled to
WebAssembly for the browser.

## Start here

Read the area document before editing a subsystem.

| Area | Documentation | Primary source |
|---|---|---|
| System overview, data flow | `docs/architecture.md` | `web/main.tsx`, `web/app/`, `web/brain.ts` |
| Shared rules: prompt, body, care, turns, saves | `core/README.md` | `core/src/`, `core/wasm/`, `web/core.ts` |
| Rust host, ownership, private API, native inference | `server/README.md` | `server/src/`, `web/app/hosting.ts` |
| Prompt header, tool grammar, lookups | `docs/prompt-format.md` | `train/world.py`, `core/src/world.rs` |
| Pet simulation, storage, UI | `docs/pet-simulation.md` | `web/app/`, `web/components/`, `web/scene.ts`, `web/sprites.ts` |
| The page and embedding the app | `docs/host-page.md` | `site/index.html`, `web/components/` |
| Teacher data, training, evals | `docs/training.md` | `train/gen_data.py`, `train/mix_data.py`, `train/train.py`, `train/eval_pet.py` |
| ONNX export, quantization, runtime | `docs/export-and-runtime.md` | `train/export.sh`, `train/quantize.py`, `web/brain.ts` |
| Publishing, server requirements, GPU etiquette | `docs/deployment.md` | `train/publish.sh`, `train/vendor.sh` |
| Tests and browser checks | `docs/testing.md` | `test/`, `train/eval_pet.py`, `train/quant_eval.py` |
| Adding a tool, emote, food or stage | `docs/extending.md` | all of the above |
| Hard-won lessons | `docs/gotchas.md` | |

## Load-bearing invariants

### The prompt format is a contract with the weights

The model only understands what it saw in training. `train/world.py` and
`core/src/world.rs` render the same state header, event lines and lookup
results, and `node test/parity.mjs` (the browser's WebAssembly build) and
`node server/tests/parity.mjs` (the host's native build) must report
**0 mismatches**. Any change to the header text, a word list, a level
threshold, an event line or a tool name is a **retraining event**: change both
files, regenerate or rewrite the data, train a new model id and publish it.
Never "fix" the format in only one place.

Moving execution does not authorize a format change or an implicit brain
upgrade. A host runs one model for all its pets and refuses to start while a
hosted pet selects another brain.

### Shared rules live in core, once

Every rule the browser and the host both apply lives in `core/`: the prompt
contract, stop and context-fit rules, the lookup loop and text filters, the
body, care and offers, the brain's tag effects, the day log and diary, turn
scheduling and save parsing. `server/` links it natively; `core/wasm/` exports
it to `web/core.ts` for the page and its worker. Do not reimplement a rule in
TypeScript or in host code; add it to `core/` and call it from both. Runtimes
own only their logistics: storage, networking, locks, rendering, inference
and when to call the engine. `server/tests/parity.mjs` replays one engine
scenario in both builds (two time zones) and requires identical results.
Calendar time is the one platform difference: the host converts with the
save's time zone, the browser uses its device clock.

### Tags are a closed grammar

The pet may only emit the tags in `docs/prompt-format.md`, with values from the
fixed lists in `world.py` and `core/src/world.rs`. The data generator rejects anything else,
and the app ignores anything outside the lists. Do not add parsing that
tolerates invented tags; fix the data instead.

### Training data must match the current app

Every row is re-checked against the current rules (`gen_data.recheck`) before
training: `train.py` refuses data with stale rows unless given `--drop-stale`.
Stored text must be canonical and loss spans must match the parser's trained
character coverage, excluding owner/event lines, headers and lookup results.
`recheck` refuses corrupt masks; explicit migration rebuilds them. Training-only
untrained history is added after re-checking.
When the app's behavior or a rule changes, add the rule to `check_body` in
`train/gen_data.py` so rows written for the old behavior are caught. Never
switch the check off to get a run through.

### Lookups are written by the app, never the model

For `[clock]`, `[look]`, `[recall:…]` and `[yesterday]` the model stops after
the tag, the app appends `{result}`, and the model continues the same line.
Loss is masked on the result in training. The model must never be trusted to
invent a result, and `recall` must stay a deterministic word-overlap search so
the training data's results match the app's.

### Prompts end at `Name:` with no trailing space

GPT-2 merges the space into the next token. A bare trailing space derails
sampling (every line started with "ick"). `core/src/turn.rs` (both runtimes) and
`train/eval_pet.py` build the prefix as `Name:` plus any text so far.

### No repetition penalty

transformers.js applies it across the prompt, which penalizes tag names and
brackets and corrupts tool calls. Leave it off in `web/brain.ts`.

### Published model and runtime directories are immutable

Model directories (`<base>/models/<id>/`) and runtime directories
(`<base>/runtime/<name>/`) are meant to be served with a one-year `immutable`
cache header. Never overwrite a file in place. Ship a new model id
(`tokenling-v4`) or runtime directory instead. `train/publish.sh` refuses to
replace an existing model for this reason.

### The brain is the pet

Decisions belong to the model: what the pet says, where it walks
(`[walk:left|right]`), how it reacts to pats, whether it accepts food, a bath,
medicine or a game (`[yes]`/`[no]` after an offer event), when it naps
(`[sleep]`), what it wants and what it writes in its diary. The app runs only
the body (stat decay, poop, sickness, collapsing when exhausted, waking when
rested or by disturbance rolls when the owner interacts with it), the room, the lookups, and the physics of what the brain decided
(walking to the spot, the eating animation, the minigame). There are no canned
replies and no random wandering. Do not add deterministic behavior that makes a
decision for the pet; add a tag and train it instead.

Do not hide or disable care offers based on needs just to enforce the answer
the model is expected to give. Medicine stays offerable when healthy: the
brain learns to decline it, and medicine has no health effect in that state. Availability
checks for a missing brain, an open offer or an active game are logistics.

Without its brain the pet dozes: the egg only hatches once the brain has
loaded, and a pet whose brain is missing sleeps with the care buttons off
until it is downloaded again. The loader retries a stalled threaded runtime on
one thread, then reports an error.

### Sleep and dreams

Sleeping lines contain one `[dream:X]` and at most four mumbled words. The
only exception is an offer: a dream plus `[no]` stays asleep; a short `[yes]`
line wakes it after the answer. Offers are the brain's decision, never a body
roll. Messages, pats and lights on use the shared energy/persistence wake
formula. Keep Python's sampled outcomes and the core formula in parity.
Ignore other effects while asleep; do not add canned dream content.

### Offers and silent lines

An offer event line (`* Sam offered Mochi a cookie *`, a bath, medicine,
`asked … to play catch`) must be answered by the pet's next line with exactly
one `[yes]` or `[no]`; the app applies the offer only on `[yes]` and treats a
missing answer as no. A pet line made only of tags is a "silent" line (the pet
acting on its own). Training data allows at most two in a row and the app
keeps at most the newest two consecutive ones in the chat, so idling never
pushes the conversation out of the context.

### Saves are never overwritten blindly

`load()` migrates older saves step by step (`VERSION`, `MIGRATIONS` in
`core/src/save.rs`) and shows the "bad egg" for anything it cannot read, including a
save from a newer version; nothing is saved until the owner downloads it,
reloads or starts anew (which moves it to `tokenling.bad`). Only the tab
holding the Web Lock runs and saves the pet. Changing a stored shape means a
version bump and a migration, tested against a real older save.

### Visitors choose when to upgrade the brain

A new model goes in `site/release.json`; the previous one moves to `older` if
the prompt format is unchanged. Visitors who have an older brain keep using it
until they accept the upgrade dialog, and anything only newer brains can do is
gated by a named feature (`docs/extending.md`). After a prompt-format change,
`older` must be empty.

### Local WebAssembly and hosted native inference

The local brain runs on ONNX Runtime Web's WebAssembly backend. WebGPU was removed
in v2: for a model this small, per-token dispatch overhead outweighs the GPU,
and one CPU path is simpler to keep correct. The published file stores the
body 4-bit and unpacks it to fp32 with constant-foldable ops at load
(`train/quantize.py`), because ORT Web's 4-bit `MatMulNBits` kernel is about
5x slower than fp32 in WebAssembly. Keep the tied embedding / LM head at 8 bits.

Hosted inference uses native ONNX Runtime with the same graph and prompt
protocol. Each worker owns one mutable session. The host serializes turns
per pet and commits only matching epochs/context revisions. Prepare inactive
state before remote activation; park the host before local resumption.
Offline recovery is an explicit branch from the last checkpoint, not proof
that an unreachable host stopped. Never silently overwrite newer local
progress with its eventual response.

### Keep it cute

Pet text is filtered through `NOT_CUTE` in `core/src/world.rs` before display, because
GPT-2's base weights can produce explicit text. Keep the filter and the retry.
Do not hardcode real people's names or details in training data, fixtures or
docs; the sample owner and pet names are generic.

## Keeping the docs current

Documentation is part of the change, not a follow-up. A change that alters
behavior, numbers, commands or file layout is not done until the docs that
describe it are updated **in the same commit**:

| If you change… | Update |
|---|---|
| prompt header, tags, word lists, stop rules | `docs/prompt-format.md`, and `docs/extending.md` if the recipe changes |
| shared rules or their boundary (`core/`, `web/core.ts`) | `core/README.md`, and the doc for the rule's area |
| simulation rates, stages, storage fields, UI behavior | `docs/pet-simulation.md` |
| data generation, validation, training settings, evals | `docs/training.md` (including the dataset stats if you regenerate) |
| export, quantization, runtime version, brain loading or fallbacks | `docs/export-and-runtime.md` |
| element ids, config keys or CSS classes the app uses, or `site/index.html` | `docs/host-page.md` |
| `site/build.sh`, `site/release.json`, `site/sw.js` or server requirements | `docs/deployment.md` (and `docs/pet-simulation.md` for offline and alerts behavior) |
| publish steps, server requirements, GPU handling | `docs/deployment.md` |
| test tooling or recipes | `docs/testing.md` |
| a new module, script or directory | the layout tables in `README.md` and `docs/architecture.md` |
| an invariant or a rule agents must follow | this file |
| a new surprise that cost you time | `docs/gotchas.md` |

For a new model, record its run receipt (data, training, eval and quantization
numbers, dates) in `docs/training.md`, and update the numbers quoted in the
other docs.

Keep interactive markup and event handlers in Preact components; use refs for
canvas/input integration. Do not reintroduce DOM-building controllers. Keep
the state and worker protocol typed, and preserve `window.tokenling` debug
controls for browser checks. Animation frames update the canvas, not the
component tree.

Before committing, run `npm run format:check`, `npm test` (it builds the
WebAssembly rules first, then types, parity, contract, Node tests, data and
script rules), re-read the docs you touched against the code, and check that
relative links and `core/`, `web/`, `train/`, `test/` and `docs/` paths in the
docs still exist. For `core/` changes also run
`cargo fmt --check --manifest-path core/Cargo.toml` (and `core/wasm/Cargo.toml`)
and `cargo test --locked --manifest-path core/Cargo.toml`. State only what you
measured: mark anything untested as untested, and put estimates and choices
made by reasoning under that label, as `docs/training.md` does for the
two-instance teacher setup.

For host or `core/` changes, also run `cargo fmt --check --manifest-path server/Cargo.toml`,
`cargo test --locked --manifest-path server/Cargo.toml`, build the
`tokenling-contract` binary and run `node server/tests/parity.mjs` (Python
parity plus the native/WebAssembly engine replay). Exercise
both transfer directions and lost responses with the real CPU brain; record
CUDA as untested until the native CUDA worker has been checked separately.

**Keep these docs host-neutral.** They describe the project, not any
particular machine or deployment: no host names, domains, hardware inventories
beyond the GPU model used, service names, absolute paths on a specific machine,
or operator procedures. Use placeholders (`$DATA`, `$VENV`, `<base>`, `HOST`).
Deployment-specific notes belong in your own operations documentation.

**Keep the code portable too.** Scripts and code must not embed machine-specific
defaults: no absolute paths, host or service names, model names or ports that
belong to one deployment. Required inputs are arguments or environment
variables that fail with a clear message when unset; defaults must be generic
and safe. Document every variable in the script's header and in the relevant
doc, and test a new script against a stand-in before relying on it.

## Everyday commands

```sh
npm ci                                  # install pinned frontend dependencies (Node.js 24+)
npm run core                            # build the shared rules to WebAssembly (core/wasm/pkg)
npm test                                # core build, types, parity, UI contract, engine/save, data and script rules
node test/parity.mjs                      # the core's WebAssembly build and Python prompts agree
$VENV/bin/python train/train.py --help    # training environment (torch + CUDA)
$VENV/bin/python train/eval_pet.py CHECKPOINT 40
python3 train/mix_data.py OLD.jsonl NEW.jsonl OUT.jsonl   # reuse older rows the format still accepts
train/export.sh CHECKPOINT OUT_DIR        # needs the separate export environment ($EXP)
train/publish.sh OUT_DIR MODEL_ID RUNTIME_DIR RUNTIME_NAME
BASE=/ site/build.sh OUT_DIR              # the static site from site/release.json
node test/contract.mjs                    # app, reference page and docs agree on element ids
MODEL=… RUNTIME=… test/dev-site.sh /tmp/dev-site   # local preview with the isolation headers
train/teachers.sh start|wait|stop         # teacher servers for data generation (TEACHER_MODEL=…)
```

Datasets, checkpoints, ONNX exports and logs (`$DATA`) stay out of Git. The
browser app uses Preact with strict TypeScript. Run `npm ci` (Node.js 24+)
once before building; building also needs Rust 1.88+ with the
`wasm32-unknown-unknown` target and `wasm-bindgen-cli` at the version
`core/wasm/Cargo.lock` pins. `site/build.sh` builds the shared rules to
WebAssembly, checks types, bundles the app and worker with esbuild, fills the
host page template and copies assets. Generated `core/wasm/pkg/` stays out of
Git. The repository must build and deploy on its own, with no host-specific tooling.

## Working agreements

- Match the surrounding code's style. The browser app uses typed Preact
  components and focused controllers; the train scripts are small standalone Python files.
- Test in a real browser with `playwright-cli` (see `docs/testing.md`). Capture
  the LCD with `playwright-cli screenshot "#pet-lcd"` when changing art or
  animation, and check the mobile layout at 360 px for horizontal overflow.
- Say which machine and thread count a speed came from. Phones were not
  measured unless the receipt says so.
- GPU jobs on a shared machine: look first, free the GPUs, and hand them back
  as soon as the GPU work is done (`docs/deployment.md`).
- Keep the page's explanation and build details readable without JavaScript;
  the interactive app is hidden until the module runs.
- Local mode never sends or stores pet/owner data outside the visitor's
  browser. Explicitly selected remote mode sends private data only to the
  chosen host. Keep access keys out of exports, logs, analytics and Git;
  protect the host's private SQLite state and hash its access keys.
  The optional, deployment-configured Cloudflare Web Analytics
  beacon may report page traffic and performance only; never include pet or
  owner fields, conversations, diaries, saves or gameplay events. Keep its
  site token in deployment configuration and disclose analytics on the page.
  Builds without a configured token have no analytics beacon.