# Extending subtokenling

Anything the model must say or understand is a **retraining event**: the
weights only know what was in the data. Anything purely visual or in the
simulation is not. Each recipe says which it is.

Rules shared by the browser and the host live once in `core/` (see
[its guide](../core/README.md)); never add them to `web/` or `server/` code.
After any change that touches `world.py` or `core/`, run `npm test`, build
`tokenling-contract` and run `node server/tests/parity.mjs`.

## Retrain loop

1. Update `train/world.py` and `core/src/world.rs` together. Run
   `node test/parity.mjs`, build `tokenling-contract` and run
   `node server/tests/parity.mjs`.
2. Update the teacher prompt in `train/gen_data.py` (it builds word lists from
   `world.py`, but scenarios and examples are written out) and the validator
   so it accepts exactly the new grammar.
3. Probe: `gen_data.py probe.jsonl 6 2`, read the rows, fix prompt or checks.
4. Generate the full set (free the GPUs first, see `deployment.md`), train, run
   `eval_pet.py`, export, publish as a **new** model id, update
   `site/release.json` and deploy.

## Add an emote

Model-facing:

- `EMOTES` in `train/world.py` and `core/src/world.rs`.

App-facing:

- `FACE` in `web/scene.ts` (eyes, mouth) and any motion branch in `drawPet`;
- an entry in `ACT_WORD` in `web/app/text.ts` and, if it has a floating icon,
  the `emote` branch of `playEffect()` in `web/app/animation.ts`.

Retrain so the model produces it. A new word the model has not seen will not
appear. (For a quick visual-only animation, add it to `FACE`/`drawPet` and
trigger it from a care action with `animate()`; no retraining.)

## Add a tool

**An action tool** (like `[gift]` or `[walk:left]`): add the name to
`EFFECT_TOOLS` in both world files (and a value list such as `WALKS`, which
`test/parity.mjs` compares), validate its arguments in `check_chat()`, apply
it in `Engine::apply_line` in `core/src/engine.rs` (emit an effect for the LCD
and play it in `playEffect()`), show a chip or an action word in `chipsFor()` /
`ACT_WORD`, describe it in the teacher prompt, then retrain. Prefer a tag for
anything the pet decides; the app should only carry out the decision.

**A new offer** (something the owner proposes and the pet accepts or refuses):
add the event line to `event_patterns()` and `offer_patterns()` in
`world.py`, the builder to `events` in `core/src/world.rs` (and `event()` in
`core/wasm/src/lib.rs` for the parity test), branches in the `Care` command and
`answer_offer` in `core/src/engine.rs`, a `NO_POOL` entry and scenarios in
`gen_data.py`, and a case in the parity test's event list. Then retrain.

**A lookup tool** (like `[clock]`): in addition to the above, add it to
`LOOKUP_TOOLS` and `tool_result()` with identical logic in `world.py` and
`core/src/world.rs`, and update the two stop regexes: `LOOKUP_END` in
`core/src/world.rs` (both brains stop with it) and in `train/eval_pet.py`.
Mention it in the static part of the teacher prompt and add scenarios that
call it. Results must be deterministic functions of the prompt state, because
the validator replaces whatever the teacher wrote.

## Add a dream

Add the value to `DREAMS` in both world files and provide its sprite. Foods
and gifts are included automatically. `dream_thing`/`dreamThing` supplies the
day-log and chip label. Update sleep teacher coverage and validation, retrain,
and ship a new model. Preserve the closed sleeping grammar and bounded dream
log. Body-only wake tuning uses `wake_chance` in `core/src/world.rs` and
`world.py`; parity covers it.

## Add or change a food, gift or want

- Lists: `FOODS`, `GIFTS` or `WANTS` in both world files.
- Art: add a sprite to `SPRITES` in `web/sprites.ts` (rows of 8 characters, see
  `KEY` for the palette) and, for wants, `WANT_ICON` in `web/app/config.ts`.
- Behavior: feeding effects are in `answer_offer` in `core/src/engine.rs`.
- Foods and gifts appear in prompts, so this is a retraining event. A
  food that exists in the app but not in the training data confuses the model.

## Add a life stage or change timing

- Names: `STAGES` in both world files; voice text in `STAGE_STYLE` in
  `gen_data.py`; sampling weights in `rand_state()`.
- App: `STAGE_AGES` in `core/src/body.rs`, a `SIZE` entry and decorations in `scene.ts`.
- The prompt only contains the stage name and day number, so changing **ages**
  alone is not a retraining event, but new stage names are.

## Change the simulation

Rates, poop timing and sleep rules are constants in `core/src/body.rs`
(`simulate_body`); care, offers, scheduling and the day log are in
`core/src/engine.rs`. Minigame tuning is in `web/app/game.ts` (`stepGame()`).
Keep them in sync with `pet-simulation.md`; `node server/tests/parity.mjs`
checks that both builds still agree. If you change the **stat words** or thresholds
(`levelWords`), it is a retraining event: the model learned those exact words.

## Change the look of the page

The host page is `site/index.html` (styles inline, tokens on `:root`, light
and dark). Interactive markup and event handlers are Preact components in
`web/components/`; body and model behavior stay in `web/app/`. Pixel art lives inside the LCD canvas and the sprite icons. Keep focus
rings, the 360 px layout, reduced motion and the no-JavaScript notes working,
and run `node test/contract.mjs` if you rename or remove an element.

## Add a stored field

Add it to `PetState` in `core/src/types.rs`, `fresh()` in `core/src/save.rs`
and `PetState` in `web/types.ts`. A new top-level key is filled for older
saves from a new save's defaults, and a new setting whose absence means "off"
needs nothing. Nested changes or renames need `VERSION` bumped and a step
appended to `MIGRATIONS` in `core/src/save.rs` (it receives the previous
version's object and must return one with `v` one higher). Never make `load()` accept a newer version:
it shows the bad egg instead, and the save stays untouched. Test with an old
save (export one before the change) and document it in `pet-simulation.md`.

## Ship a feature only newer brains have

When a feature needs training (a new tool, offer or event line the old brains
never saw), the app must not use it with an older brain the visitor kept. Name
the feature, list it under `features` in `site/release.json` (and in each
`older` entry that has it), describe it in `FEATURE_TEXT` in `web/app/config.ts` for the
upgrade dialog, and gate it: `brainHas("name")` for component disabled/class/title props,
`data-needs="name"` for inspection, `ACT_NEEDS` for care actions, and `brainHas("name")` around any other code path that writes
the new lines. A change to the prompt format itself cannot be gated this way:
empty `older` instead (`deployment.md`).

## Change the base model

The small-model matrix in `train/experiments.json` now has runnable native
GPT-2/Pythia, vocabulary-reduced Pythia and tied standard Llama experiments;
this does not select or publish a replacement. The inherited baseline v1 had
355M parameters; v2–v6 have 124M, and the current published file is
97,276,118 bytes. Those are baseline facts, not target sizes or architecture
requirements. The current engine remains WebAssembly.

The source still has baseline-architecture assumptions. A new checkpoint path
or an automatic tokenizer/browser loader does not make the pipeline universal.
For each candidate, adapt the affected contracts rather than papering over
unsupported shapes:

- `train/prepare_model.py`, `train/model_data.py` and `train/train.py`:
  generic causal-model loading does not imply support for every tokenizer.
  Current preparation supports real ByteLevel BPE native/prefix controls and
  scratch Llama. Preserve strict tokenizer character offsets/loss boundaries,
  real end/pad tokens, declared context, unchanged canonical spans and
  reproducible checkpoint/config receipts. Explicit text-only pretraining must
  never become a bypass for masked pet data.
- `train/eval_pet.py`: model loader, context trimming, special/end tokens,
  decoding and past-cache API. Preserve the app's offer/sleep/idle/diary and
  deterministic lookup loop, including `Name:` without a trailing space.
- `train/export.sh` and `train/quantize.py`: verify exporter support, real
  constant/Gemm/MatMul topology, configured tied versus untied tables,
  quantization axes/block format and last-token slicing. The current
  quantizer derives these from supported graph/config contracts, not an
  initializer alias or the baseline's 48 weights / `[768, 50257]` head.
  Unsupported or ambiguous shapes fail; add a real adaptation, not a name shim.
- `train/onnx_utils.py`, `train/check_onnx.py` and `train/quant_eval.py`:
  supported separate rank-four GPT-2/NeoX/Llama past tensors derive KV heads,
  head width, dtypes and matching presents from actual metadata and config,
  including GQA. Merged/unfamiliar caches still require adaptation. Preserve
  fp32 numeric/greedy parity, cached-versus-fresh parity for every dtype, and
  full-sequence masked loss. Quantized greedy drift is a measurement, not an
  automatic requirement to match every fp32 token.
- `web/brain.ts`, `web/app/brain.ts` and the generation calls in
  `web/app/conversation.ts`/`web/app/diary.ts`: supported browser model/config,
  typed worker messages in `web/types.ts`, the vendored API boundary in
  `web/runtime-types.ts`, context budgets (currently 1024 react / 512 idle),
  tokenizer and merged-boundary handling such as `][`, stop/end-token rules,
  q4 filename and session options. Verify the resulting artifact with the
  pinned runtime; do not invent a fallback engine or silently replace pet
  decisions. The worker source is TypeScript but its emitted URL stays `brain.js`.
- `server/src/brain.rs`: native support is explicit for GPT-2, NeoX and
  Llama/GQA separate rank-four caches. Configure the export dtype, preserve
  actual vocabulary/EOS/context and metadata validation, and exercise real
  native generation plus private HTTP/transfers before hosting another model.
  Keep model-ID pinning; a successful native warmup is not a quality gate.
- `train/world.py`, `core/src/world.rs`, `train/gen_data.py` and the data: retain
  the exact frozen wire markers and closed grammar for compatible weights.
  If those contracts change, update both world files, teacher/validator and
  data, retrain, and empty the release's incompatible `older` list.

Before publishing, require prompt parity and data-rule checks, behavioral evals
and transcript review (offers, sleep/dreams, recall, idle voice and diaries),
quantization quality versus the candidate's unquantized checkpoint, measured
download size, load/prefill/decode speed and page memory. Run `bench.html` and
the full real-browser flow in `testing.md`: fresh download/hatch, cached and
offline reload, every lookup, diary, upgrade/incompatible-brain handling,
legacy-save preservation, alerts/install and the 360 px surface. Identify the
hardware, threads, dates and unmeasured devices; baseline receipts are not
proof of a candidate's performance. Publish only a proved new immutable
`MODEL_ID`, update release size/runtime/features and visitor-facing details,
and keep the historical receipts labelled as inherited baselines.