Gotchas #

Each of these cost real time while building or maintaining tokenling. Format: what you see, why, and what to do.

Model and prompting #

Every line starts with "ick", "icky", or other odd repeated words. The prompt ended with Name: (trailing space). GPT-2 has " word" tokens with the space attached, so a bare space token derails sampling. End prompts with Name:. Applies to the app (petLine), eval_pet.py and any new tool.

Tags come out mangled: [want_play], [clock speed], [got]. repetition_penalty in transformers.js penalizes tokens already in the prompt, which includes [, ], : and every tag name. Do not set it. The first browser tests with the trained model had it at 1.08 and tags were garbage; removing it fixed them.

[look][gift:flower] runs two tools together, or the lookup never stops. GPT-2 has a single ][ token, so the stop regex that expects a clean ] end never fires. The stopper accepts an optional trailing [ and the app trims it.

A format change that "looks harmless" breaks the pet. The model learned the exact header, word lists and stat thresholds. See AGENTS.md.

Lines pass eval_pet.py but say nonsense. The eval checks syntax, not meaning. Read transcripts. Known weak spots: irrelevant recall keywords, emotes that do not fit the question, diaries that embellish the day log.

The pet answers an offer in its own separate line, or answers when nothing was offered. Teachers like a bare Mochi: [yes] followed by a second line, and sprinkle [yes] into ordinary replies. The prompt asks for words in the answering line and the validator rejects answers without an open offer (the most common v2 rejection).

Teachers write stage directions (*purrs*). check_chat strips short *…* asides from pet lines, since emotes carry that meaning, and rejects anything left; the app also strips * from owner input.

The base model can say explicit things. GPT-2 was trained on the open web. NOT_CUTE in web/app/text.ts filters and retries; the fine-tune makes it rare, not impossible. Keep the filter.

Chat history seems ignored. The context is 1024 tokens and the header takes about 150. fit() drops the oldest chat lines first. Durable facts belong in remembers: / today: or behind a lookup.

Data generation #

Braces in the teacher prompt. The static rules are inserted into a str.format template, so literal {...} must be doubled; gen_data.py does this with SPEC_STATIC.replace("{", "{{"). Edit the template, not the escaped copy.

Teacher output slips in invented values ([emote:curious], [want:fun], [yawn]). Add an alias in EMOTE_ALIAS only when the meaning is clear; otherwise let validation reject it. About 22% of samples are rejected.

Topics over-represented in the data come from the state sampler, not the teacher. Poop appears in a third of rows through the today: line and [look] results. See training.md.

[recall] must see memories saved earlier in the same chat. Through v2, check_chat computed recall results from the sample's starting memories only, so "remember that X … what do you remember about X?" was rewritten to {nothing} and taught the pet that saving does not work. The validator now keeps a live memory list, as the app does with S.memories.

Do not pass current-format data to mix_data.py as OLD. It drops OLD rows that use [walk] (v1 had none), so newer rows given as OLD vanish. Pass them as NEW (or OLD as /dev/null): NEW rows are only re-checked.

An offer answered "yes" with no [yes]. v4 sometimes answered food with yes!! [emote:bounce] [diary:ate a cookie], so the food vanished uneaten. The eval (one offer after a short chat) showed 80/80 answered and missed it. Teachers put [yes]/[no] first or right after the words in 99% of answers, so once sampling put an emote first (rare), the model took the answer as skipped: forced to start yes yes!! [emote:bounce], v4 still answered only 65% of the time. Training now moves a share of answers to the end of the line (--late-answer), and eval_pet.py measures this case. Look at where in the line a required tag sits, not only whether it is there.

Stale rows outlive the app behavior they were written for. While chasing the bug above, 676 v4 rows turned out to put retired care events in the owner's mouth (Bea: * Bea fed Byte a rice ball *, then the pet eating with no answer); the old mix_data.py only checked lines starting with * . Dropping them alone did not fix the offers, but train.py now re-checks every row with gen_data.recheck and refuses stale data, so a new rule catches old rows at the next run.

The held-out set is defined by a random seed. train.py shuffles rows with random.seed(0) and takes the first 120; quant_eval.py repeats that shuffle. Anything that consumes random before the shuffle, or a changed row count, changes the split.

A valid transcript can still have a corrupt loss mask. Checking a canonicalized body without checking the stored text and spans accepts empty, shifted or owner/header/result-trained masks. recheck now compares canonical text and semantic span coverage and refuses these rows; explicit migration rebuilds the masks instead. Training-only donor history is added afterward and stays untrained.

Sleep migration drops a valid dream after a message or pat. A disturbance can fail to wake the pet. Preserve already-valid no-wake transcripts, and in legacy mixed logs insert a wake only where an awake reply needs it. Forcing a wake at every disturbance turns later dream tags into invalid awake replies.

GPUs #

A shared model server is too small for bulk generation (a handful of concurrent requests). Start dedicated batch servers with train/teachers.sh start, and give the GPUs back afterwards (deployment.md).

pkill -f / pgrep -f can match your own shell. The command text is in the shell's command line, so the pattern finds it. teachers.sh stop kills by process group instead; do the same for anything else, or anchor patterns like [v]llm serve.

A teacher server dies at startup with a FlashInfer version error. A vLLM install whose flashinfer and flashinfer-cubin versions disagree fails in attention-backend selection. Fix the install, or pass VLLM_ARGS="--attention-backend FLASH_ATTN" (and VLLM_USE_FLASHINFER_SAMPLER=0 in the environment). teachers.sh wait reports a server that exited, with the log tail, instead of waiting for it.

Check the GPUs before sharing one. Training at batch 16 and 1024 tokens peaks around 15 GB per 24 GB card, and the 50K-vocabulary logits make the longest batches allocator-tight. Read nvidia-smi before starting a second job on a card.

Export and runtime #

Linux Firefox says "Error in input stream" during the brain download. Reproduced with a throttled model transfer: the service worker's cache-first proxy cut off the stream after about 30 seconds. .onnx fetches must bypass respondWith; transformers.js already owns their offline cache. Returning fetch(req) inside respondWith still proxies the body. Test a transfer longer than 30 seconds, not just a fast local download (testing.md). An interrupted response can also leave a padded invalid ONNX in the model cache, followed by protobuf parsing failed. The loader evicts that brain's ONNX on either error before offering retry; clearing pet saves is unnecessary.

Firefox leaves the egg warming without starting a download. Waiting for navigator.storage.persist() can wait on an unanswered browser permission prompt. Request persistence without awaiting it; the download works without the grant. A persistent test profile alone does not grant that permission.

ONNX export fails or the fp32 file is too big. Use the separate export venv (optimum-onnx needs transformers 4.57, training uses 5.x) and install accelerate, which de-duplicates the tied embeddings.

The 4-bit model is several times slower than fp32 in the browser. ONNX Runtime's MatMulNBits (its 4-bit matmul, which MatMulNBitsQuantizer emits and its QDQ fusion creates) has no fast WebAssembly kernel: 7 tokens/s against 34 for fp32 on the same machine. Store 4-bit weights behind DequantizeLinear and load with session.disable_quant_qdq=1 so they fold to fp32 (quantize.py, brain.ts).

The model never finishes loading (or is very slow) in a new runtime or test script. It was loaded without session.disable_quant_qdq=1.

Page memory is gigabytes after loading a small model. ONNX Runtime keeps every intermediate of a constant-folded subgraph until folding ends, and the WebAssembly heap never shrinks. Unpacking nibbles with BitShift/Cast/Mul chains peaked at 3.1 GB for GPT-2 small; one DequantizeLinear per weight, fp32 scales, no prepacking and a single shared embedding/head table brought it to 0.8 GB. Measure with bench.html (window.mem()).

GPT-2's LM head will not quantize to 4 bits. Tied embedding at 4 bits: loss 4.23 → 6.98 on the base model. Keep it int8 per token row.

Int8 activation quantization wrecks GPT-2. quantize_dynamic on the head and MatMulNBits with accuracy_level=4 quantize activations, and GPT-2 has large activation outliers. Quantize weights only.

ORT aborts with _M_is_engaged() at session creation. A per-axis DequantizeLinear with QDQ fusion enabled. Set disable_quant_qdq.

Wrong or missing .wasm. The runtime directory must hold the ORT build that the transformers.js bundle was compiled against. vendor.sh pins and checks it.

import … from "onnxruntime-web" fails in the browser. You used transformers.web.js; use transformers.min.js.

ERR_BLOCKED_BY_RESPONSE for ort-wasm-simd-threaded.asyncify.mjs. Seen once in a profile that had cached the runtime before COOP/COEP were enabled. Clearing the HTTP cache fixed it. Not root-caused.

Third-party resources vanish on the page. COEP: require-corp blocks anything cross-origin without CORP headers. Host it same-origin (the page's font is vendored for this reason).

Changing a published model or runtime file has no effect. They are served immutable for a year. Publish a new directory name.

Browser testing #

The model downloads every reload in a Playwright session. In-memory profiles reject Cache Storage writes. Use --profile=DIR and a named session.

The chat keeps changing under a test. Idle turns run every 8–20 s once the brain is ready. Read state().chat right after the turn you caused, or turn the "lively" setting off.

The egg never hatches in a test. Hatching waits for the brain; a test must wait for brain.state === "ready" after "get an egg".

You cannot catch an animation with separate CLI calls. Trigger and screenshot inside a single run-code.

Stats jump twice when returning to a tab. Catch-up replay and the live animation-frame tick have separate clocks. After replaying an absence, advance the live baseline too; otherwise its next frame applies that same elapsed time again and can bypass the seven-day cap. Short gaps not replayed by catch-up must still reach the live tick once.

A storage change made in a test vanishes. The running app saves the whole pet every second, so a localStorage.setItem on the pet page is overwritten before a reload reads it. Write from another same-origin page that does not run the app (for example the font license under assets/), then navigate back.

playwright-cli eval reports "Unexpected token ';'". It evaluates an expression. Wrap statements in a function: eval "() => { …; return 1 }".

Alerts never show in a headless test. Headless Chromium reports every page as visible, and the worker never alerts while a pet page is visible. Navigate the tab to another page in the worker's scope first.

A build env var silently pointed somewhere else. test/dev-site.sh takes RUNTIME as a directory, so site/build.sh names its URL overrides RUNTIME_URL and MODELS_URL; reusing one name sent the browser to a disk path.

Chat looks empty but the model "answered". Lines of only tags render as chips or an ellipsis. Inspect window.tokenling.state().chat for the raw text.

Deployment #

The model keeps loading the old file. Models and the runtime are served immutable; a changed file under the same URL is ignored by every browser that has it. Publish a new directory name.

A rejected build deletes the previous page, or a failed publish cannot be retried. Do not replace live output before staging and validating a build. Likewise, finish the runtime copy before committing an immutable model id; otherwise a runtime failure strands the model and the existing-id guard blocks retry. The builder stages first, and the publisher completes both partials before publishing the runtime and then the model.

Static files unreadable by the web server. train/publish.sh writes files 0644 and directories 0755 by default. If your server runs as a different user in a shared group, set FILE_MODE and DIR_MODE (for example 0640 and 2750 under a setgid parent).

Stage catchphrases can drown out needs #

The v5 teacher asked teens to say "ugh" and "literally". In 3,037 teen chats, 2,992 used one of them; "literally starving" became filler even with a stuffed tummy. v6 replaces those chats and teen diaries with a varied, sincere voice and checks positive hunger claims against full/stuffed needs. Offer tags alone do not prove the surrounding words agree with the body.

Checking bracket pairs misses broken diary stickers #

The older diary validator accepted stray [ and split doodle: fragments because it only examined complete tag pairs. Strip valid doodle tags, then reject remaining brackets or doodle:. Tell the teacher to sign with the actual pet name; a literal "- PET" example was copied into five old rows.

Sleep teachers should not choose body events #

The first v6 sleep probe accepted 11/40 chats: the teacher skipped events or invented wakes. The generator now samples the wake rolls and supplies the exact events, asking for a JSON array of pet lines only. It inserts the wake after an accepted sleeping offer, then validates the assembled transcript. The next probe accepted 25/40. This keeps app-owned transitions out of the teacher's creative output without choosing dreams or offer answers for it.

When moving a sampled sleep scene out of the morning natural-wake window, update its weather along with its hour. An early v6 batch moved the hour to night but left daytime weather in 113 rows; final rechecking drops them.

Diary logs need explicit subjects #

A v6 review found diaries reading took medicine as the owner taking it, or gave OWNER a heart as the owner giving the pet a heart. The first teen diary batch was replaced after review. The teacher now gets explicit log semantics, keeps remembered facts separate from today's events, and writes shorter entries instead of inventing activity to fill a long paragraph. These are semantic errors: bracket validation alone cannot catch them.

A cached reload says the pet is in another tab #

The old document's Web Lock can briefly outlive navigation. A single immediate ifAvailable request may fail even though the old tab is going away. Release the held promise on pagehide and retry acquisition briefly without stealing (web/app/lifecycle.ts). In browser tests, wait for the closing tab's lock to leave navigator.locks.query() before asserting ownership in another tab.