zzstoatzz.io / systemone
This repository has no description
Star 0
Fork 0
Watch
Python 93%
Just 7%
5 1 0
docs
scripts
systemone
tests
.gitignore
app.py
justfile
pyproject.toml
README.md
uv.lock
README.md

systemone #

a TypeSafe System One endpoint on Modal, served by an open kev checkpoint. anything built on typesafe-sdk switches over by pointing TYPESAFE_BASE_URL at it:

TYPESAFE_BASE_URL=$(just url) TYPESAFE_API_KEY=$(cat .env.key) uv run your_thing.py

just url asks Modal for the endpoint, since the URL depends on the workspace it's deployed in.

by default it runs kev-4b in bf16 on an L4 in a US region and scales to zero. a cold start takes about a minute, most of it loading weights and the startup check below. while it starts, Modal answers waiting clients with a 303, which typesafe-sdk does not follow, so warm it first with just health.

auth #

the endpoint requires a Modal proxy auth token. Modal checks it at its proxy, so an unauthenticated request never reaches the app. typesafe-sdk sends its api key as Authorization: Bearer <key>, which is one of the forms Modal accepts, so a token works as the key unchanged.

just mint-key prints a new token as wk-….ws-…. mint one per consumer. the recipes here read theirs from .env.key, which is gitignored. tokens are workspace-wide: one opens every proxy-authed endpoint in the Modal workspace. revoke a token in the Modal dashboard under settings, proxy auth tokens.

setup #

just mint-key > .env.key
just deploy
just health   # waits out the cold start
just smoke

just deploy-9b serves kev-9b on an A100-80GB. just deploy-gpu H100 deploys a side-by-side copy as the app systemone-h100, and every recipe that talks to the endpoint takes the app name (just health systemone-h100). deploy settings are the SYSTEMONE_* variables at the top of app.py.

what it changes about kev #

the container runs kev.serve from a pinned commit, with changes that matter on CUDA:

  • concurrent requests with the same state share one forward pass (systemone/coalesce.py). kev serializes requests behind one lock.
  • question rows pad per length group instead of all to the longest row (systemone/buckets.py).
  • multi-question requests always compute the state once. kev only did that for states of 384 tokens or more.
  • the causal_conv1d kernel loads. the container refuses to start without it, instead of silently running reference code.

the row changes replace kev internals. every container answers two check requests both ways at startup and refuses to serve if the answers differ by more than 0.02. /health reports the difference as patched_row_drift.

docs #

  • performance: where the time goes, what did and didn't help, and what it costs against jev
  • operations: Modal gotchas hit while running this

development #

just test
just fmt