diff --git a/localinfra/readme.md b/localinfra/readme.md index 3d19024b9..6587d1c9f 100644 --- a/localinfra/readme.md +++ b/localinfra/readme.md @@ -8,6 +8,11 @@ To make that work: - Caddy's dev root CA is mounted into every container that talks to another service over HTTPS. - The Docker network uses an unrouted "public" subnet so the SSRF dialer doesn't reject container IPs as private. +- Only the following services can fetch: `ncps` and `spindle` for nix, and + `knot2` for the `core` clone off `knot1.tangled.sh`. `docker-compose.yml` puts + those on a second `upstream-cache` bridge and leaves the rest on `tngl`, which + is marked `internal: true`. DNS on that network fails for any name outside the + project. ## What's inside: @@ -57,64 +62,39 @@ To make that work: ./localinfra/certs/root.crt ``` - Depending on your browser you may have to import the certificate into your browser profiles too as some have their own certs do not use your system ones -3. Fetch the appview's vendored static assets. The tailwind service writes - `tw.css` alone, so skipping this leaves the appview without htmx, mermaid, - mathjax, fonts or icons: +3. Please fetch the appview's other vendored static assets, the ones the + tailwind service doesn't write alongside `tw.css`: ```bash ./localinfra/scripts/appview-static-files.sh ``` -4. For the `linux` profile's spindle, prepare its microVM images, written under - `out/localinfra-spindle-images`: +4. Prepare the spindle microVM images: ```bash ./localinfra/scripts/prepare-spindle-images.sh ``` - - spindle's nix builds pull store paths through `ncps`, which proxies - `cache.nixos.org`. They also fetch `github:` flakerefs straight from - `github.com` as a tarball, so `ncps` never sees those. Both services join a - second bridge, `upstream-cache`, and so does knot2. `repo.create` for `core` - passes `https://knot1.tangled.sh/...` as its `source`, and the clone runs - inside knot2. The seed only posts the URL, so knot2 is the container that - needs the route out. Everything else stays on `tngl` alone, which is - `internal: true` and answers NXDOMAIN for any name outside the project. - Without that bridge the profile fails on the first flake input or store - path it tries to fetch. - -5. `podman compose build` (or `docker compose build`), then warm the dev-loop - cache volumes while you still have a route out. air rebuilds the appview, - cargo rebuilds bobbin, and pnpm installs web/camo/avatar, all into volumes - that a container can't fill from inside `internal: true`: + This writes the image directory under `out/localinfra-spindle-images`. +5. `podman compose build`, then please warm the build cache volumes, + since crates.io and the npm registry are unreachable from inside `tngl`: ```bash ./localinfra/scripts/warm-caches.sh ``` - The script calls `podman run` with the volume and image names podman-compose - uses, so on docker you run its five commands by hand. Run it again after a - `Cargo.lock` or `pnpm-lock.yaml` change. Otherwise the container-start - `cargo build` and `pnpm install` try crates.io and the npm registry from - inside `tngl`, where a stale cache reads as a network error. -6. `podman compose up` (or `docker compose up`) + The script calls `podman run` directly and won't work on docker. Please run + it again after a `Cargo.lock` or `pnpm-lock.yaml` change, or `cargo build` + and `pnpm install` loop at container start. +6. `podman compose up` 7. AppView will be running on `127.0.0.1:3000` with four test users: `alice`, `bob`, `charlie`, and `david`, all under `pds.tngl.boltless.dev` and using the password `password`. Deliberi also bootstraps a verified `${user}@pds.tngl.boltless.dev` address for each account. Use that address as `user.email` when making local Git commits that should appear in profile activity. - On rootless podman, caddy's published `80:80`/`443:443` need - `net.ipv4.ip_unprivileged_port_start` at or below 80 (`sudo sysctl -w - net.ipv4.ip_unprivileged_port_start=80`, persisted in the host config). - - Rootless podman with `internal: true` has one more trap: netavark doesn't - write firewall rules for an internal network, and that includes the accept - rule for aardvark-dns on the bridge gateway. Where the host input policy - drops, containers lose name resolution rather than egress, and `postgres` - and `redis` failing to resolve reads as a broken DNS setup. See - [netavark#1055](https://github.com/containers/netavark/issues/1055) and - [podman#26917](https://github.com/containers/podman/issues/26917). Docker - answers on 127.0.0.11 without a gateway hop and is unaffected. - - Signup is the one flow `internal: true` shuts off: the appview and deliberi - both verify a Turnstile token against `challenges.cloudflare.com` from - `tngl`. With the secret key unset, which is how the dev stack runs, each - bails out before the call, so signup answers a captcha error. Set - `TANGLED_CLOUDFLARE_TURNSTILE_SECRET_KEY` or - `DELIBERI_TURNSTILE_SECRET_KEY` and it answers 500 until you put both - services on `upstream-cache` too. + On rootless podman, caddy's `80:80`/`443:443` need + `sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80`. netavark writes no + firewall rules for an internal network, and aardvark-dns answers on the + gateway. If your host drops input by default, trust the `11.0.0.0/24` subnet + (`firewall-cmd --zone=trusted --add-source` under firewalld). Until you do, + `postgres` won't resolve and egress still will + ([netavark#1055](https://github.com/containers/netavark/issues/1055), + [podman#26917](https://github.com/containers/podman/issues/26917)). + + Signup stays broken here: the appview and deliberi both check the Turnstile + token against `challenges.cloudflare.com` from `tngl`. With the secret key + unset you get a captcha error, and with it set you get a 500. `TANGLED_APPVIEW_HOST` must be a loopback IP with the mapped port (`127.0.0.1:3000`), not `localhost`: atproto's dev OAuth client requires a loopback IP for the redirect URI. If you remap the published appview port, update `TANGLED_APPVIEW_HOST` in `docker-compose.yml` to match. @@ -182,6 +162,6 @@ The executors differ on the two axes placement cares about, so you can watch can | executor-b | `linux, slow` | full | `image/alpine, image/nixos` | | executor-c | `linux, gpu` | alpine only | `image/alpine` | -So `image: nixos` has two candidates, `image: alpine` has three, and `runs_on: [gpu]` pins to executor-c. The alpine-only image set is staged by `prepare-spindle-images.sh` (step 5) alongside the full one, no extra step. +So `image: nixos` has two candidates, `image: alpine` has three, and `runs_on: [gpu]` pins to executor-c. The alpine-only image set is staged by `prepare-spindle-images.sh` (step 4) alongside the full one, no extra step. Each executor needs its own identity (one live session per token), so the `mill-tokens` service registers a token per executor in the mill db and drops it into the shared volume for the executor to read.