Heavily inspired by [frontpage dev environment](https://github.com/frontpagefyi/frontpage/blob/10678df9c3f72cbd82f0856a9f99c74dd22326d8/apps/frontpage/local-infra/README.md). Tangled's setup is slightly more involved because services inside the network need to reach the PDS over its **public** hostname with **valid TLS** — federation paths (DID resolution, OAuth, etc.) round-trip through the same URLs an external client would use. For example, resolving `alice.pds.tngl.boltless.dev` yields an `#atproto_pds` service pointing at `https://pds.tngl.boltless.dev`. Knot and spindle running inside docker must hit that exact URL and trust its cert. To make that work: - Caddy's dev root CA is mounted into every container that talks to another service over HTTPS. - The Docker network uses an unrouted "public" subnet so the SSRF dialer doesn't reject container IPs as private. - Only the following services can fetch: `ncps` and `spindle` for nix, and `knot2` for the `core` clone off `knot1.tangled.sh`. `docker-compose.yml` puts those on a second `upstream-cache` bridge and leaves the rest on `tngl`, which is marked `internal: true`. DNS on that network fails for any name outside the project. ## What's inside: - [did-method-plc](https://github.com/did-method-plc/did-method-plc) () - atproto_pds () - knot () - spindle () - knotmirror () - appview () (live reloading) - [ncps](https://github.com/kalbasit/ncps) nix binary cache (internal, `http://ncps:8501`) - pdsls () - bobbin (, host `:8090`) - hydrant indexer + record/identity resolver feeding bobbin; jetstream - [pocket](https://tangled.org/microcosm.blue/microcosm-rs) preferences store (, host `:3100`). - camo () and avatar (), the workers from `camo/` and `avatar/` under wrangler. avatar resolves against the local plc, so it serves real avatars for accounts that have one and a generated placeholder otherwise - web/ sveltekit frontend (host `127.0.0.1:5174`, live reloading) - caddy reverse proxy ## Setup 1. Generate the dev CA and the local secrets from the repo root: ```bash ./localinfra/scripts/make-certs.sh ``` It writes root.crt plus the env files the services read (github.env, migrator.env, service-auth.env) and skips whatever already exists. 2. Trust generated `localinfra/certs/root.crt` in your system's trust store. - For example in MacOS, run ```bash sudo security add-trusted-cert -d -r trustRoot \ -k /Library/Keychains/System.keychain \ ./localinfra/certs/root.crt ``` - Depending on your browser you may have to import the certificate into your browser profiles too as some have their own certs do not use your system ones 3. Please fetch the appview's other vendored static assets, the ones the tailwind service doesn't write alongside `tw.css`: ```bash ./localinfra/scripts/appview-static-files.sh ``` 4. Prepare the spindle microVM images: ```bash ./localinfra/scripts/prepare-spindle-images.sh ``` This writes the image directory under `out/localinfra-spindle-images`. 5. `podman compose build`, then please warm the build cache volumes, since crates.io and the npm registry are unreachable from inside `tngl`: ```bash ./localinfra/scripts/warm-caches.sh ``` The script calls `podman run` directly and won't work on docker. Please run it again after a `Cargo.lock` or `pnpm-lock.yaml` change, or `cargo build` and `pnpm install` loop at container start. 6. `podman compose up` 7. AppView will be running on `127.0.0.1:3000` with four test users: `alice`, `bob`, `charlie`, and `david`, all under `pds.tngl.boltless.dev` and using the password `password`. Deliberi also bootstraps a verified `${user}@pds.tngl.boltless.dev` address for each account. Use that address as `user.email` when making local Git commits that should appear in profile activity. On rootless podman, caddy's `80:80`/`443:443` need `sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80`. netavark writes no firewall rules for an internal network, and aardvark-dns answers on the gateway. If your host drops input by default, trust the `11.0.0.0/24` subnet (`firewall-cmd --zone=trusted --add-source` under firewalld). Until you do, `postgres` won't resolve and egress still will ([netavark#1055](https://github.com/containers/netavark/issues/1055), [podman#26917](https://github.com/containers/podman/issues/26917)). Signup stays broken here: the appview and deliberi both check the Turnstile token against `challenges.cloudflare.com` from `tngl`. With the secret key unset you get a captcha error, and with it set you get a 500. `TANGLED_APPVIEW_HOST` must be a loopback IP with the mapped port (`127.0.0.1:3000`), not `localhost`: atproto's dev OAuth client requires a loopback IP for the redirect URI. If you remap the published appview port, update `TANGLED_APPVIEW_HOST` in `docker-compose.yml` to match. ## Observability Prometheus, Grafana, Tempo, and Loki are available through the optional `observability` profile. To run the standalone spindle with metrics, traces, and remote logs: ```bash SPINDLE_TRACING_ENDPOINT=tempo:4318 \ SPINDLE_TRACING_INSECURE=true \ SPINDLE_LOGGING_ENDPOINT=loki:3100 \ SPINDLE_LOGGING_INSECURE=true \ docker compose --profile linux --profile observability up ``` The endpoint values above are addresses on the Compose network. Leaving either endpoint empty disables that OTLP exporter; Prometheus metrics and JSON logs on stderr remain enabled. The profile exposes: - Grafana at , with the `Spindle Overview` dashboard and Prometheus, Tempo, and Loki datasources provisioned - Prometheus at ; use `/targets` to verify the internal `spindle:9091/metrics` scrape - Tempo at - Loki at For mill mode, include both Compose files: ```bash SPINDLE_TRACING_ENDPOINT=tempo:4318 \ SPINDLE_TRACING_INSECURE=true \ SPINDLE_LOGGING_ENDPOINT=loki:3100 \ SPINDLE_LOGGING_INSECURE=true \ docker compose -f docker-compose.yml -f docker-compose.mill.yml --profile linux --profile observability up ``` The mill Prometheus configuration scrapes the mill host and all three executors. Grafana links metric exemplars to Tempo traces and trace IDs in Loki logs back to Tempo. ## bobbin stack The `web` service runs `vite dev` against the local bobbin on `http://127.0.0.1:5174` (port 5174 so it doesn't clash with a host-side `pnpm dev` on 5173). For host-side `web/` dev, point the frontend at the local bobbin (e.g. in `web/.env`): ```bash BOBBIN_URL=http://127.0.0.1:8090 # leave unset to read repositories directly from their knots KNOTMIRROR_URL=https://mirror.tngl.boltless.dev VITE_HANDLE_RESOLVER_URL=https://pds.tngl.boltless.dev VITE_PLC_DIRECTORY_URL=https://plc.tngl.boltless.dev ``` ## Mill mode The default stack runs one standalone spindle. `docker-compose.mill.yml` is an overlay that splits that into the distributed arch: the primary spindle becomes the mill host (`role=mill`, it only places jobs) and a three-executor fleet (`role=executor`) runs the engines. ```bash docker compose -f docker-compose.yml -f docker-compose.mill.yml --profile linux up ``` The executors differ on the two axes placement cares about, so you can watch candidates get filtered and ranked: | executor | labels | images | runs | |------------|---------------|-------------|-----------------------------| | executor-a | `linux, fast` | full | `image/alpine, image/nixos` | | executor-b | `linux, slow` | full | `image/alpine, image/nixos` | | executor-c | `linux, gpu` | alpine only | `image/alpine` | So `image: nixos` has two candidates, `image: alpine` has three, and `runs_on: [gpu]` pins to executor-c. The alpine-only image set is staged by `prepare-spindle-images.sh` (step 4) alongside the full one, no extra step. Each executor needs its own identity (one live session per token), so the `mill-tokens` service registers a token per executor in the mill db and drops it into the shared volume for the executor to read.