localinfra · core

Tangled's setup is slightly more involved because services inside the network need to reach the PDS over its public hostname with valid TLS — federation paths (DID resolution, OAuth, etc.) round-trip through the same URLs an external client would use.

For example, resolving alice.pds.tngl.boltless.dev yields an #atproto_pds service pointing at https://pds.tngl.boltless.dev. Knot and spindle running inside docker must hit that exact URL and trust its cert.

To make that work:

Caddy's dev root CA is mounted into every container that talks to another service over HTTPS.
The Docker network uses an unrouted "public" subnet so the SSRF dialer doesn't reject container IPs as private.
Only the following services can fetch: ncps and spindle for nix, and knot2 for the core clone off knot1.tangled.sh. docker-compose.yml puts those on a second upstream-cache bridge and leaves the rest on tngl, which is marked internal: true. DNS on that network fails for any name outside the project.

What's inside: #

did-method-plc (https://plc.tngl.boltless.dev)
atproto_pds (https://pds.tngl.boltless.dev)
knot (https://knot2.tngl.boltless.dev)
spindle (https://spindle.tngl.boltless.dev)
knotmirror (https://mirror.tngl.boltless.dev)
appview (https://tngl.boltless.dev) (live reloading)
ncps nix binary cache (internal, http://ncps:8501)
pdsls (https://pdsls.tngl.boltless.dev)
bobbin (https://bobbin.tngl.boltless.dev, host :8090)
hydrant indexer + record/identity resolver feeding bobbin; jetstream
pocket preferences store (https://pocket.tngl.boltless.dev, host :3100).
camo (https://camo.tngl.boltless.dev) and avatar (https://avatar.tngl.boltless.dev), the workers from camo/ and avatar/ under wrangler. avatar resolves against the local plc, so it serves real avatars for accounts that have one and a generated placeholder otherwise
web/ sveltekit frontend (host 127.0.0.1:5174, live reloading)
caddy reverse proxy

Setup #

Generate the dev CA from the repo root:

mkdir -p localinfra/certs &&
openssl req -x509 -newkey rsa:2048 \
    -keyout localinfra/certs/root.key \
    -out localinfra/certs/root.crt \
    -days 3650 -nodes \
    -subj "/CN=Tangled Dev CA" \
    -addext "basicConstraints=critical,CA:TRUE,pathlen:1" \
    -addext "keyUsage=critical,keyCertSign,cRLSign" \
    -addext "nameConstraints=critical,permitted;DNS:tngl.boltless.dev"

test -f localinfra/certs/service-auth.env || (
    umask 077
    nix develop .#default -c go run ./cmd/knotmirror generate-service-auth-key \
        > localinfra/certs/service-auth.env
)

Trust generated localinfra/certs/root.crt in your system's trust store.

For example in MacOS, run

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain \
  ./localinfra/certs/root.crt

Depending on your browser you may have to import the certificate into your browser profiles too as some have their own certs do not use your system ones

Please fetch the appview's other vendored static assets, the ones the tailwind service doesn't write alongside tw.css:
```
./localinfra/scripts/appview-static-files.sh
```
Prepare the spindle microVM images:
```
./localinfra/scripts/prepare-spindle-images.sh
```
This writes the image directory under out/localinfra-spindle-images.
podman compose build, then please warm the build cache volumes, since crates.io and the npm registry are unreachable from inside tngl:
```
./localinfra/scripts/warm-caches.sh
```
The script calls podman run directly and won't work on docker. Please run it again after a Cargo.lock or pnpm-lock.yaml change, or cargo build and pnpm install loop at container start.
podman compose up
AppView will be running on 127.0.0.1:3000 with four test users: alice, bob, charlie, and david, all under pds.tngl.boltless.dev and using the password password. Deliberi also bootstraps a verified ${user}@pds.tngl.boltless.dev address for each account. Use that address as user.email when making local Git commits that should appear in profile activity.

On rootless podman, caddy's 80:80/443:443 need sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80. netavark writes no firewall rules for an internal network, and aardvark-dns answers on the gateway. If your host drops input by default, trust the 11.0.0.0/24 subnet (firewall-cmd --zone=trusted --add-source under firewalld). Until you do, postgres won't resolve and egress still will (netavark#1055, podman#26917).

Signup stays broken here: the appview and deliberi both check the Turnstile token against challenges.cloudflare.com from tngl. With the secret key unset you get a captcha error, and with it set you get a 500.

TANGLED_APPVIEW_HOST must be a loopback IP with the mapped port (127.0.0.1:3000), not localhost: atproto's dev OAuth client requires a loopback IP for the redirect URI. If you remap the published appview port, update TANGLED_APPVIEW_HOST in docker-compose.yml to match.

Observability #

Prometheus, Grafana, Tempo, and Loki are available through the optional observability profile. To run the standalone spindle with metrics, traces, and remote logs:

SPINDLE_TRACING_ENDPOINT=tempo:4318 \
SPINDLE_TRACING_INSECURE=true \
SPINDLE_LOGGING_ENDPOINT=loki:3100 \
SPINDLE_LOGGING_INSECURE=true \
docker compose --profile linux --profile observability up

The endpoint values above are addresses on the Compose network. Leaving either endpoint empty disables that OTLP exporter; Prometheus metrics and JSON logs on stderr remain enabled.

The profile exposes:

Grafana at http://127.0.0.1:3001, with the Spindle Overview dashboard and Prometheus, Tempo, and Loki datasources provisioned
Prometheus at http://127.0.0.1:9090; use /targets to verify the internal spindle:9091/metrics scrape
Tempo at http://127.0.0.1:3200
Loki at http://127.0.0.1:3100

For mill mode, include both Compose files:

SPINDLE_TRACING_ENDPOINT=tempo:4318 \
SPINDLE_TRACING_INSECURE=true \
SPINDLE_LOGGING_ENDPOINT=loki:3100 \
SPINDLE_LOGGING_INSECURE=true \
docker compose -f docker-compose.yml -f docker-compose.mill.yml --profile linux --profile observability up

The mill Prometheus configuration scrapes the mill host and all three executors. Grafana links metric exemplars to Tempo traces and trace IDs in Loki logs back to Tempo.

bobbin stack #

The web service runs vite dev against the local bobbin on http://127.0.0.1:5174 (port 5174 so it doesn't clash with a host-side pnpm dev on 5173). For host-side web/ dev, point the frontend at the local bobbin (e.g. in web/.env):

BOBBIN_URL=http://127.0.0.1:8090
# leave unset to read repositories directly from their knots
KNOTMIRROR_URL=https://mirror.tngl.boltless.dev
VITE_HANDLE_RESOLVER_URL=https://pds.tngl.boltless.dev
VITE_PLC_DIRECTORY_URL=https://plc.tngl.boltless.dev

Mill mode #

The default stack runs one standalone spindle. docker-compose.mill.yml is an overlay that splits that into the distributed arch: the primary spindle becomes the mill host (role=mill, it only places jobs) and a three-executor fleet (role=executor) runs the engines.

docker compose -f docker-compose.yml -f docker-compose.mill.yml --profile linux up

The executors differ on the two axes placement cares about, so you can watch candidates get filtered and ranked:

executor	labels	images	runs
executor-a	`linux, fast`	full	`image/alpine, image/nixos`
executor-b	`linux, slow`	full	`image/alpine, image/nixos`
executor-c	`linux, gpu`	alpine only	`image/alpine`

So image: nixos has two candidates, image: alpine has three, and runs_on: [gpu] pins to executor-c. The alpine-only image set is staged by prepare-spindle-images.sh (step 4) alongside the full one, no extra step.

Each executor needs its own identity (one live session per token), so the mill-tokens service registers a token per executor in the mill db and drops it into the shared volume for the executor to read.