Aether Substrate #

Containerized Linux environment that powers Aether OS.

Allows you to run atproto tools like Tap and Goat, use local LLMs to work with containers and a dedicated workspace, and power Aether OS system services for doing more with your local AT Protocol data.

What It Powers in Aether OS #

• ATProto firehose + repo tracking: Tap indexer with SQLite storage, runtime repo management, and WebSocket event streaming.
• GOAT CLI access: resolve handles and DIDs, fetch records, list collections, run curated Goat commands through an authenticated gateway.
• Semantic search: optional EmbeddingGemma vector indexing with Tap sync, backfills, similar-record lookup, and vector export.
• Persistent agent workspace: long-lived container with file I/O, foreground/background exec, guarded installs, SSE events, and terminal access.
• Managed sandbox containers: allowlisted images with CPU/memory caps, toggleable internet, repo cloning, exec, background jobs, and HTTP app proxying.
• LLM gateway: server-side provider credentials, pluggable providers, and OpenAI-compatible streaming chat with tool use.
• Web access utilities: plain scrape, search, and agentic browse for when a fetch is not enough.
• Local-first networking: binds to 127.0.0.1 by default, local HTTPS via Caddy, optional Cloudflare Tunnel or Tailscale for remote access.
• Security guardrails: authenticated REST/WebSocket, signed proxy tokens, origin checks, rate limits, size limits, SSRF checks, and image allowlists.

Requirements #

Requirement	Details
Container runtime	Any OCI runtime with a Docker-compatible CLI and a Compose plugin. See Supported runtimes below.
Compose	Compose v2 (docker compose / podman compose). The standalone docker-compose v1 binary is not supported.
Disk	~4 GB for the image + model cache. Vector search adds ~150 MB of ONNX model downloads on first run.
Memory	The core stack typically sits around ~600 MB RAM without workspace containers. Plan for about 1 GB for Substrate + Tap + Caddy, then add headroom for workspace containers, managed containers, embeddings, or model workloads. docker-compose.yml sets a 4 GB upper limit for headroom by default; that is not the minimum required.
OS	macOS, Linux, or Windows with WSL2. The source build works on ARM64 and x86_64 hosts.
Node.js (CLI only)	Node 20+ is required to run the aether-substrate CLI. Only needed if you want the interactive operator workflow. Running the stack directly via docker compose / podman compose has no Node dependency.

Installation #

Use one of these paths:

1. Substrate CLI: recommended for most people. It handles .env, runtime detection, build/start, health checks, and optional tunnel setup.
2. Build it yourself: use this if you are changing the container, API, or compose files directly.

Path 1: Substrate CLI (recommended) #

Best for: getting a working local Substrate quickly with the least manual setup.

The aether-substrate CLI wraps build, startup, config editing, health checks, and remote-access setup. It auto-detects Docker or Podman on your PATH.

# Clone this repo if you do not already have it
git clone git@tangled.org:aetheros.computer/aether-substrate
cd aether-substrate

# Build and install the CLI from the local checkout
cd cli
npm install
npm run build
npm link
cd ..

# Interactive bootstrap: env, feature flags, build, startup, optional tunnels
aether-substrate init

# Useful follow-up commands
aether-substrate doctor
aether-substrate status
aether-substrate logs
aether-substrate config --enable containers --enable workspace --restart
aether-substrate tunnel cloudflare --hostname substrate.example.com
aether-substrate tunnel tailscale

Notes:

• aether-substrate init --runtime podman forces Podman instead of Docker autodetection.
• aether-substrate up is the non-interactive path once your .env is already configured.
• The CLI resolves the substrate/ directory automatically, so it does not matter where you run it from after linking it.
• Once the stack is up, docker compose ps or Docker Desktop should show at least aether-substrate and aether-caddy. If workspace support is enabled, you will also see aetheros-user-workspace.

Path 2: Build It Yourself From Source #

Best for: custom images, local development, debugging, and changing compose or Docker behavior directly.

git clone git@tangled.org:aetheros.computer/aether-substrate
cd aether-substrate

cp .env.example .env
# Edit .env and set SUBSTRATE_API_KEY (generate one: openssl rand -hex 32)

docker compose up -d --build
# or: podman compose up -d --build

curl http://127.0.0.1:3100/health
docker compose ps
docker compose logs -f substrate

This path builds from the local Dockerfile and the substrate-api/ source tree.

Once the stack is up, docker compose ps or Docker Desktop should show at least aether-substrate and aether-caddy. If workspace support is enabled, you will also see aetheros-user-workspace.

Architecture #

┌─────────────────────────────────────────────────┐
│  Host Machine                                   │
│                                                 │
│  ┌──────────┐    ┌────────────────────────────┐ │
│  │  Caddy   │───▶│   Substrate Container      │ │
│  │  :8443   │    │                            │ │
│  │  (HTTPS) │    │  ┌──────────────────────┐  │ │
│  └──────────┘    │  │  Hono API  :3100     │  │ │
│                  │  │  REST + WebSocket    │  │ │
│                  │  └──────────────────────┘  │ │
│                  │  ┌──────┐ ┌─────┐          │ │
│                  │  │ Tap  │ │ PTY │          │ │
│                  │  │:2480 │ │     │          │ │
│                  │  └──────┘ └─────┘          │ │
│                  │  ┌──────────────────────┐  │ │
│                  │  │  /data (SQLite vol)  │  │ │
│                  │  └──────────────────────┘  │ │
│                  └────────────────────────────┘ │
└─────────────────────────────────────────────────┘

Services #

Service	Port	Description
Substrate API	3100	Hono REST/WebSocket gateway
Tap	2480	ATProto firehose indexer (Go)
Embedding server	3101	Internal ONNX embedding server (Go)
Caddy	8443	HTTPS reverse proxy (internal TLS)

API Endpoints #

Public #

• GET /health: Container and service health

Authenticated (require x-substrate-key header) #

System #

• GET /services: Detailed service status
• GET /fs/read: Read an allowed file as base64
• POST /exec: Run legacy allowlisted commands when SUBSTRATE_ENABLE_EXEC=true

Repo Management #

• GET /repos: List tracked repos
• GET /collections: List indexed collections
• GET /records: Query indexed records
• POST /repos/add: Add DIDs to track
• POST /repos/remove: Remove tracked DIDs
• GET /repos/:did: Get repo info

Goat CLI #

• GET /goat/resolve/:identifier: Resolve handle/DID
• GET /goat/get?uri=: Get AT Protocol record
• GET /goat/ls/:did: List records
• POST /goat/exec: Execute Goat command

Containers #

• GET /containers: List managed containers
• POST /containers: Create container
• GET /containers/images/allowed: Get image allowlist
• GET /containers/:id/logs: Fetch logs
• POST /containers/clone: Create container and clone repo
• POST /containers/:id/clone: Clone repo into existing container
• GET /containers/:id/files: List files
• GET /containers/:id/files/read: Read file
• POST /containers/:id/files/write: Write file
• POST /containers/:id/exec: Execute command in shell or argv mode
• GET /containers/:id/jobs: List background jobs
• GET /containers/:id/jobs/:jobId: Inspect one background job
• POST /containers/:id/jobs/:jobId/cancel: Cancel a background job
• DELETE /containers/:id: Remove container

Workspace #

• POST /workspace/ensure: Create or fetch the persistent workspace container
  • When workspace is enabled, Substrate also bootstraps this container during startup.
• GET /workspace/files: List workspace files
• GET /workspace/files/read: Read a workspace file
• POST /workspace/files/write: Write a workspace file
• POST /workspace/exec: Execute a workspace command in shell or argv mode
• POST /workspace/install: Install packages with strict validation
• GET /workspace/jobs: List workspace background jobs
• GET /workspace/jobs/:jobId: Inspect a workspace background job
• POST /workspace/jobs/:jobId/cancel: Cancel a workspace background job

WebSocket #

• WS /ws?type=events: Stream Tap firehose events
• WS /ws?type=terminal: Substrate host PTY terminal session
• WS /ws?type=container-terminal&workspace=1: Workspace terminal
• WS /ws?type=container-terminal&containerId=X: Managed container terminal

WebSocket auth uses a first message of { "type": "auth", "key": "..." }. Raw API keys in WebSocket query strings are intentionally rejected.

Configuration #

Environment Variables #

Variable	Default	Description
SUBSTRATE_API_KEY	(required)	Shared auth key. Generate with openssl rand -hex 32.
CONTAINER_RUNTIME	docker (auto-detected)	Container CLI the aether-substrate tool shells out to. Supported: docker, podman, nerdctl, or any docker-compatible binary on your PATH.
DOCKER_GID	20 on macOS, Linux-dependent elsewhere	Container socket group ID. For rootless Podman, set to your user GID (id -g).
PORT	3100	API server port
TAP_DATABASE_URL	sqlite:///data/tap.db	Tap SQLite database path
TAP_URL	http://localhost:2480	Tap base URL used by the API
TAP_RELAY_URL	relay1.us-east.bsky.network	ATProto relay to connect to
TAP_DISABLE_ACKS	true	Fire-and-forget mode
TAP_LOG_LEVEL	info	Tap log verbosity
TAP_FULL_NETWORK	false	Index full network (warning: heavy)
TAP_COLLECTION_FILTERS	computer.aetheros.*,app.bsky.*	Comma-separated collection patterns
SUBSTRATE_ALLOWED_ORIGINS	(empty)	Extra allowed HTTP/WebSocket origins
SUBSTRATE_ENABLE_EXEC	false	Enable legacy system exec endpoint
SUBSTRATE_ENABLE_CONTAINERS	false	Enable container management (Docker/Podman socket required inside the container)
SUBSTRATE_MAX_CONTAINERS	5	Max managed containers
SUBSTRATE_CONTAINER_MAX_CPU	1.0	CPU limit per container
SUBSTRATE_CONTAINER_MAX_MEMORY_MB	512	Memory limit per container (MB)
SUBSTRATE_CONTAINER_IMAGE_ALLOWLIST	built-in allowlist	Images allowed for managed containers
SUBSTRATE_ENABLE_WORKSPACE	false	Enable workspace container and pre-create it during Substrate startup
SUBSTRATE_WORKSPACE_IMAGE	node:22-alpine	Workspace container image
SUBSTRATE_ENABLE_TERMINAL_WS	false	Enable Substrate host terminal WebSocket
SUBSTRATE_HOST	localhost	Hostname for Caddy HTTPS proxy
SUBSTRATE_HTTPS_PORT	8443	HTTPS port for Caddy

.env.example #

# Required. Generate with: openssl rand -hex 32
SUBSTRATE_API_KEY=

# Container runtime used by the CLI. docker (default), podman, nerdctl, or a custom binary
# CONTAINER_RUNTIME=docker

# Container socket group ID
# macOS: 20 (staff)  |  Linux docker: `getent group docker | cut -d: -f3`  |  Rootless Podman: `id -g`
DOCKER_GID=20

# Feature flags
SUBSTRATE_ENABLE_CONTAINERS=true
SUBSTRATE_ENABLE_WORKSPACE=true
SUBSTRATE_ENABLE_TERMINAL_WS=false

Networking & Remote Access #

By default, all ports bind to 127.0.0.1, so the Substrate is only accessible from the local machine. For remote or persistent HTTPS access, use one of these tunnel options:

Method	Best For	Guide
Cloudflare Tunnel	Always-on HTTPS, custom domains, zero open ports	docs/cloudflared.md
Tailscale	Private mesh VPN, team access, MagicDNS	docs/tailscale.md
Caddy (built-in)	Local HTTPS, LAN access	Already included in docker-compose

See docs/networking.md for an overview of all networking options and when to use each.

Data Persistence #

Data is stored in a Docker volume substrate-data mounted at /data:

• tap.db: SQLite database with tracked repos, records, and events
• models/: ONNX model cache for the embedding server

Project Structure #

substrate/
├── Dockerfile                      # Multi-stage build (Go + Node.js)
├── Caddyfile                       # HTTPS reverse proxy config
├── docker-compose.yml              # Full dev composition
├── docker-compose.cloudflared.yml  # Cloudflare tunnel overlay
├── docs/
│   ├── networking.md               # Networking overview
│   ├── cloudflared.md              # Cloudflare Tunnel guide
│   └── tailscale.md                # Tailscale guide
├── cli/                            # `aether-substrate` operator CLI (Node)
├── tools/
│   ├── export-standalone-repo.sh   # Fresh standalone repo export helper
│   └── embedding-server/           # Go ONNX embedding service
└── substrate-api/
    ├── src/
    │   ├── index.ts                # Entry point (HTTP + WebSocket)
    │   ├── app.ts                  # Hono app, middleware, routes
    │   ├── config.ts               # Environment config
    │   ├── routes/                  # API route handlers
    │   │   ├── containers.ts
    │   │   ├── embeddings.ts
    │   │   ├── events.ts
    │   │   ├── goat.ts
    │   │   ├── llm.ts
    │   │   ├── system.ts
    │   │   ├── tap.ts
    │   │   ├── vm.ts
    │   │   ├── web.ts
    │   │   └── workspace.ts
    │   ├── services/               # Business logic
    │   │   ├── container-manager.ts
    │   │   ├── db.ts
    │   │   ├── embeddings/
    │   │   ├── execution-core.ts
    │   │   ├── goat.ts
    │   │   ├── llm.ts
    │   │   ├── lsp-bridge.ts
    │   │   ├── scraper.ts
    │   │   ├── service-manager.ts
    │   │   ├── tap.ts
    │   │   ├── terminal.ts
    │   │   ├── vm-registry.ts
    │   │   └── workspace-watcher.ts
    │   ├── types/
    │   └── utils/
    ├── package.json
    └── tsconfig.json

Supported runtimes #

Runtime	Status	Notes
Docker Desktop	Supported	Reference implementation.
OrbStack	Supported	Ships a drop-in docker CLI; nothing else to configure.
Rancher Desktop	Supported	Use the dockerd (moby) backend.
Colima	Supported	colima start gives you a working docker CLI.
Podman (4.6+)	Supported	Set CONTAINER_RUNTIME=podman. Requires either the built-in podman compose subcommand or podman-compose. See Podman notes.
nerdctl / containerd	Experimental	Set CONTAINER_RUNTIME=nerdctl. Compose plugin from nerdctl must be present.

Any other binary that implements the docker CLI surface can be passed via CONTAINER_RUNTIME=<binary> or aether-substrate init --runtime <binary>.

Podman notes #

• Rootless Podman does not have a docker group, so set DOCKER_GID to your user GID (id -g). The CLI does this automatically during init.
• Enable the Podman API socket before starting the stack so Substrate's container manager can talk to it:

  systemctl --user enable --now podman.socket
  export DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock
  

  Then pass DOCKER_HOST into compose, or add a volumes: mapping in docker-compose.yml from /run/user/$UID/podman/podman.sock → /var/run/docker.sock.
• SUBSTRATE_ENABLE_CONTAINERS=true needs the Podman socket accessible inside the container. If you only want the API and Tap, leave that flag off and none of the socket wiring is needed.

License #

Aether Substrate is licensed under the Apache License 2.0. You are free to use, modify, and redistribute the software, including for commercial purposes, provided you preserve attribution and the license notice. See docs/DISCLAIMER.md for the experimental-status and liability disclaimer that accompanies this license.