Halcyon is a single Go binary that replaces Traefik and some associated bits in my highly personal setup. It's here mainly just because fuck it, why not, and also because well, fuck it, why not.

It comes with raft based leader election, a raft-replicated Let's Encrypt certificate store, built-in ATproto OAuth authentication, and a slightly more nginx-ish way of configuring things.

Oh and a dashboard. Can't forget the dashboard!

AI usage statement #

This entire thing was built by AI. Designed by angrydutchman.online, Claude wrote code. I reviewed it and suggested changes. Claude made changes. Rinse, repeat, and here we are. If this offends your sensibilities then by all means don't use this - in fact I actually recommend nobody use this in production because I built it, for me, to scratch my own itches. The usual disclaimers apply. Here be dragons. Batteries not included. It may murderhobo all your things.

Security #

I've given this a rather brief once-over for the most common security issues but there's zero guarantees Halcyon can't be convinced to pwn your shit. Again. Don't use in production unless you're insane.

Before you start #

This thing requires that you have a domain at marque.at - the Let's Encrypt certificate issuer does DNS-01 challenges and needs to write DNS records. You will also need to prepare a hand-issued Let's Encrypt certificate (via acme.sh or whatever you feel comfortable using) - or any other valid SSL certificate that covers --admin-host and --bootstrap-domain - in order to get this to work because in order to issue certificates, it needs to write to your PDS, whcih it can't do until you've logged in through the admin panel once, which requires a certificate.

Yeah sorry, catch-22 of epic proportions but hey. Deal.

So to get around that, take your existing certificate (full chain), concatenate the key on the end, put it in a file, and point Halcyon at it with --bootstrap-certificate /path/to/file - it will load that certificate into the raft store so it can "do the thing".

Certificates that need to be issued for domains you (or whoever --admin-did is) don't own can only be issued via CNAME delegated verification, which requires you to set up your --verify-zone. All delegations must point to <domain>.verify.<verify-zone> - assuming a verify zone of example.com and I want to issue a certificate for test.org, test.org needs an _acme-challenge CNAME record pointing to test.org.verify.example.com - Halcyon does the rest.

Quick start #

make build

produces bin/halcyon.

bin/halcyon serve --dev --entrypoint web=8080 --routes-file dev/routes-dev.yaml

--dev relaxes every flag that is otherwise required in production (see the table below), binds the admin listener to loopback, and fills in safe fixed defaults for the cookie secret and admin PSK. It still requires at least one non-TLS --entrypoint, since /_halcyon/auth/* is served on the data plane, not the admin listener.

How to run this as a proper raft quorum-capable setup locally is left as an exercise to the reader. That'd be you.

Configuration reference #

Every flag also has a HALCYON_* environment variable equivalent (shown below); command-line flags take precedence. --entrypoint and --raft-peer are repeatable on the command line; from a single environment variable, separate multiple entries with | (not ,, since an --entrypoint spec uses commas for its own options), e.g. HALCYON_ENTRYPOINT="web=80|websecure=443,tls".

Flags marked required are enforced by internal/config/validate.go and fail halcyon serve/halcyon validate at startup with a named error; flags marked required outside --dev are only enforced in production mode.

Flag	Environment variable(s)	Default	Description
`--entrypoint`	`HALCYON_ENTRYPOINT`	none	Entrypoint spec (repeatable). Required — at least one, in every mode, dev included.
`--no-https-redirect`	`HALCYON_NO_HTTPS_REDIRECT`	`false`	Disable the automatic HTTP→HTTPS redirect on plain entrypoints.
`--advertise-addr`	`HALCYON_ADVERTISE_ADDR`	none	Routable IP this node advertises to the raft cluster. Required outside `--dev`; if set, `--raft-peer` must list it.
`--raft-peer`	`HALCYON_RAFT_PEER`	none	Raft peer address (repeatable). Not independently required, but the list must include `--advertise-addr` when that is set. A single `nomad:<service>` entry discovers peers from that Nomad service catalog entry instead — mutually exclusive with static peer addresses.
`--raft-data-dir`	`HALCYON_RAFT_DATA_DIR`	none	Directory for the raft log, snapshots, and stable store. Omitted means in-memory stores (no persistence across restarts).
`--bootstrap-expect`	`HALCYON_BOOTSTRAP_EXPECT`	`3`	Number of Nomad service registrations to wait for before bootstrapping, only meaningful with `--raft-peer nomad:<service>`; ignored with static `--raft-peer` addresses. Below 3 is accepted (a single-node dev cluster is legitimate) but logs a warning: any node beyond the first `--bootstrap-expect` to register waits forever and never opens its listeners.
`--node-id`	`HALCYON_NODE_ID`	none	This node's persistent raft identity, used only with `--raft-peer nomad:<service>` (with static peers, identity is always the advertise IP). Seeded once into a `node-id` file in `--raft-data-dir`, which then wins on every later start.
`--nomad-address`	`HALCYON_NOMAD_ADDRESS`, `NOMAD_ADDR`	none	Nomad API address.
`--nomad-token`	`HALCYON_NOMAD_TOKEN`, `NOMAD_TOKEN`	none	Nomad API token.
`--nomad-namespace`	`HALCYON_NOMAD_NAMESPACE`, `NOMAD_NAMESPACE`	`default`	Nomad namespace to watch for services.
`--routes-file`	`HALCYON_ROUTES_FILE`	none	YAML static routes file for backends that are not Nomad services.
`--admin-host`	`HALCYON_ADMIN_HOST`	none	Host the admin UI is reachable at through the proxy; the admin OAuth base URL is derived from it. Required outside `--dev` — without it there is no admin dashboard, no admin login, and so no PDS session for DNS-01 issuance. This domain is reserved: a tenant server (Nomad tags or `--routes-file`) that also declares it, on any entrypoint, is dropped and reported as a conflict rather than taking the dashboard offline — the admin server always wins.
`--admin-did`	`HALCYON_ADMIN_DID`	none	The single AT Protocol DID (`did:plc:...` or `did:web:...`) allowed to log into the admin UI. Required outside `--dev`; also required under `--dev` if `--admin-host` is set.
`--admin-psk`	`HALCYON_ADMIN_PSK`	none	Pre-shared bearer token gating the admin listener (including `/metrics`), minimum 32 bytes. Required outside `--dev`; under `--dev` falls back to a fixed, publicly-known development constant, and the real dev/production constants may never be swapped (a value matching the dev constant outside `--dev` is rejected).
`--cookie-secret`	`HALCYON_COOKIE_SECRET`	none	HMAC-SHA256 secret for signed session cookies, minimum 32 bytes. Required outside `--dev`; same dev-constant fallback/anti-swap rule as `--admin-psk`.
`--cookie-lifetime`	`HALCYON_COOKIE_LIFETIME`	`24h`	Default session lifetime before re-authentication is required (must be > 0).
`--verify-zone`	`HALCYON_VERIFY_ZONE`	none	DNS zone used for delegated ACME DNS-01 challenges (customer domains CNAME `_acme-challenge` here). Required outside `--dev`.
`--bootstrap-domain`	`HALCYON_BOOTSTRAP_DOMAIN`	none	Wildcard domain issued immediately on every leadership acquisition, ahead of the normal renewal sweep.
`--dns-propagation-timeout`	`HALCYON_DNS_PROPAGATION_TIMEOUT`	`2m`	Maximum time to wait for a DNS-01 TXT record to propagate (must be > 0).
`--bootstrap-certificate`	`HALCYON_BOOTSTRAP_CERTIFICATE`	none	Path to an operator-supplied PEM file imported into the raft cert store on leadership, breaking the admin dashboard's ACME chicken-and-egg deadlock (see Certificates below). By default holds both the certificate chain and the private key, concatenated in any order; pair with `--bootstrap-certificate-key` to keep them in separate files instead. Safe to leave set permanently — a bootstrap certificate already covered by a raft-held certificate with an equal or later expiry is not imported.
`--bootstrap-certificate-key`	`HALCYON_BOOTSTRAP_CERTIFICATE_KEY`	none	Path to the bootstrap certificate's private key, when kept separate from `--bootstrap-certificate` (which then holds the certificate chain only). Setting this without `--bootstrap-certificate` is a validation error.
`--body-spill-dir`	`HALCYON_BODY_SPILL_DIR`	none	Directory buffered request bodies spill to on disk; checked to exist and be writable at startup. Required outside `--dev` — no longer defaults to the OS temp dir, which is tmpfs (RAM) on many hosts; `--dev` falls back to it.
`--body-threshold`	`HALCYON_BODY_THRESHOLD`	`1MiB`	Bytes held in memory before a buffered body spills to disk (must be ≤ `--body-max`).
`--body-max`	`HALCYON_BODY_MAX`	`32MiB`	Maximum accepted request body size, in any mode (must be > 0); a hard ceiling — a server or location's own `body-max` may only lower it, never raise it.
`--body-spill-budget`	`HALCYON_BODY_SPILL_BUDGET`	`4GiB`	Process-wide aggregate byte ceiling on buffered request bodies (memory and disk combined) reserved at once, across every concurrent request (must be > 0); a request that cannot reserve its resolved `body-max` is answered `503` with `Retry-After`, never silently truncated.
`--response-header-timeout`	`HALCYON_RESPONSE_HEADER_TIMEOUT`	`30s`	Maximum time to wait for a backend's response headers before marking it down; `0` disables the timeout. Global default for `halcyon.server.backend.response-header-timeout`.
`--backend-failure-window`	`HALCYON_BACKEND_FAILURE_WINDOW`	`10s`	Passive-health failure window before a backend is marked down (must be > 0). Global default for `halcyon.server.backend.failure-window`.
`--backend-max-failures`	`HALCYON_BACKEND_MAX_FAILURES`	`3`	Failures within the window before a backend is marked down (must be > 0). Global default for `halcyon.server.backend.max-failures`.
`--backend-cooldown`	`HALCYON_BACKEND_COOLDOWN`	`30s`	How long a down backend sits out before one probe request (must be > 0). Global default for `halcyon.server.backend.cooldown`.
`--inactivity-timeout`	`HALCYON_INACTIVITY_TIMEOUT`	`0` (unlimited)	Maximum gap between response body bytes reaching the client before the request is aborted; `0` disables it, so long polling and held-open connections are never cut off unless a server or location explicitly sets one; a server or location may likewise set `0` to opt back out of a global value.
`--read-header-timeout`	`HALCYON_READ_HEADER_TIMEOUT`	`10s`	Maximum time a connection may take to send its request headers, on the data-plane entrypoints and the admin listener (must be > 0).
`--idle-timeout`	`HALCYON_IDLE_TIMEOUT`	`120s`	Maximum time a keep-alive connection may sit idle between requests, on the data-plane entrypoints and the admin listener (must be > 0).
`--request-inactivity-timeout`	`HALCYON_REQUEST_INACTIVITY_TIMEOUT`	`60s`	Maximum gap between bytes of a REQUEST body reaching this node before the connection is aborted — the request-direction counterpart of `--inactivity-timeout`, closing the slow/absent-body slowloris window net/http itself leaves open once headers are parsed; `0` disables it. Applies to the data-plane entrypoints and the admin listener.
`--max-connections`	`HALCYON_MAX_CONNECTIONS`	`10000`	Maximum simultaneous connections accepted per data-plane entrypoint; a connection past the cap waits for a slot rather than being refused. `0` disables the cap.
`--max-in-flight`	`HALCYON_MAX_IN_FLIGHT`	`4096`	Maximum requests processed concurrently per data-plane entrypoint; a request past the cap is answered `503` with `Retry-After`. `0` disables the cap.
`--max-header-bytes`	`HALCYON_MAX_HEADER_BYTES`	`32KiB`	Maximum size of request headers, on the data-plane entrypoints and the admin listener (must be > 0).
`--access-log`	`HALCYON_ACCESS_LOG`	`-` (stdout)	Access log path, or `-` for stdout.
`--log-level`	`HALCYON_LOG_LEVEL`	`info`	Log level: `debug`, `info`, `warn`, `error`.
`--dev`	`HALCYON_DEV`	`false`	Development mode: relaxes the flags marked above and fills in safe local defaults (dev cookie secret, dev admin PSK, loopback admin bind). Never use in production.

Entrypoint spec syntax #

--entrypoint name=port[,tls][,proxyproto][,trusted=CIDR[;CIDR...]]

name: must match ^[A-Za-z0-9_-]+$ (also used as a Prometheus label and as the halcyon.server.entrypoints tag value).
port: 1–65535; entrypoint names and ports must each be unique across the whole --entrypoint list.
tls: bare (tls) or tls=true/tls=false — terminates TLS on this entrypoint. Boolean options accept a bare key for true, or an explicit strconv.ParseBool-compatible value.
proxyproto: bare or proxyproto=<bool> — expect a PROXY protocol header ahead of the connection (Octavia); both v1 (text) and v2 (binary) headers are decoded, whichever the sender uses. Without trusted=..., proxyproto requires --dev (production must scope trust to real load-balancer CIDRs).
trusted=CIDR[;CIDR...]: semicolon-separated CIDR list of sources allowed to set PROXY protocol headers and forwarded headers (X-Forwarded-*); untrusted sources have those headers stripped and PROXY protocol ignored (never rejected, so health probes without a header still connect).

The one list is consulted at two different points, against two different addresses, and the distinction matters when sizing it:

The PROXY policy checks the raw TCP peer — the load balancer — and decides whether to believe its PROXY header (internal/entrypoint/listener.go).
The forwarded-header middleware runs afterwards, by which point RemoteAddr has already been rewritten to the real client address the PROXY header carried, and it is that corrected address the CIDRs are matched against (internal/entrypoint/forwarded.go).

So behind an L4 load balancer the header check effectively never matches, and every client-supplied X-Forwarded-* is stripped — which is the desired outcome, since an L4 balancer forwards the client's bytes verbatim and sets no forwarded headers of its own. Halcyon then sets X-Forwarded-For itself from the PROXY-derived client address (internal/proxy/backend.go), and does not add its own address: the backend already sees Halcyon as its TCP peer.

The caveat that follows: a client whose own address falls inside a trusted CIDR is believed for headers. Ingress node ports are usually reachable directly on the cluster network, so keep the list scoped to the balancer's actual addresses rather than the subnet containing them.

Unknown options, duplicate options on the same entrypoint, and a malformed name=port are all rejected at parse time. Example: --entrypoint web=80 --entrypoint websecure=443,tls,trusted=10.0.0.0/8.

Routes file schema #

--routes-file points at a YAML file for backends that are not Nomad services (its own admin UI is the motivating case: the Nomad UI cannot discover itself through the Nomad provider). Its shape mirrors the halcyon.* tag schema field for field, with one difference: backends is a list of URL strings, not objects, since there is no Nomad allocation to resolve an address from.

servers:
  - name: nomad-ui
    domains:
      - nomad.example.net
    entrypoints:
      - websecure
    backends:
      - http://10.0.2.29:4646
      - http://10.0.1.60:4646
      - http://10.0.1.65:4646
    passhost: true
    cert: auto
    auth:
      enabled: true
      allow:
        - did:plc:aaaaaaaaaaaaaaaaaaaaaaaa
      scopes:
        - atproto
        - account:email
      lifetime: 24h
    buffering:
      enabled: false
      threshold: 1MiB
    body-max: 1GiB
    hooks:
      quota:
        url: http://127.0.0.1:8081/v1/api/quota-check
        timeout: 2s
        priority: 50
      ratelimit:
        off: true
    locations:
      - name: metrics
        path-prefix: /metrics
        exclude: true
      - name: deny-write
        path-prefix: /
        return: 403

(did:plc:aaaaaaaaaaaaaaaaaaaaaaaa above is a placeholder shaped like a real did:plc: — 24 lowercase base32 characters — not a real account; a did:web:... identifier works the same way.)

Top level: servers (list). Per server: name (must match ^[A-Za-z0-9_.-]+$), domains (list), entrypoints (list, defaults to all TLS entrypoints), backends (list of URL strings — scheme must be http, https, or h2c, and if scheme is not set explicitly every backend URL must share one scheme), scheme, passhost (bool), cert (auto/off), locations-only (bool, default false — see "Sharing a domain between services" below; must not be combined with entrypoints, cert, or auth.cookie-domain on the same server), auth (enabled, allow, scopes, lifetime, and cookie-domain — the session cookie's Domain override, server-only: default host-only, and if set must equal or be a parent of every one of this server's own domains and must not itself be a public suffix; a location may not set this), buffering (enabled, threshold), body-max (size string), hooks (map keyed by hook kind — only quota and ratelimit are recognised), backend (server-only tuning — failure-window, max-failures, cooldown, response-header-timeout, hash — round-robin or ip; a location may not set this), inactivity-timeout (duration string), and locations (list). Per location: name (must match ^[A-Za-z0-9_-]+$), path or path-prefix (exactly one, must start with /), methods (list), return (int, 0 or 200–599, not 3xx; 0 is not "return status 0" — it means no static response is set, so the location proxies normally, same as omitting return entirely), exclude (bool), case-insensitive (bool), redirect (URL string; return, redirect, and exclude are mutually exclusive on the same location), auth/buffering/body-max/hooks/inactivity-timeout (same shape as server level, overriding it). Duration fields (auth.lifetime, hook.<kind>.timeout, backend.*, inactivity-timeout) use Go's time.ParseDuration grammar ("5m", "2s"); size fields (buffering.threshold, body-max) accept B/KB/MB/GB (decimal) or KiB/MiB/GiB (binary), e.g. "1MiB" - units are KB/MB/GB (decimal), KiB/MiB/GiB (binary), or a bare k/m/g as a binary alias (32m is 32 MiB); a bare number is bytes. Unknown keys are a hard decode error (yaml.KnownFields(true)), not silently ignored. dev/routes-dev.yaml is the minimal working example used by dev/run-dev.sh:

servers:
  - name: demo
    domains:
      - demo.local
    entrypoints:
      - web
    backends:
      - http://10.99.0.1:8090
    passhost: false

Tag schema (`halcyon.*` on a Nomad service) #

A server is one backend (a Nomad service, or a routes-file entry) plus the domains it answers for. A location is a path scope inside a server carrying its own options. Every option a server sets is the default for every location inside it; a location may override any of them. Resolution is three levels: flag default → halcyon.server.* → halcyon.location.<name>.*, checked in that order for every location-overridable option: auth.*, buffering.*, body-max, hook.<kind>.*, and inactivity-timeout. domain, entrypoints, scheme, passhost, and cert are server-only, with no location form at all (halcyon.location.<name>.scheme is an unknown-tag error); backend.* is likewise server-only (see below — locations share their server's balancer), and so is auth.cookie-domain — two locations on one server cannot each set a different Domain for the same session cookie. body-max resolves three-level exactly like everything else — a location's own body-max can be higher than its server's, not just lower — but it is always bounded by --body-max as a hard ceiling: a resolved value above that flag is silently clamped down to it, never rejected as an error.

Tag	Default	Notes
`halcyon.enable=true`	—	Required, opt-in. A service with no `halcyon.enable` is ignored entirely.
`halcyon.server.domain=foo.com *.foo.com`	—	Required. Space and/or comma separated; a wildcard is only valid as the leading label.
`halcyon.server.entrypoints=websecure`	all TLS entrypoints
`halcyon.server.scheme=http\|https\|h2c`	`http`	Used to talk to the backend, excepting we don't really do this in my setup so most of this plumbing remains an open unplumbed sewer. http works.
`halcyon.server.passhost=true\|false`	`true`
`halcyon.server.cert=auto\|off`	`auto`	Derive the certificate order from the server's domains.
`halcyon.server.locations-only`	off	Bare boolean. This server answers only for its own locations, composed into an "owner" server's routing on domains they share — see "Sharing a domain between services" below. Must not be combined with `entrypoints`, `cert`, or `auth.cookie-domain` on the same server (it inherits all three from its owner).
`halcyon.server.auth.enabled=true`	off
`halcyon.server.auth.allow=did:plc:...,did:web:...`	—	Empty with `enabled=true` means any authenticated DID.
`halcyon.server.auth.scopes=atproto account:email`	`atproto account:email`	`atproto` is required when auth is enabled.
`halcyon.server.auth.lifetime=86400`	`--cookie-lifetime`	Whole seconds.
`halcyon.server.auth.cookie-domain=foo.com`	host-only (no `Domain` attribute)	Server-only. Must equal or be a parent of every `halcyon.server.domain` on this server, and must not itself be a public suffix.
`halcyon.server.buffering=on\|off`	`off` (stream)
`halcyon.server.buffering.threshold=1MiB`	`--body-threshold`
`halcyon.server.body-max=1GiB`	`--body-max`	Request-body size limit, enforced in both streaming and buffered mode; setting it does not turn buffering on. Always bounded by `--body-max`; a higher resolved value is silently clamped, not rejected.
`halcyon.server.hook.quota=http://127.0.0.1:8081/v1/api/quota-check`	—
`halcyon.server.hook.quota.timeout=2s`	`2s`
`halcyon.server.hook.quota.priority=50`	`50`
`halcyon.server.hook.ratelimit=http://127.0.0.1:8082/ratelimit`	—
`halcyon.server.hook.ratelimit.timeout=1s`	`2s`
`halcyon.server.hook.ratelimit.priority=100`	`100`
`halcyon.server.inactivity-timeout=5m`	`--inactivity-timeout` (unlimited)	An explicit `0` means unlimited and overrides a non-zero `--inactivity-timeout` - the opt-out for a backend that legitimately goes silent for minutes, such as a PDS repository import.
`halcyon.server.backend.failure-window=10s`	`--backend-failure-window`	Server-only: locations share their server's balancer.
`halcyon.server.backend.max-failures=3`	`--backend-max-failures`	Server-only.
`halcyon.server.backend.cooldown=30s`	`--backend-cooldown`	Server-only.
`halcyon.server.backend.response-header-timeout=30s`	`--response-header-timeout`	Server-only.
`halcyon.server.backend.hash=ip`	`round-robin` (no flag)	Server-only. `round-robin` (default) or `ip` — rendezvous hashing on the client IP; see "Backend selection" below.
`halcyon.location.<name>.path=/xrpc/com.atproto.repo.uploadBlob`	—	Exact match.
`halcyon.location.<name>.path-prefix=/xrpc/`	—	Prefix match. Exactly one of `path`/`path-prefix` is required.
`halcyon.location.<name>.methods=POST,PUT`	all methods	Optional filter. Mutually exclusive with `exclude` — a methods filter on an exclude location would only partially block it (a request whose method is not listed falls through to the next-shorter matching location, or the server's fallback, and is proxied), so the combination is rejected at validation.
`halcyon.location.<name>.return=403`	—	Terminal kind: static response instead of proxying (`0` or 200-599, not 3xx).
`halcyon.location.<name>.exclude`	off	Terminal kind: bare boolean. Not routed at all - 404 as if absent.
`halcyon.location.<name>.case-insensitive`	off	Bare boolean. Match this location's `path`/`path-prefix` ASCII case-insensitively against the (normalised) request path. Matching only: the request is forwarded exactly as received. See "Location matching precedence" below.
`halcyon.location.<name>.redirect=https://elsewhere/`	—	Terminal kind: 302 to this exact URL, nothing appended.
`halcyon.location.<name>.hook.<kind>=off`	—	The only way a location can disable a hook its server set.
`halcyon.location.<name>.auth.* / buffering.* / body-max / hook.* / inactivity-timeout`	the server's value	Overrides the server's value for this location.

Backend selection. backend.hash (D33) picks how a server's balancer chooses among its live backends for each request. The default, round-robin, ignores the client and cycles through backends in turn, same as every pre-D33 release. backend.hash=ip instead uses rendezvous (highest-random-weight) hashing on the client IP — the full address for IPv4, the /64 prefix for IPv6 (so privacy-extension address rotation within one prefix stays sticky to the same backend) — so requests from one client IP consistently reach the same backend as long as it stays up, with no cookie and no shared state: every Halcyon node computes the same hash from the same (client IP, backend address) pair and sees the same Nomad-derived backend set, so they agree independently. A backend marked down by passive health is skipped for its clients (falling through to the next-highest-hashing live backend) until it recovers, exactly as round-robin skips a down backend. Clients behind one NAT or forward proxy share a single client IP and therefore a single backend. When every backend is down at once, stickiness is suspended and traffic goes to the least-recently-failed backend instead, exactly as round-robin's own all-down fallback works. backend.hash is server-wide, like the rest of backend.* — locations share their server's balancer and cannot each pick their own strategy.

The four terminal kinds a location can resolve to are mutually exclusive: proxy (the default — forward to the server's backends), return=<code> (answer a fixed status with no body, 3xx forbidden — use redirect instead), redirect=<url> (302 to an exact URL, nothing appended — url is validated the same way as hook.<kind>.url: an absolute http/https URL with a host and no userinfo), and exclude (as if the location did not exist at all — 404). Booleans accept =true/=false, or a bare tag for true; the explicit form is strconv.ParseBool-compatible, so 1/0/t/T/f/F also work, not just true/false. buffering additionally accepts on/off as well as true/false (bare tag = on). Every value is validated on ingestion; a service with any invalid tag is rejected entirely and reported in the dashboard — none of its servers are applied.

Location matching precedence: an exact path= match always beats every path-prefix= match, and among path-prefix= matches the longest one wins. This means a proxied exact path= DOES punch through an excluded path-prefix= that would otherwise cover it — intended, not a bug: path=/metrics/debug/pprof proxies normally even under path-prefix=/metrics, exclude on the same server, since the exact match is chosen first regardless of what it excludes. For this reason exclude should always use path-prefix=, never path=: an exact-path exclude is trivially sidestepped by anything the router treats as a different path — most simply a trailing slash (path=/metrics, exclude does not exclude /metrics/), so it protects far less than it looks like it does. Path matching (both path= and path-prefix=) is case-sensitive by default — /METRICS is a different path from /metrics and is not covered by an exclude on the latter. A location with case-insensitive matches every ASCII case spelling of its path instead (non-ASCII bytes are compared exactly); use it on an exclude in front of a backend that itself routes case-insensitively. Ranking with it: a case-sensitive exact path=, then a case-insensitive exact path=, then path-prefix= longest first, with a case-sensitive prefix ahead of a case-insensitive one of the same length. A case-sensitive and a case-insensitive location on the same path are not a conflict (the case-sensitive one wins for its own spelling); two case-insensitive ones whose paths differ only in case are.

Reserved path prefix /_halcyon/ on every server serves the OAuth login/callback/logout/client-metadata handlers; on a Nomad service's halcyon.* tags, a location may not claim it (a path or path-prefix reaching into it is a validation error) — except path=/_halcyon exactly, with no trailing slash, which is deliberately allowed since the router only ever intercepts the /_halcyon/ prefix, not that bare path; path-prefix=/_halcyon is still rejected, since as a prefix it would claim the whole reserved subtree. This check is tag-schema-only: a --routes-file location is not validated against it, so a routes-file location pointed at /_halcyon/... is accepted but will never see traffic — the router dispatches that prefix itself first. Every domain must not be an IP literal unless cert=off (Let's Encrypt cannot issue a certificate for an IP).

Example, PDS with a quota hook on the upload path:

halcyon.enable=true
halcyon.server.domain=pds.example.org *.pds.example.org
halcyon.location.upload-blob.path=/xrpc/com.atproto.repo.uploadBlob
halcyon.location.upload-blob.hook.quota=http://127.0.0.1:8081/v1/api/quota-check

Example, suspended tenant narrowed to read-only backup endpoints:

halcyon.enable=true
halcyon.server.domain=suspended.example.org
halcyon.location.get-repo.path=/xrpc/com.atproto.sync.getRepo
halcyon.location.get-blob.path=/xrpc/com.atproto.sync.getBlob
halcyon.location.deny.path-prefix=/
halcyon.location.deny.return=403

Normally two services cannot declare the same domain at all — dynamic.Merge drops both as a conflict (see "Certificates" below and dev/SECURITY-REVIEW.md's finding 6). halcyon.server.locations-only (D32) is the one exception: a server marked locations-only answers ONLY for its own declared locations, composed into a single "owner" server's routing on the domains they share, instead of claiming those domains itself.

The motivating case: service A (a PDS) owns foo.com and *.foo.com and serves everything; service B (a sidecar with its own backend) must serve only /xrpc/com.atproto.repo.importRepo on those same domains, and nothing else.

# service A (the owner)
halcyon.server.domain=foo.com *.foo.com
halcyon.location.import.path=/xrpc/com.atproto.repo.importRepo
halcyon.location.import.exclude

# service B (locations-only)
halcyon.server.domain=foo.com *.foo.com
halcyon.server.locations-only=true
halcyon.location.import.path=/xrpc/com.atproto.repo.importRepo

Rules:

A locations-only server needs exactly one owner: a single non-locations-only server that declares literally every one of its domains — x.foo.com is not covered by an owner's *.foo.com; only an identical domain entry counts (spelling differences that canonicalise to the same domain, such as a trailing dot or a unicode/punycode pair, still count as identical). Only some of its domains owned, or the domains split across more than one owner, drops the locations-only server (never the owner) and is reported as a conflict in the dashboard. No owner for any of its domains at all (typically: the owner's service was deregistered) drops it silently - it disappears from the dashboard along with its owner rather than lingering as a disabled entry.
Several locations-only servers may attach to the same owner at once, as long as they don't collide with each other (below) — each is composed into the owner's routing independently, so one sidecar can own /xrpc/com.atproto.repo.importRepo and a second, unrelated sidecar can own /xrpc/com.atproto.sync.subscribeRepos on the same domain.
If the owner itself is dropped (a name conflict, a domain conflict with another full server, or the --admin-host reservation), every locations-only server attached to it is dropped too — the dashboard/JSON API still show the owner itself as enabled; a conflict naming a survivor never disables it.
A locations-only location colliding with one of the owner's locations (same exact-vs-prefix kind, same path — compared case-insensitively when either side is case-insensitive — overlapping methods) is only allowed when the owner's own location there is exclude and the locations-only location's methods list is a superset of the excluded one's (an owner exclude with no methods filter excludes every method, so only a locations-only location with no methods filter of its own can override it) and, if the owner's exclude is case-insensitive, the locations-only location is too — otherwise a method the owner meant to exclude would silently fall through to the owner's own fallback and be proxied. When it does qualify, the locations-only location overrides the owner's exclude for as long as the sidecar is present. Any collision that doesn't qualify drops the locations-only server, owner untouched. Two locations-only servers attached to the same owner colliding with each other, on a domain they both list, drops both of them, owner untouched.
A locations-only location may be path-prefix=/ (a catch-all) only as an override of an owner path-prefix=/ exclude with no methods filter, and must itself have no methods filter. Without such an owner exclude it is a conflict and the locations-only server is dropped, so a locations-only server can never turn itself into a full server. This is the restricted-mode shape: the owner proxies an allowlist of path=/longer path-prefix= locations and excludes /, and the sidecar's catch-all answers everything else, including a method miss on an owner location with a methods filter (it falls through to the next-shorter match, which is the sidecar's /). With the sidecar absent, the owner's exclude answers 404.
The owner's exclude on the shared location is optional and decides what happens while the sidecar is absent: exclude means 404 for that path; no exclude (an ordinary proxying or terminal location, or nothing at all) means the request falls through to the owner's own routing instead. Either way, an exact path= still beats a path-prefix= as usual, on both sides.
The owner may have domains the locations-only server does not list; the owner's own routing on those domains is completely unaffected.
A locations-only server always inherits its owner's entrypoints, certificate mode, and auth.cookie-domain — it must not set any of the three itself. It keeps its own backends, balancer tuning, hooks, and auth.allow/auth.scopes. The one exception is the auth session realm: since the session cookie is host-scoped and the reserved /_halcyon/auth/* paths are always dispatched under the domain owner's server name, a locations-only location with auth enabled authenticates under its OWNER's realm, using its own resolved auth.allow list.
It stays its own server everywhere else: its own balancer, its own dashboard row (linking to its owner; the owner's own page lists every server attached to it), its own status, its own GET /v1/servers/{name} entry (with locations_only and attached_to fields) — it just never drives certificate issuance or a fallback of its own, and is never reachable on a domain it does not itself list even if the owner is.

This deliberately relaxes the "a domain is claimed process-wide" rule for a locations-only server specifically (dev/SECURITY-REVIEW.md's 2026-09-21 entry) — bounded by requiring a single, literal owner, the owner always winning any collision (including a partial, method-mismatched override attempt), and a locations-only server never being reachable at all without a live, valid owner.

Headers sent to backends #

On top of the client's own headers (minus hop-by-hop headers, which net/http handles), the proxy sets five headers on every request it forwards, each from its own knowledge of the connection rather than from anything the client sent:

Header	Value
`X-Real-Ip`	The client address alone, with no chain. Usually what a backend actually wants.
`X-Forwarded-For`	The trusted inbound chain with the client address appended, or just the client address when there is no trusted chain — which, behind an L4 balancer, is always the case (see the entrypoint section).
`X-Forwarded-Proto`	`https` when the entrypoint terminated TLS, else `http`.
`X-Forwarded-Host`	The inbound `Host`, regardless of `passhost`.
`X-Forwarded-Port`	The port the connection actually arrived on.

"Client address" throughout means the PROXY-corrected RemoteAddr — the real client behind an L4 load balancer, not the balancer.

A client cannot spoof any of them. httputil.ReverseProxy strips only X-Forwarded-For, -Host and -Proto from the outbound request before Halcyon's rewrite runs, so X-Real-Ip and X-Forwarded-Port are deleted explicitly first. When Halcyon has no value to report — a RemoteAddr that will not parse, or no local address on the connection — the header is omitted rather than sent empty or left holding the client's claim. Backends should treat absence as "unknown" and must not fall back to a client-supplied value.

Hook contract #

A hook is a pre-flight GET Halcyon sends before contacting the backend, for each resolved hook.<kind> on the matched location. Hooks run in a fixed order — priority descending, then kind name ascending — and stop at the first one that rejects; only if every hook allows does the request reach the backend. Hooks run after auth and before body policy, so a rejected request never spills to disk. Each hook kind has its own dedicated http.Client (redirects disabled) and its own timeout (default 2s), applied via a per-call context deadline; a timeout, connection failure, or any other transport error maps to 502 Bad Gateway with no upstream detail leaked.

Request: always GET to the configured URL verbatim (no path or query appended), with no body — the client's own request body is never sent to a hook. Headers: every client request header except hop-by-hop headers (Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, Te, Trailer, Transfer-Encoding, Upgrade) and Cookie/Authorization, which are always stripped; then Halcyon's own eight headers are set last, after first deleting any client-supplied value of the same name (so a client cannot spoof them by sending its own copy): X-Forwarded-Method, X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Uri, X-Forwarded-For (the PROXY-corrected client IP), X-Halcyon-Server, X-Halcyon-Location (- if the request matched no location — a server's own hooks still run for it), and X-Halcyon-Content-Length (used instead of Content-Length because net/http drops that header from a bodyless outgoing request). Both of the last two are conditional: when Halcyon has no value to report — an unknown request length (Content-Length chunked/absent) for X-Halcyon-Content-Length, or a RemoteAddr that fails to parse for X-Forwarded-For — the header is omitted entirely, not sent empty or passed through with whatever the client claimed. A quota hook keying off X-Halcyon-Content-Length must handle its absence explicitly.

Response: any 2xx allows the request through and is interpreted per hook kind (below). Any other status is passed to the client verbatim — status, headers minus hop-by-hop, and body — except a 3xx is mapped to a bare 502 (never leaking the hook's own redirect target), and Set-Cookie is always stripped (a hook must never set or clear a cookie on a client it never served).

quota: reads response header X-Halcyon-Allowance (bytes). Genuinely absent means unlimited. Present, it must be well-formed — parse as one non-negative integer, appear at most once, and not exceed 1<<60 — or the request is rejected 502: present-but-empty, malformed, negative, overflowing, or repeated all fail closed, deliberately, so a sidecar bug can never silently disable enforcement. 0 means zero bytes, never unlimited. The request body is wrapped in a counting reader; exceeding the allowance mid-stream aborts the upstream request, answering the client 413 if nothing has been written yet, or simply truncating the connection if the response has already started. Basically, if you are enforcing a quota on something, you'll want to return the number of bytes remaining in the quota.
ratelimit: the sidecar answers 429 (limited, passed through verbatim via the generic non-2xx path) or 200 (allowed), in both cases with RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset (and optionally Retry-After) per draft-polli-ratelimit-headers-02. On 200, Halcyon copies those headers onto the eventual client response, replacing rather than appending to any value the backend later sets for the same name — a backend header that arrives after the sidecar's is dropped, so the sidecar is always the value the client sees. This collapsing runs whenever any hook is configured on the route, not only ratelimit — a quota-only route still has a backend's own repeated RateLimit-*/Retry-After values truncated to one. Halcyon never computes a limit itself, only relays the sidecar's verdict.

Auth flow #

Halcyon has two distinct AT Protocol OAuth client shapes, never shared:

Backend auth (halcyon.server.auth.* on a tenant server): a public OAuth client. Each server publishes its own client metadata at https://<first domain>/_halcyon/auth/client-metadata.json (that URL is the client_id), DPoP-bound, no client secret. After the callback, indigo's token session is discarded entirely — never persisted to raft — deliberately: the store backing backend logins silently discards every write (SaveSession/DeleteSession both succeed and do nothing) and GetSession always errors reporting "not persisted," so no earlier write is ever readable back even though the write itself never fails. Only the session's DID and a best-effort email (fetched once, immediately after the callback, before the tokens are thrown away) are kept, in a backend_sessions raft record.
Admin dashboard (--admin-host): the single confidential OAuth client for the whole cluster, using private_key_jwt with a raft-replicated assertion key. Its tokens are kept — a real, refreshable indigo session persisted in the admin_sessions raft keyspace — because the DNS-01 writer resumes that same session (internal/pdsdns's provider fetches the newest valid admin session and calls ResumeSession on it) to authenticate its TXT-record writes against the PDS. A refresh safety net, checked once a day, proactively refreshes that session's refresh token once it has gone 150 days unused — a 30-day margin under the spec's 180-day cap on an individual refresh token — so it never expires just because DNS-01 traffic has gone quiet. Without a completed admin login there is no PDS session, and issuance for PDS-owned domains backs off until the operator logs in again.

Identity headers: on an auth-enabled location, an authenticated request reaches the backend with X-Halcyon-Auth-Did always and X-Halcyon-Auth-Email when the session recorded one (best-effort, at login). Every inbound X-Halcyon-Auth-* header is stripped first on every location, gated or not, so a backend can trust these values; the session cookie itself is stripped too, so the backend learns nothing beyond them. Other headers, including an application's own X-Auth-*, are passed through unchanged.

A backend session never grants admin access. The two live in separate raft keyspaces with separate cookie names (halcyon_session for backend, halcyon_admin for admin) and a realm bound inside each signed envelope (the serving server's own name for backend, the literal string admin for the dashboard); a lookup checks the realm twice — once against the envelope before any raft read, once against the stored record after — so a cookie from one store is rejected by the other even before the keyspace separation itself would already prevent the session ID from resolving at all. The admin gate additionally re-validates the session's DID against --admin-did on every request, not just at login.

Cookie envelope: base64url(JSON{sid, realm, exp}) + "." + base64url(HMAC-SHA256(...)), keyed by --cookie-secret (at least 32 bytes). No token material or DID travels in the cookie itself — only a session ID, realm, and expiry; the HMAC is checked in constant time before any JSON is even parsed, so a forged or tampered cookie never reaches a raft lookup. Max-Age is always now + auth.lifetime (default --cookie-lifetime). The raft record's own expiry starts the same way, but a re-login never shortens it: if the session already has a later stored expiry (the admin realm's session can carry a longer-lived expiry than a single cookie's lifetime, kept by the DNS-01 writer), the later of the two wins, so logging in again cannot accidentally cut a longer-lived record short. Session cookies are host-only by default: no Domain attribute at all, so per RFC 6265 §5.3.6 the cookie is scoped to exactly the host that set it and is never sent to, nor accepted from, a sibling host — even a same-registrable-domain one. The OAuth callback lands on server.Domains[0] (the first domain, since that is the host baked into the public client's redirect_uri — see internal/auth/oauth.go) and return_to is reduced to a path before the redirect back to the app, so a server with several domains that wants one session shared across all of them needs halcyon.server.auth.cookie-domain set to their common parent explicitly — the tag schema section above covers its validation. The admin dashboard never sets this (it is only ever served on --admin-host, so host-only is exactly right) — this also narrows what used to be a global --cookie-domain, which stamped the same wide Domain onto the admin cookie as every tenant cookie, and so rode along to every fronted host under it; host-only closes that by construction. CSRF validation on the dashboard's mutating routes stays in place regardless, as defence in depth — SameSite=Lax alone does not stop a same-site cross-origin POST.

Certificates #

DNS-01 only — no HTTP-01, no TLS-ALPN. --bootstrap-domain, if set, is issued (with its *. wildcard) immediately on every leadership acquisition, ahead of and independent of the normal renewal sweep, and skips the delegation pre-check below (it is assumed operator-owned infrastructure). Every other domain comes from server.domain tags: foo.com *.foo.com packs into one order for foo.com with SAN *.foo.com; a name already covered by the bootstrap domain's wildcard is skipped.

Staging vs production is not a flag — it is a raft-replicated cluster setting (ACMESettings.Staging, default true), changed through the admin dashboard or halcyon raft config set staging=false. Changing it on a running node does more than go unnoticed until restart: the node detects the change, logs an error naming both the old and new ACME directory, and stops all discovery-driven issuance on itself entirely until it is restarted — it will not pick up the new directory and resume on its own. A certificate whose actual issuer doesn't match the current setting is reissued automatically, but only once a node has restarted onto the new setting.

Delegation pre-check: before queueing an order for a domain outside a zone the admin DID owns directly in the PDS, Halcyon resolves _acme-challenge.<domain> and confirms it CNAMEs to <domain>.verify.<--verify-zone>.. A missing or mismatched CNAME marks the domain awaiting_delegation (shown in the dashboard with the exact CNAME target expected) and is re-checked on every sweep rather than burning an ACME order and a backoff slot. Owned-zone domains skip this check entirely.

Leader-only issuance: every process runs its own issuance manager, but every code path that would actually contact the ACME CA is gated behind the raft leadership check (cluster.IsLeader(), plus a 30s post-election cooldown) — a follower's manager runs the same loop but never dispatches a real order.

--bootstrap-certificate breaks the admin dashboard's own chicken-and-egg deadlock. The admin dashboard is served through the data plane over HTTPS, using a certificate ACME issues via DNS-01 — but DNS-01 needs an admin PDS session, which only exists once an operator has logged into the dashboard, over HTTPS, using the certificate that cannot yet be issued. There is nothing in front of Halcyon to hold an interim certificate the way Traefik once did for cert-fetcher. If you hit this — the admin host has no valid certificate and there is no other way to reach it — hand-issue (or otherwise obtain, from any CA) a certificate for --admin-host, point Halcyon at it with --bootstrap-certificate (and --bootstrap-certificate-key if the key is a separate file), restart or wait for the next leadership check, log in once, and everything else issues normally from then on. The flag is safe to leave set permanently: once ACME has issued a real certificate covering the same name(s) with an equal or later expiry, the bootstrap file is inert and is never imported again. Matching is by the certificate's DNS SANs, not its CN (Let's Encrypt does not reliably populate CN, and Halcyon's own cert store indexes by SAN throughout). Import happens once this node is leader — followers simply wait for it to replicate, the same as any other raft write — and a bad or unparsable bootstrap file is logged loudly but never stops the node from serving; it just means the deadlock is not broken yet.

Supply the full chain, leaf first, then any intermediates — with acme.sh that is fullchain.cer, not <domain>.cer. Everything in the file is kept and served, so a leaf-only file works for clients that happen to have the issuer's intermediate cached and fails for those that do not. The leaf must come first: its SANs and expiry are what the import and the skip-if-covered check are read from.

cat fullchain.cer admin.example.net.key > /opt/halcyon/bootstrap.pem
chmod 600 /opt/halcyon/bootstrap.pem

Within the combined file the private key may sit before or after the chain; only the leaf-before-intermediates order matters.

Building and publishing the image #

You'll have to have a gander at Makefile and Dockerfile - I've got it set to go to my github repo as a private image (because again, using this in production if you're not me is probably not the best idea).

Operations #

Admin listener (:6072) is the machine surface: raft API, ACME API, /metrics, and node-to-node auth forwarding. Every route on it requires Authorization: Bearer <--admin-psk>, compared in constant time, except /healthz, which stays open for liveness probes — this is an exact-match path exemption, not a prefix, so nothing under /healthz/... is accidentally exempted too. The exemption is on the path alone; it does not look at the HTTP method, so any method on that one exact path skips the PSK gate — an unsupported method there gets a plain 405 from the handler itself, not a 401 from the PSK check. /metrics and /v1/node are gated like everything else — /v1/node used to be exempt too (liveness/peer polling), but it reports version and full raft topology to anyone reaching :6072, so peers now send the PSK for it exactly as they do for every other admin route.

The admin dashboard (--admin-host) is a completely separate surface, gated by the admin OAuth session cookie plus the per-request --admin-did re-check, never the PSK — a browser cannot send a bearer token on plain navigation.

Raft membership (halcyon raft add-voter / remove-server, PUT/DELETE /v1/raft/voters[/{id}]) is the cluster's join path: BootstrapCluster is one-shot, so a node whose identity changed, or a genuinely new node, can only enter an existing cluster's raft configuration this way. Both are leader-only, PSK-gated like every other /v1 route, and automatically retried against the current leader by the CLI (same leaderRetry behaviour as raft snapshot restore/raft config set) if aimed at a follower. add-voter refuses (409) an ID already registered at a different address rather than silently repointing it — remove the old registration first (remove-server) if the intent is genuinely to relocate a member. Removing the current leader's own ID is allowed; raft steps it down and elects a new leader among the rest. Neither route touches a node's own --raft-peer/nomad: startup configuration — a node added this way starts (or restarts) with no local peer list of its own and simply waits to be added, the same way a freshly bootstrapped cluster's nodes never call this path at all.

A nomad:<service>-discovered node refuses to bootstrap a new cluster if any other node in its freshly-resolved peer set already reports a leader via its own /v1/node (sending the shared --admin-psk, now that the route requires it) — the fix for a real split-brain window phase 9's design review left open (two nodes settling on two different, individually valid --bootstrap-expect-sized peer sets after a rescheduled alloc). The failure names the existing leader and points at halcyon raft add-voter instead. An unreachable/timed-out/404 peer is never treated as evidence either way — only a peer that positively answers with a known leader blocks the bootstrap; every node in a genuinely fresh bootstrap is silent on this check by construction (no admin listener is up yet on any of them at this point in their own startup), so this cannot false-positive on a legitimate first boot.

Metrics are served in Prometheus text format at GET /metrics on the admin listener. Every Halcyon-specific series is prefixed halcyon_; the registry also carries the Go runtime/process collectors, a halcyon_build_info gauge, and raft's own internal metrics bridged in via armon/go-metrics — that bridge installs a single process-wide sink guarded by a sync.Once, so only the first Registry created in a process actually gets raft's metrics; a later one still works normally, just without the raft series. Notable series: per-server proxy metrics (halcyon_service_requests_total, halcyon_service_request_duration_seconds, halcyon_service_in_flight, halcyon_service_backend_up), hook outcomes (halcyon_hook_calls_total, halcyon_quota_aborts_total, halcyon_ratelimit_rejections_total), certificate state (halcyon_tls_cert_not_after_seconds, halcyon_acme_domains_awaiting_delegation), and config-reload health (halcyon_config_reloads_total, halcyon_config_last_reload_success).

You'll have to use the --admin-psk pre-shared key as a Bearer token authentication in Prometheus. How to do this? You guessed it, left up to the reader.

Clone this repository

Halcyon #