# entangle Move a GitHub account's repos and websites to Tangled. This repo holds the docs, the code, and the infrastructure that runs the move, all of which run on a single self-hosted machine. Three parts: - **Inventory and planning** — the `entangle` CLI below. Which repos serve a website, from what branch and directory, on what domain, and how much of that can be recreated on Tangled without a human. - **Content sync, GitHub → Tangled.** For most repos a separate autosync service mirrors GitHub to Tangled one-way, unattended. - **Custom-domain sites, Tangled → GitHub.** Tangled cannot serve custom domains, so for every site on one, Tangled becomes the source of truth and GitHub becomes a push mirror that keeps GitHub Pages serving the domain. Autosync is off for those repos. See [Mirroring custom-domain sites to GitHub](#mirroring-custom-domain-sites-to-github); the workflow runs on the self-hosted spindle in [`infra/spindle/`](infra/spindle/README.md). **Status:** the custom-domain mirror is live. The inventory and planning halves work. The apply half (Tangled site config) cannot run yet — see [Blockers](#blockers). ``` node bin/entangle.mjs scan # → data/.json node bin/entangle.mjs report # readable inventory node bin/entangle.mjs plan # → data/.plan.json node bin/entangle.mjs migrate # dry run node bin/entangle.mjs migrate --apply # write site configs on Tangled ``` GitHub auth comes from the `gh` CLI (`gh auth token`). Tangled auth comes from `TANGLED_HANDLE` and `TANGLED_APP_PASSWORD` (an atproto app password, not an account password). No dependencies; Node 20+. ## What the scan looks at GitHub reports Pages state in three places that routinely disagree, so the scan collects all of them: - the repo's `has_pages` flag, which stays true for years after a site is removed - `GET /repos/{o}/{r}/pages`, the authoritative branch/path/CNAME config - live DNS and an actual HTTP request, the only evidence anyone can reach the site On top of that it reads the repo tree to work out whether servable files exist — an `index.html` at the Pages source path, committed build output, or nothing at all — and looks for deploy config belonging to other hosts (Netlify, Vercel, Cloudflare Pages, Firebase, Render) so sites that left GitHub Pages still show up. ## Why sites get classified the way they do Tangled serves a branch and directory **verbatim**. There is no build step. That single fact drives most of the verdicts: | Verdict | Meaning | | --- | --- | | `automatic` | an `index.html` already sits where Tangled would serve from, and nothing builds it first | | `domain-decision` | deployable, but it currently lives on a custom domain | | `needs-build` | a generator, an Actions workflow, or GitHub's implicit Jekyll build produces the homepage, and the output never lands in the repo | | `elsewhere` | no Pages config, but deploy config for another host | | `unknown` | the file tree could not be read, or was truncated with nothing servable seen — rescan | | `dead` | nothing servable and nothing answering — it stopped being a site a while ago | The `needs-build` case that catches people out is **implicit Jekyll**: GitHub renders a bare `README.md` into the homepage unless a `.nojekyll` file says otherwise. Those sites look alive and have no `index.html` anywhere. On Tangled they would serve nothing. ## Constraints on the Tangled side Taken from `tangled.org/core` and the hosting docs: - one claimed domain per account — `.tngl.sh` for accounts on Tangled's PDS, or a claimable subdomain under `TANGLED_SITES_DOMAIN` (default `tngl.io`) - exactly one repo may be the **index**, served at the domain root; everything else is served at `/` - **no custom domains yet**, stated as planned - no build step, no documented SPA or 404 handling - no size or file-count limit in `appview/sites` `Deploy()` - redeploys on every push to the configured branch The last constraints combine into the main migration hazard: a site that used to sit at a domain root and links to `/style.css` breaks when it moves under a sub-path. `entangle plan` fetches each `index.html` and flags this as an `absolute-paths` risk rather than leaving it to be discovered after cutover. ## The Tangled API this drives Site config is not an atproto record — it lives in the appview's database behind XRPC procedures in the **`org.tangled.temp.*`** namespace: | Method | Purpose | | --- | --- | | `org.tangled.temp.site.getDomainClaim` | current claimed domain | | `org.tangled.temp.site.claimDomain` | claim `` | | `org.tangled.temp.site.releaseDomain` | release it (not permitted for handle-bound `tngl.sh` domains) | | `org.tangled.temp.repo.getSiteConfig` | read a repo's site config | | `org.tangled.temp.repo.updateSiteConfig` | set branch, dir, isIndex — and deploy | | `org.tangled.temp.repo.disableSite` | remove the site and its deployed files | `temp` means unstable by declaration. Expect these to move. Authentication is standard atproto inter-service auth: resolve handle → DID → PDS, create a session with an app password, ask the PDS for a service-auth JWT scoped to the appview's DID (`did:web:tangled.org`, from `APPVIEW_HOST`) and the exact method being called, then send it as a bearer token. `src/tangled.mjs` does all of this. The one identifier that matters is `repoDid`, the DID a knot mints for a repo. It is not the account DID and it is not derivable from the repo name — it lives in the `sh.tangled.repo` record in the owner's PDS, which is where `migrate` reads it from. Repo records created before repo DIDs existed do not have one and cannot host a site until they do. ## Blockers Work is paused on two things outside this repo. **The site endpoints are switched off.** The appview only mounts `/xrpc` when `XRPC_ENABLED` is set, it defaults to off, and `https://tangled.org/xrpc/_health` returns the appview's HTML 404 page. `migrate` preflights this and reports it rather than failing obscurely. The only other way to set site config is the web UI, which issues `PUT /{owner}/{repo}/settings/sites` with form fields `branch`, `dir`, and `is_index`, authenticated by a session cookie from atproto OAuth login. Scripting that means driving the OAuth flow — a much bigger commitment than an app password. Until XRPC is reachable, the `plan` output doubles as a manual checklist. **The autosync service needs fixes** before any GitHub-side change should be committed, because everything written to GitHub propagates to Tangled automatically. ## Redirecting old GitHub URLs Not built yet; the design is settled and verified against live behaviour. GitHub Pages cannot issue a 301, so any redirect is client-side. The routing detail that makes one file sufficient: a request to a path under `.github.io` that no Pages-enabled repo claims is served by the **user site's** `404.html`, with the original path intact. A request under a repo that *does* have Pages enabled falls through to GitHub's generic 404 instead. So the shape is one `404.html` on the `.github.io` repo, plus Pages turned off on each project repo whose site has moved. Because sync is one-way GitHub → Tangled, that file lands in the repo Tangled serves and must neutralize itself on arrival: ```js if (location.hostname.endsWith('github.io')) { location.replace('https://.tngl.sh' + location.pathname + location.search + location.hash) } ``` Without the hostname guard, a synced `404.html` at the Tangled index would bounce `.tngl.sh/missing` back to itself. The same reasoning rules out committing a shim to a repo's actual Pages branch: it would sync straight to Tangled and become the site. Known cost: fallthrough paths return HTTP 404 with a redirecting body, so browsers follow it but crawlers treat the URL as gone. A `` pointing at the Tangled URL is the best available mitigation. ## Mirroring custom-domain sites to GitHub Tangled cannot serve a custom domain, so a site on one keeps GitHub Pages as its host while its source moves to Tangled. Each such repo carries `.tangled/workflows/mirror.yml`, copied from `infra/mirror/mirror.yml`, which runs on every push to Tangled: 1. The spindle's clone step fetches only the pushed commit at depth 1, so the first step fetches every branch and tag from the knot with full history. 2. The second step pushes all branches and tags to the GitHub repo over SSH, with a write-enabled deploy key and GitHub's host key pinned. Pushes are fast-forward only: a GitHub branch that has diverged fails the run instead of being overwritten. 3. GitHub Pages rebuilds from the pushed branch as before. The workflow file reaches GitHub too, where nothing reads it. Limits: - Deleting a branch or tag on Tangled does not reach GitHub; the knot starts no pipeline for a deletion. - The nixery image has no passwd entry for the step's user, and OpenSSH refuses to run without one, so the push step adds one before calling ssh. - A repo whose Pages branch is built rather than committed by hand moves its build to the spindle, and lists that branch in `MIRROR_EXCLUDE` so the mirror never pushes it. metafluff does this: its `.tangled/workflows/publish.yml` builds with eleventy on every push to `main` and pushes the output to GitHub's `gh-pages` as one `deploy: ` commit, and its GitHub Actions build is deleted. Tangled's own copy of the built branch is not updated, since pushing to Tangled from a run would need an SSH key with write access to every repo on the account. - A push from the mirror does not always start a Pages build on GitHub; for two repos whose run pushed two branches at once, none started. `gh api -X POST repos///pages/builds` requests one. Adding a repo: 1. `ssh-keygen -t ed25519 -N '' -f ~/.config/entangle/mirror-keys/` on the spindle host. The private half stays there, outside any repo. 2. `gh api repos///keys -f title=tangled-mirror -F read_only=false -F key=@.pub` 3. Copy `infra/mirror/mirror.yml` to `.tangled/workflows/mirror.yml`, set `GITHUB_REPO` (and `MIRROR_EXCLUDE` if a build owns a branch), commit, push to Tangled. Check which remote is Tangled first: an older checkout's `origin` may still be GitHub, and a push there skips Tangled entirely. 4. `entangle mirror-connect --spindle=` sets the repo's spindle by rewriting its `sh.tangled.repo` record, then stores the key as the secret `GITHUB_MIRROR_KEY_B64` through the spindle's `sh.tangled.repo.addSecret`. The web UI does the same: select the spindle in the repo's settings, and add the secret with the output of `base64 -w0 `. 5. Push any commit and confirm the GitHub branch head moves. Autosync must be off for the repo, or the two would push at each other. ## Layout ``` bin/entangle.mjs CLI src/github.mjs REST client, auth from gh, pagination, rate-limit backoff src/detect.mjs what a file tree says about how a site is produced src/scan.mjs per-repo scan: Pages config, tree, workflows, DNS, liveness src/report.mjs inventory as markdown src/plan.mjs map sites onto a Tangled domain, flag sub-path breakage src/tangled.mjs atproto identity, service auth, the site XRPC calls src/migrate.mjs apply a plan, conservatively test/ node --test (npm test); no network beyond the gh token infra/mirror/ Tangled → GitHub mirror workflow template infra/spindle/ the self-hosted spindle: units, config template, restore steps ``` `data/` is gitignored: scan output names private repos and describes their contents.