diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..f3e1cb0 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,72 @@ +# CLAUDE.md + +`entangle` inventories the websites in a GitHub account and plans their migration +to Tangled's static site hosting. Repo content migration is not in scope — a +separate autosync service mirrors GitHub → Tangled one-way. + +Read `README.md` first; it carries the design and the verified facts about +Tangled. This file is the working context that is easy to get wrong. + +## Current state + +Paused. `scan`, `report`, and `plan` work and have been run against the `autonome` +account. `migrate --apply` has never executed against a live endpoint because +there isn't one yet — see Blockers in `README.md`. Do not treat the apply path as +tested. + +## Ground rules for this repo + +- **No dependencies.** Node ESM, `.mjs`, built-in modules only. `fetch`, `node:dns/promises`, + `node:child_process` cover everything needed. +- **GitHub auth comes from the `gh` CLI**, via `github.mjs ghToken()`. Never prompt + for or store a PAT. +- **`data/` is gitignored.** Scan output names private repos and enumerates their + file trees. Do not commit it, do not paste it into docs or commit messages. +- **404 is a normal answer.** `github.mjs api()` returns `{ok, status, body}` and + callers branch on status. A missing Pages config or missing file is data, not an + exception. +- **Anything touching GitHub content propagates.** Autosync is one-way GitHub → + Tangled and unattended, so a commit to a Pages branch becomes a live Tangled site. + Treat writes to GitHub as writes to production on both hosts. + +## Facts that cost effort to establish + +Re-verify before relying on them; all of these are moving targets. + +- The site endpoints live in the `org.tangled.temp.*` namespace and are **not + reachable on the flagship appview** — `XRPC_ENABLED` defaults to off in + `appview/config` `CoreConfig`, and `https://tangled.org/xrpc/_health` returns the + appview's HTML 404. `tangled.mjs xrpcAvailable()` checks for a JSON content-type, + not just a 2xx, because of exactly this. +- Service-auth audience is `did:web:tangled.org`, derived from `APPVIEW_HOST` via + `serviceauth.DidWeb()`. The JWT must be scoped to the specific method (`lxm`). +- `repoDid` is minted by the knot and stored on the `sh.tangled.repo` record in the + owner's PDS. It is not the account DID and cannot be derived from the repo name. + Older records lack it entirely. +- `TANGLED_SITES_DOMAIN` defaults to `tngl.io`. `tngl.sh` is the PDS user domain — + two different `tngl.*` hosts, easy to conflate. Accounts on Tangled's PDS get a + handle-bound `.tngl.sh` sites domain that cannot be released. +- `appview/sites` `Deploy()` fetches a `tar.gz` of the branch from the knot and + diff-syncs it to R2. No size or file-count limit anywhere in that path. +- Tangled has no custom domain support, no build step, and only one index repo per + domain. Everything else is served at `/`. + +Verified against live GitHub behaviour, and the basis of the redirect design: + +- A path under `.github.io` that no Pages-enabled repo claims is served by the + **user site's** `404.html`, path intact. +- A path under a repo that *does* have Pages enabled falls through to GitHub's + generic 404 instead, so the user-site `404.html` never sees it. + +## Classification, in one line + +Tangled serves files verbatim, so the question is whether an `index.html` exists +where Tangled would serve from. `detect.mjs` `verdict()` owns this. The subtle case +is implicit Jekyll: GitHub renders a bare `README.md` into a homepage unless +`.nojekyll` is present, which makes sites look alive while having no `index.html` +anywhere. Those are `needs-build`, not `automatic`. + +Directory guessing is deliberately narrow. `detect.mjs` `PUBLISH_DIRS` gates which +top-level directories may be promoted to a deploy dir, because an unfiltered search +for `index.html` promotes example and fixture pages into "the homepage" — worse than +reporting that none was found. diff --git a/README.md b/README.md index 297e7c4..595f0d0 100644 --- a/README.md +++ b/README.md @@ -2,10 +2,13 @@ Find the websites hidden in a GitHub account and move them to Tangled. -Repo *content* migration is handled separately by repo sync. This tool only deals -with the hosting layer: which repos serve a website, from what branch and -directory, on what domain, and how much of that can be recreated on Tangled -without a human in the loop. +Repo *content* migration is handled separately by an autosync service that mirrors +GitHub → Tangled one-way. This tool only deals with the hosting layer: which repos +serve a website, from what branch and directory, on what domain, and how much of +that can be recreated on Tangled without a human. + +**Status: on hold.** The inventory and planning halves work. The apply half cannot +run yet — see [Blockers](#blockers). ``` node bin/entangle.mjs scan # → data/.json @@ -16,8 +19,8 @@ node bin/entangle.mjs migrate --apply # write site configs on Tangled ``` GitHub auth comes from the `gh` CLI (`gh auth token`). Tangled auth comes from -`TANGLED_HANDLE` and `TANGLED_APP_PASSWORD` (an atproto app password, not your -account password). +`TANGLED_HANDLE` and `TANGLED_APP_PASSWORD` (an atproto app password, not an +account password). No dependencies; Node 20+. ## What the scan looks at @@ -53,20 +56,21 @@ they would serve nothing. ## Constraints on the Tangled side -Taken from `tangled.org/core` and the hosting docs, current as of this writing: +Taken from `tangled.org/core` and the hosting docs: - one claimed domain per account — `.tngl.sh` for accounts on Tangled's - PDS, or a claimable `.tngl.io` + PDS, or a claimable subdomain under `TANGLED_SITES_DOMAIN` (default `tngl.io`) - exactly one repo may be the **index**, served at the domain root; everything else is served at `/` - **no custom domains yet**, stated as planned - no build step, no documented SPA or 404 handling +- no size or file-count limit in `appview/sites` `Deploy()` - redeploys on every push to the configured branch -The last two combine into the main migration hazard: a site that used to sit at a -domain root and links to `/style.css` breaks when it moves under a sub-path. -`entangle plan` fetches each `index.html` and flags this as an `absolute-paths` -risk rather than letting you find out after the cutover. +The last constraints combine into the main migration hazard: a site that used to +sit at a domain root and links to `/style.css` breaks when it moves under a +sub-path. `entangle plan` fetches each `index.html` and flags this as an +`absolute-paths` risk rather than leaving it to be discovered after cutover. ## The Tangled API this drives @@ -84,22 +88,11 @@ XRPC procedures in the **`org.tangled.temp.*`** namespace: `temp` means unstable by declaration. Expect these to move. -**These endpoints are not reachable on the flagship instance today.** The appview -only mounts `/xrpc` when `XRPC_ENABLED` is set, it defaults to off, and -`https://tangled.org/xrpc/_health` currently returns the appview's HTML 404 page. -`entangle migrate` checks this first and says so rather than failing obscurely. - -Until that flips, the only way to set site config is the web UI, which issues -`PUT /{owner}/{repo}/settings/sites` with form fields `branch`, `dir`, and -`is_index`, authenticated by a session cookie from atproto OAuth login. Scripting -that means driving the OAuth flow, which is a much bigger commitment than an app -password — so this tool stops at producing the plan, and the plan doubles as the -manual checklist. - Authentication is standard atproto inter-service auth: resolve handle → DID → PDS, create a session with an app password, ask the PDS for a service-auth JWT -scoped to the appview's DID and the exact method being called, send it as a -bearer token. `src/tangled.mjs` does all of this. +scoped to the appview's DID (`did:web:tangled.org`, from `APPVIEW_HOST`) and the +exact method being called, then send it as a bearer token. `src/tangled.mjs` does +all of this. The one identifier that matters is `repoDid`, the DID a knot mints for a repo. It is not the account DID and it is not derivable from the repo name — it lives in @@ -107,6 +100,57 @@ the `sh.tangled.repo` record in the owner's PDS, which is where `migrate` reads from. Repo records created before repo DIDs existed do not have one and cannot host a site until they do. +## Blockers + +Work is paused on two things outside this repo. + +**The site endpoints are switched off.** The appview only mounts `/xrpc` when +`XRPC_ENABLED` is set, it defaults to off, and `https://tangled.org/xrpc/_health` +returns the appview's HTML 404 page. `migrate` preflights this and reports it +rather than failing obscurely. + +The only other way to set site config is the web UI, which issues +`PUT /{owner}/{repo}/settings/sites` with form fields `branch`, `dir`, and +`is_index`, authenticated by a session cookie from atproto OAuth login. Scripting +that means driving the OAuth flow — a much bigger commitment than an app +password. Until XRPC is reachable, the `plan` output doubles as a manual +checklist. + +**The autosync service needs fixes** before any GitHub-side change should be +committed, because everything written to GitHub propagates to Tangled +automatically. + +## Redirecting old GitHub URLs + +Not built yet; the design is settled and verified against live behaviour. + +GitHub Pages cannot issue a 301, so any redirect is client-side. The routing +detail that makes one file sufficient: a request to a path under +`.github.io` that no Pages-enabled repo claims is served by the **user +site's** `404.html`, with the original path intact. A request under a repo that +*does* have Pages enabled falls through to GitHub's generic 404 instead. + +So the shape is one `404.html` on the `.github.io` repo, plus Pages turned +off on each project repo whose site has moved. + +Because sync is one-way GitHub → Tangled, that file lands in the repo Tangled +serves and must neutralize itself on arrival: + +```js +if (location.hostname.endsWith('github.io')) { + location.replace('https://.tngl.sh' + location.pathname + location.search + location.hash) +} +``` + +Without the hostname guard, a synced `404.html` at the Tangled index would bounce +`.tngl.sh/missing` back to itself. The same reasoning rules out committing a +shim to a repo's actual Pages branch: it would sync straight to Tangled and +become the site. + +Known cost: fallthrough paths return HTTP 404 with a redirecting body, so browsers +follow it but crawlers treat the URL as gone. A `` pointing +at the Tangled URL is the best available mitigation. + ## Layout ``` @@ -119,3 +163,6 @@ src/plan.mjs map sites onto a Tangled domain, flag sub-path breakage src/tangled.mjs atproto identity, service auth, the site XRPC calls src/migrate.mjs apply a plan, conservatively ``` + +`data/` is gitignored: scan output names private repos and describes their +contents.