diff --git a/.beans/ATFS-0oko--devatfsfile-lexicon-record-references-to-atfs-host.md b/.beans/ATFS-0oko--devatfsfile-lexicon-record-references-to-atfs-host.md index b36e1bd..0ea6b97 100644 --- a/.beans/ATFS-0oko--devatfsfile-lexicon-record-references-to-atfs-host.md +++ b/.beans/ATFS-0oko--devatfsfile-lexicon-record-references-to-atfs-host.md @@ -89,18 +89,24 @@ only the rkey needs passing. **DNS.** Resolution is explicitly *non-hierarchical* — a TXT record is needed for every authority section, and resolvers never recurse. The -authority is the NSID minus its last segment, reversed into a domain. Three -records, all pointing at the same DID: +authority is the NSID minus its last segment, reversed into a domain. Two +records, both pointing at the same DID: | TXT name | covers | |---|---| | `_lexicon.atfs.dev` | dev.atfs.file, dev.atfs.server | -| `_lexicon.repo.atfs.dev` | dev.atfs.repo.{uploadFile,getFile,pinFile,deleteFile} | -| `_lexicon.sync.atfs.dev` | dev.atfs.sync.listFiles | - -Value: `did=`. Simplest choice is the owner DID already in -use, did:plc:ephkzpinhaqcabtkugtbzrwu — that is the repo the records live -in, and the two must agree. +| `_lexicon.repo.atfs.dev` | dev.atfs.repo.{uploadFile,getFile,pinFile,deleteFile,listFiles} | + +Value: `did=`. ATFS-jrzm's rename is what keeps this to two: +while listFiles lived in `dev.atfs.sync` it needed a third record at +`_lexicon.sync.atfs.dev`, since resolution never recurses. + +The DID must be the account whose repo holds the schema records — the two +have to agree. JP is standing up an **@atfs.dev** account for exactly this +(2026-08-09), so the authority is that account's DID rather than the owner +DID the bench instance runs under. That also keeps the lexicons independent +of any one operator's identity, which matters if atfs is ever run by +someone else. **Caveat on the `--no-validate` sharp edge.** Publishing makes the schemas resolvable; it only stops eurosky rejecting `dev.atfs.*` records if that PDS diff --git a/.beans/ATFS-1gpu--follow-another-atfs-instance.md b/.beans/ATFS-1gpu--follow-another-atfs-instance.md index 34c92b5..1ac0d25 100644 --- a/.beans/ATFS-1gpu--follow-another-atfs-instance.md +++ b/.beans/ATFS-1gpu--follow-another-atfs-instance.md @@ -12,4 +12,4 @@ An instance keeps a copy of another instance's data. Explicitly NOT to be implem ## Reasons for Scrapping -Superseded by ATFS-ofke, the richer post-pinFile write-up of the same feature — follow's design notes should have one home. The door-keeping this bean existed for is done: pins carry {did, source} precisely so follow claims slot in without a schema change, and dev.atfs.sync.listFiles (ATFS-3x09) now provides the enumeration surface a follower would poll. +Superseded by ATFS-ofke, the richer post-pinFile write-up of the same feature — follow's design notes should have one home. The door-keeping this bean existed for is done: pins carry {did, source} precisely so follow claims slot in without a schema change, and dev.atfs.repo.listFiles (ATFS-3x09, renamed out of dev.atfs.sync by ATFS-jrzm) now provides the enumeration surface a follower would poll. diff --git a/.beans/ATFS-3x09--devatfssynclistfiles-cursored-enumeration-for-pin.md b/.beans/ATFS-3x09--devatfssynclistfiles-cursored-enumeration-for-pin.md index c5ce421..7875987 100644 --- a/.beans/ATFS-3x09--devatfssynclistfiles-cursored-enumeration-for-pin.md +++ b/.beans/ATFS-3x09--devatfssynclistfiles-cursored-enumeration-for-pin.md @@ -1,11 +1,11 @@ --- # ATFS-3x09 -title: 'dev.atfs.sync.listFiles: cursored enumeration for pin replication' +title: 'dev.atfs.repo.listFiles: cursored enumeration for pin replication' status: completed type: feature priority: normal created_at: 2026-08-07T15:12:21Z -updated_at: 2026-08-07T15:42:46Z +updated_at: 2026-08-09T01:19:56Z --- Cursored, public enumeration of this instance's committed, pinned files, so another instance can replicate pins by poll-and-set-diff — the origin-side enabler for follow (ATFS-ofke). Design agreed in discussion, 2026-08-07. @@ -30,3 +30,14 @@ Cursored, public enumeration of this instance's committed, pinned files, so anot ## Summary of Changes lexicons/dev/atfs/sync/listFiles.json defines the query (limit/cursor params, {cursor?, files[]} of dev.atfs.file refs) with the poll-not-subscription and public rationale in its description. internal/xrpc/listfiles.go implements it: GET-only, unauthenticated, limit clamped to [1, 1000] (default 500), cursor resumed by plain string compare over the CID-lexical List order; skips half-writes (Stat ErrNotFound), empty pin sets, and blobs whose IPFS root is unresolvable (reusing resolveIPFSRoot); 500 only when List fails before yielding anything. Mounted in xrpc.New; the package doc now introduces the dev.atfs.sync namespace and the no-listBlobs-alias rationale. Tests reuse the shared harness — fakeIndexer gained a per-cid errFor override so indexing can fail for one blob among several in the same request. CLAUDE.md protocol surface and README gained Enumeration entries. + +## Renamed (2026-08-09, ATFS-jrzm) + +The NSID chosen here, `dev.atfs.sync.listFiles`, is now +`dev.atfs.repo.listFiles`. The `sync` namespace this bean opened — "the +replication surface, where future follow-facing endpoints live" — never +gained a second member, and atfs's repo/sync boundary never matched +atproto's anyway (`dev.atfs.repo.getFile` is aliased as +`com.atproto.sync.getBlob`), so the split bought neither company nor +familiarity. Everything else decided here stands, including the deliberate +absence of a `com.atproto.sync.listBlobs` alias. diff --git a/.beans/ATFS-9a6p--unixfs-directory-dags-store-pin-and-serve.md b/.beans/ATFS-9a6p--unixfs-directory-dags-store-pin-and-serve.md index 09b1f61..5569869 100644 --- a/.beans/ATFS-9a6p--unixfs-directory-dags-store-pin-and-serve.md +++ b/.beans/ATFS-9a6p--unixfs-directory-dags-store-pin-and-serve.md @@ -14,7 +14,7 @@ Open design questions: - **The way in**: extend pinFile to accept a directory root, or a sibling procedure? The fetch walks the dag-pb tree; each contained file's bytes should land as an ordinary blessed blob (deduping with direct uploads for free), with directory/interior nodes stored verbatim — generalizing Indexer.Adopt's external-DAG model. But pinFile's input (dev.atfs.file) is defined by a single file's blessed CID, and a directory root has no blessed-file equivalent — the input shape needs deciding. - **Claim model**: what is the claim unit — one claim on the root covering the whole tree, or a claim per contained file with a source like "site:"? Shared blobs (a file in two sites, or in a site and a direct upload) must compose with the pin-set/GC rules; deleting a site root should release the whole tree's claims. -- **Enumeration/follow**: do directory roots appear in dev.atfs.sync.listFiles (its entries are dev.atfs.file-shaped, which a directory root is not)? A follower mirroring a site needs the whole tree — interacts with ATFS-3x09's lexicon and ATFS-ofke's sync loop. +- **Enumeration/follow**: do directory roots appear in dev.atfs.repo.listFiles (its entries are dev.atfs.file-shaped, which a directory root is not)? A follower mirroring a site needs the whole tree — interacts with ATFS-3x09's lexicon and ATFS-ofke's sync loop. - **Serving**: subpath resolution + index.html semantics. Strong candidate: boxo/gateway (boxo is already a dependency) over an OFFLINE blockservice backed by the same local blockstore adapter Bitswap serving already uses — full path/index.html semantics with local-only guaranteed by construction. Alternative: extend the hand-rolled internal/gateway. Evaluate both. - **Bitswap/DHT**: directory blocks should be served to the swarm and their roots provided, like file-DAG blocks are. diff --git a/.beans/ATFS-jrzm--move-listfiles-into-devatfsrepo-retiring-the-sync.md b/.beans/ATFS-jrzm--move-listfiles-into-devatfsrepo-retiring-the-sync.md new file mode 100644 index 0000000..512c765 --- /dev/null +++ b/.beans/ATFS-jrzm--move-listfiles-into-devatfsrepo-retiring-the-sync.md @@ -0,0 +1,56 @@ +--- +# ATFS-jrzm +title: Move listFiles into dev.atfs.repo, retiring the sync namespace +status: completed +type: task +priority: normal +created_at: 2026-08-09T01:15:17Z +updated_at: 2026-08-09T01:20:38Z +--- + +JP, 2026-08-09: rename dev.atfs.sync.listFiles to dev.atfs.repo.listFiles, reversing ATFS-3x09's choice to open a separate sync namespace for it. + +ATFS-3x09 put listFiles under a new `sync` namespace as "the replication surface, where future follow-facing endpoints live". A year of hindsight it didn't get: the namespace still has exactly one member, and the promise of future neighbours is doing all the work justifying it. Meanwhile atfs's repo/sync boundary doesn't line up with atproto's anyway — dev.atfs.repo.getFile is aliased as com.atproto.sync.getBlob — so the split buys no familiarity either. One namespace for the whole file surface is the simpler thing to explain. + +Timing: the lexicons are not published yet (ATFS-0oko), so the rename is free on that side. It is a wire break for the shipped v0.1.0 surface — the README's follow example and internal/follow both name the path — but the only instance pointing at it today is the bench node. + +Knock-on for ATFS-0oko: the authority DNS drops from three TXT records to two (_lexicon.atfs.dev and _lexicon.repo.atfs.dev), since no dev.atfs.sync.* NSID remains. + +- [x] Lexicon moves to lexicons/dev/atfs/repo/listFiles.json with the new id; drop the sync-namespace framing from its description, keep the poll-not-subscription, public-by-design and transient-omission rationale +- [x] internal/xrpc: rename the route/LXM, rewrite the package and uploadfile.go comments that explain the two-namespace split (the no-com.atproto.sync.listBlobs-alias rationale still stands and should survive) +- [x] internal/follow: listing.go's LXM const and the doc comments naming the old path +- [x] internal/store: the comment naming listFiles' export rule +- [x] README (follow example + Enumeration section) and CLAUDE.md protocol surface +- [x] lexicons/dev/atfs/server.json: the follows field description names the old path +- [x] Reconcile the beans that record the old decision: ATFS-3x09, ATFS-ofke, ATFS-wira, ATFS-1gpu, ATFS-9a6p, ATFS-0oko + +## Summary of Changes + +`dev.atfs.sync.listFiles` is now `dev.atfs.repo.listFiles`, and the +`dev.atfs.sync` namespace is gone. A rename, not an alias: the old path +stops being served, deliberately without a back-compat shim. + +The lexicon moved to lexicons/dev/atfs/repo/listFiles.json; its description +lost the namespace framing (the "unlike the repo.* calls" clause became +"unlike uploadFile/pinFile/deleteFile") and kept the poll-not-subscription, +public-by-design, directly-claimed-only and transient-omission rationale +intact. internal/xrpc's uploadfile.go package doc no longer argues from +namespaces — listFiles is now explained as the odd one out because it is +unauthenticated and public, being the replication surface a follower polls +— while the no-`com.atproto.sync.listBlobs`-alias rationale survives +verbatim beneath it. The LXM consts in internal/xrpc and internal/follow, +comments in internal/store, the `follows` description in +lexicons/dev/atfs/server.json, README's follow example and Enumeration +section, and CLAUDE.md's protocol surface all follow. No router or auth +change was needed: xrpc.New mounts the method individually and listFiles +carries no `lxm` claim. + +Also fixed a pre-existing typo in the same description, which named +`dev.atfs.repo.pinFiles` — an NSID that does not exist. Worth catching +before these get published. + +Reconciled the beans holding the old decision: ATFS-3x09 keeps its +reasoning but is retitled and carries a rename note, and ATFS-ofke, +ATFS-wira, ATFS-1gpu and ATFS-9a6p now name the new NSID. ATFS-0oko's +publishing recipe drops from three authority TXT records to two, since no +`dev.atfs.sync.*` NSID remains to need `_lexicon.sync.atfs.dev`. diff --git a/.beans/ATFS-ofke--follow-mirror-another-atfs-instances-claimed-conte.md b/.beans/ATFS-ofke--follow-mirror-another-atfs-instances-claimed-conte.md index a4825d3..d4dea24 100644 --- a/.beans/ATFS-ofke--follow-mirror-another-atfs-instances-claimed-conte.md +++ b/.beans/ATFS-ofke--follow-mirror-another-atfs-instances-claimed-conte.md @@ -14,7 +14,7 @@ The long-anticipated follow feature. Design settled in discussion (2026-08-07/08 The design questions, now all settled: -- **Enumeration surface — SETTLED (ATFS-3x09)**: `dev.atfs.sync.listFiles`, a cursored public query the follower polls, set-diffing each full walk against its own mirror set. Deliberately not firehose-shaped: the store keeps no event log, and a set-diff over current state yields both pins and unpins with no history to replay. Its public-by-design rationale (every pinned cid is already a DHT provider record) also bears on the consent question below. +- **Enumeration surface — SETTLED (ATFS-3x09)**: `dev.atfs.repo.listFiles` (named `dev.atfs.sync.listFiles` until ATFS-jrzm), a cursored public query the follower polls, set-diffing each full walk against its own mirror set. Deliberately not firehose-shaped: the store keeps no event log, and a set-diff over current state yields both pins and unpins with no history to replay. Its public-by-design rationale (every pinned cid is already a DHT provider record) also bears on the consent question below. - **Consent — SETTLED (2026-08-07)**: unilateral. listFiles is public and every pinned cid is already a DHT provider record — any scraper could mirror the content regardless, so opt-in would be unenforceable theatre. No server-record schema change. - **Claim holder — SETTLED (2026-08-08)**: the followed *server's* DID (its serviceDid, e.g. `did:web:atfs.byjp.me`), NOT the local owner. Uploader DIDs never leave an instance (listFiles exposes no claimants), and a distinct DID means follow claims can never collapse into the owner's own upload claims under did-keyed merging — including the same-owner multi-instance case, which owner-held follow claims would have broken. Carried wrinkle: the same did:web could also be registered as a real atproto *account* (and could then mint service-auth JWTs), so claims need a server-vs-account class marker — merge/unpin treat (did, class) as claim identity, and deleteFile only ever touches account-class claims; follow claims belong exclusively to the sync loop. Open implementation detail: a serviceDid-less instance is still followable, so its claims need a fallback identity (peer-id-derived did:key?). - **Sync cadence & drift — SETTLED (2026-08-08)**: fixed poll constant (order-of-hourly, no config knob). Additions act immediately through the pinFile intent machinery; releases require the file absent for two consecutive polls, since listFiles documents transient omission (GC-pending / not-yet-indexed) and release-repin churn is worse than an hour of staleness. diff --git a/.beans/ATFS-wira--reconcile-the-follow-beans-with-the-listfiles-deci.md b/.beans/ATFS-wira--reconcile-the-follow-beans-with-the-listfiles-deci.md index 7995c87..d0aeede 100644 --- a/.beans/ATFS-wira--reconcile-the-follow-beans-with-the-listfiles-deci.md +++ b/.beans/ATFS-wira--reconcile-the-follow-beans-with-the-listfiles-deci.md @@ -12,9 +12,9 @@ blocked_by: The listFiles decision (ATFS-3x09) answers ATFS-ofke's first open question — the enumeration surface — and the follow feature is currently described by TWO draft beans: ATFS-1gpu (the early door-keeping placeholder) and ATFS-ofke (the richer post-pinFile write-up). Give follow a single home and record what's now decided: -- [x] ATFS-ofke: resolve the "Enumeration surface" open question — dev.atfs.sync.listFiles, poll-and-set-diff on the self-heal cadence (link ATFS-3x09); note the public-enumeration rationale also bears on the consent question +- [x] ATFS-ofke: resolve the "Enumeration surface" open question — dev.atfs.repo.listFiles (then named dev.atfs.sync.listFiles), poll-and-set-diff on the self-heal cadence (link ATFS-3x09); note the public-enumeration rationale also bears on the consent question - [x] Scrap ATFS-1gpu as superseded by ATFS-ofke (with Reasons for Scrapping pointing there), so follow's design notes aren't split across two drafts ## Summary of Changes -ATFS-ofke: the Enumeration surface question is marked settled, pointing at dev.atfs.sync.listFiles (ATFS-3x09) and noting the public-enumeration rationale also feeds the consent question. ATFS-1gpu is scrapped as superseded by ATFS-ofke — its door-keeping purpose (pluggable claim sources in the store) is fulfilled, so follow now has a single home. +ATFS-ofke: the Enumeration surface question is marked settled, pointing at dev.atfs.repo.listFiles (ATFS-3x09; the sync name it carried then was retired by ATFS-jrzm) and noting the public-enumeration rationale also feeds the consent question. ATFS-1gpu is scrapped as superseded by ATFS-ofke — its door-keeping purpose (pluggable claim sources in the store) is fulfilled, so follow now has a single home. diff --git a/CLAUDE.md b/CLAUDE.md index ee22006..12bd146 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -104,7 +104,7 @@ manifest-format change — tracked as a deferred bean. server-class claims belong exclusively to the follow sync loop. Old sidecars need no migration — everything written before follow is account-class by construction. -- **Enumeration:** `dev.atfs.sync.listFiles` is a cursored, public query +- **Enumeration:** `dev.atfs.repo.listFiles` is a cursored, public query listing this instance's committed, **directly-claimed** files — at least one account-class claim (upload or pin). Content held only as a mirror of a followed instance is deliberately never exported, so a mirror never diff --git a/README.md b/README.md index e92c018..288d90b 100644 --- a/README.md +++ b/README.md @@ -212,7 +212,7 @@ Public, no token, cursor-paginated: ```sh CURSOR= while :; do - PAGE=$(curl -sS "$ATFS_SERVER/xrpc/dev.atfs.sync.listFiles?limit=500&cursor=$CURSOR") + PAGE=$(curl -sS "$ATFS_SERVER/xrpc/dev.atfs.repo.listFiles?limit=500&cursor=$CURSOR") jq -c '.files[]' <<<"$PAGE" CURSOR=$(jq -r '.cursor // empty' <<<"$PAGE") [ -z "$CURSOR" ] && break @@ -251,7 +251,7 @@ It answers immediately with a state — `seeking`, `fetching`, `pinned` or `fail ## Enumeration -`GET /xrpc/dev.atfs.sync.listFiles?limit=&cursor=` lists this instance's committed, *directly claimed* files, a page at a time — `dev.atfs.file`-shaped entries, the same shape `pinFile` takes as input. `limit` (default 500, max 1000) bounds the page; `cursor` resumes from a previous response's `cursor` field, which is present only when the page filled (its absence marks the last page). Unauthenticated, unlike the upload/pin/delete calls — every cid it lists is already public, announced to the DHT and served at `/ipfs/`. +`GET /xrpc/dev.atfs.repo.listFiles?limit=&cursor=` lists this instance's committed, *directly claimed* files, a page at a time — `dev.atfs.file`-shaped entries, the same shape `pinFile` takes as input. `limit` (default 500, max 1000) bounds the page; `cursor` resumes from a previous response's `cursor` field, which is present only when the page filled (its absence marks the last page). Unauthenticated, unlike the upload/pin/delete calls — every cid it lists is already public, announced to the DHT and served at `/ipfs/`. "Directly claimed" means uploaded here or pinned here by one of this instance's accounts. Content held only because this instance *follows* another is served and announced as normal but never listed here — a mirror doesn't re-export what it mirrors. That's what stops two instances following each other from echoing forever, lets an origin's deletions travel outward, and keeps mirroring non-transitive: follow each origin you actually want. A file mid-deletion or not yet indexed for IPFS is left off the listing too, so a follower should expect the set to shift slightly between polls and should never read a single absence as a deletion. diff --git a/internal/follow/follow.go b/internal/follow/follow.go index 3f81a52..2886a8c 100644 --- a/internal/follow/follow.go +++ b/internal/follow/follow.go @@ -4,7 +4,7 @@ // A follow is one line in the owner's dev.atfs.server record: the at-uri of // another instance's own server record. This package resolves that record // (identity first — a followed instance can move endpoints without breaking -// the follow), walks its public dev.atfs.sync.listFiles enumeration on a +// the follow), walks its public dev.atfs.repo.listFiles enumeration on a // fixed cadence, set-diffs the result against what this instance already // mirrors from it, and turns the difference into ordinary pins and unpins. // There is no second fetch engine: an addition is exactly a @@ -18,7 +18,7 @@ // it logs and waits for the next poll. Only a listing that completed can // say a file is gone. // -// Even a completed listing doesn't release on sight. dev.atfs.sync.listFiles +// Even a completed listing doesn't release on sight. dev.atfs.repo.listFiles // documents that files in transient states (mid-GC, not yet indexed) drop // out of a page and come back, so a release needs the file absent from two // consecutive successful polls. Releasing and re-fetching content is far diff --git a/internal/follow/listing.go b/internal/follow/listing.go index 6545ef7..b0571c3 100644 --- a/internal/follow/listing.go +++ b/internal/follow/listing.go @@ -18,7 +18,7 @@ import ( const ( // listFilesLXM is the public enumeration query a follower polls. - listFilesLXM = "dev.atfs.sync.listFiles" + listFilesLXM = "dev.atfs.repo.listFiles" // pageLimit is the page size asked for, the origin's own maximum. pageLimit = 1000 // pageTimeout bounds one page request. @@ -33,7 +33,7 @@ const ( maxPages = 10_000 ) -// listing is one page of dev.atfs.sync.listFiles output. +// listing is one page of dev.atfs.repo.listFiles output. type listing struct { Cursor string `json:"cursor"` Files []fileEntry `json:"files"` diff --git a/internal/store/store.go b/internal/store/store.go index 229cdb8..fd61ff5 100644 --- a/internal/store/store.go +++ b/internal/store/store.go @@ -162,7 +162,7 @@ func (m Meta) HasClaim(did string, class Class) bool { // AnyClaim reports whether anyone at all holds a claim of this class. // AnyClaim(ClassAccount) is "is this content directly claimed here" — what -// dev.atfs.sync.listFiles exports on, so a mirror never re-exports what it +// dev.atfs.repo.listFiles exports on, so a mirror never re-exports what it // merely mirrors. func (m Meta) AnyClaim(class Class) bool { for _, p := range m.Pins { diff --git a/internal/store/store_test.go b/internal/store/store_test.go index 7d71474..69345fa 100644 --- a/internal/store/store_test.go +++ b/internal/store/store_test.go @@ -615,7 +615,7 @@ func TestClaimClasses_CoexistAndReleaseIndependently(t *testing.T) { } } -// TestAnyClaim_ReportsDirectClaims checks what dev.atfs.sync.listFiles +// TestAnyClaim_ReportsDirectClaims checks what dev.atfs.repo.listFiles // exports on: a purely-mirrored blob holds claims but none of them direct. func TestAnyClaim_ReportsDirectClaims(t *testing.T) { s := open(t) diff --git a/internal/xrpc/listfiles.go b/internal/xrpc/listfiles.go index 278a51b..ee6601a 100644 --- a/internal/xrpc/listfiles.go +++ b/internal/xrpc/listfiles.go @@ -9,10 +9,10 @@ import ( "atfs.dev/internal/store" ) -// listFilesLXM is dev.atfs.sync.listFiles's canonical lexicon method name. +// listFilesLXM is dev.atfs.repo.listFiles's canonical lexicon method name. // Like getFile it needs no auth, so there's no `lxm` claim to verify — but // unlike getFile it has no com.atproto alias (see the package doc comment). -const listFilesLXM = "dev.atfs.sync.listFiles" +const listFilesLXM = "dev.atfs.repo.listFiles" // defaultListFilesLimit and maxListFilesLimit bound a page: an absent limit // gets the default, and anything outside [1, max] is clamped into range @@ -23,7 +23,7 @@ const ( maxListFilesLimit = 1000 ) -// listFilesResponse is dev.atfs.sync.listFiles's output shape: a page of +// listFilesResponse is dev.atfs.repo.listFiles's output shape: a page of // dev.atfs.file-shaped entries, plus a resume cursor present only when the // page filled — see listFiles. type listFilesResponse struct { diff --git a/internal/xrpc/uploadfile.go b/internal/xrpc/uploadfile.go index 808f9d7..0c38bdf 100644 --- a/internal/xrpc/uploadfile.go +++ b/internal/xrpc/uploadfile.go @@ -12,10 +12,10 @@ // membership rather than the upload allowlist. Neither does // dev.atfs.repo.pinFile (see pinfile.go), which is upload's mirror image — // content this instance fetches for a caller instead of being handed — and -// so shares uploadFile's allowlist gate. dev.atfs.sync.listFiles (see -// listfiles.go) belongs to a different namespace, dev.atfs.sync: the -// instance-to-instance replication surface, distinct from these repo.* file -// operations, with listFiles today as its only member. It has no +// so shares uploadFile's allowlist gate. dev.atfs.repo.listFiles (see +// listfiles.go) is the odd one out among these: unauthenticated and +// public, since it's the instance-to-instance replication surface a +// follower polls, not a file operation gated by who's calling. It has no // com.atproto.sync.listBlobs alias — that call's contract is per-repo (a // required did, a since cursor keyed to repo revision, a bare cids[] // response), with no instance-wide equivalent for listFiles to mirror. diff --git a/lexicons/dev/atfs/repo/listFiles.json b/lexicons/dev/atfs/repo/listFiles.json new file mode 100644 index 0000000..084dce6 --- /dev/null +++ b/lexicons/dev/atfs/repo/listFiles.json @@ -0,0 +1,46 @@ +{ + "lexicon": 1, + "id": "dev.atfs.repo.listFiles", + "defs": { + "main": { + "type": "query", + "description": "Enumerate every file this instance directly claims — the whole instance, not scoped to any one account, since atfs has no per-repo notion to scope by. This exists so another atfs instance can replicate pins: a follower walks every page, diffs the resulting cid set against its own mirror of this instance, dev.atfs.repo.pinFile whatever's new, and releases whatever's vanished. It's a poll rather than a subscription deliberately — the store keeps no event log, so there's nothing for a websocket-style firehose to replay, and a plain set-diff over current state already yields both pins and unpins with no history needed. Public and unauthenticated, unlike uploadFile/pinFile/deleteFile: every cid this instance pins is already announced to the IPFS DHT as a provider record and served at /ipfs/, so nothing here is secret, and an instance that has disabled uploads (no serviceDid configured) must still be enumerable. Only DIRECTLY claimed content is listed: a file must have been uploaded here or pinned here by one of this instance's accounts. Content this instance merely mirrors from an instance it follows is deliberately absent, so a mirror never re-exports what it mirrors — which is what makes an A-follows-B-follows-A pair converge instead of echoing, lets an origin's deletions propagate outward, and keeps mirroring non-transitive (follow each origin you actually want). The instance still serves mirrored content at /ipfs/ and announces it to the DHT; it just doesn't advertise it here. A file mid-GC (its pin list has emptied but the bytes haven't been swept yet) or not yet indexed (a large blob whose UnixFS DAG hasn't finished building) is omitted from every page too — both are transient states that can appear or disappear between one poll and the next, so a follower should expect the set it sees to shift slightly poll to poll even with no new uploads, and should never treat a single absence as a deletion.", + "parameters": { + "type": "params", + "properties": { + "limit": { + "type": "integer", + "minimum": 1, + "maximum": 1000, + "default": 500, + "description": "Maximum number of files to return in this page." + }, + "cursor": { + "type": "string", + "description": "Opaque resume token from a previous call's response. Omit to start from the beginning." + } + } + }, + "output": { + "encoding": "application/json", + "schema": { + "type": "object", + "required": ["files"], + "properties": { + "cursor": { + "type": "string", + "description": "Present only when this page filled up to the requested limit, meaning more files may follow. Its absence marks the final page." + }, + "files": { + "type": "array", + "items": { + "type": "ref", + "ref": "dev.atfs.file" + } + } + } + } + } + } + } +} diff --git a/lexicons/dev/atfs/server.json b/lexicons/dev/atfs/server.json index aae3971..500691d 100644 --- a/lexicons/dev/atfs/server.json +++ b/lexicons/dev/atfs/server.json @@ -42,7 +42,7 @@ }, "follows": { "type": "array", - "description": "Other atfs instances this one mirrors, each named by the at-uri of that instance's own dev.atfs.server record (at://{their-owner}/dev.atfs.server/{their-peer-id}). Identity rather than a URL, so a followed instance can move endpoints without breaking the follow: the follower resolves this record to learn where to poll (its endpoints) and whose name to hold the mirrored claims under (its serviceDid, or an identity derived from its peer ID when it declares none). The follower periodically walks the followed instance's dev.atfs.sync.listFiles, pins whatever is new, and releases whatever has been absent from two consecutive listings. Following is unilateral and needs no consent: listFiles is public and every pinned cid is already a DHT provider record, so an opt-in would be unenforceable. Only directly-claimed content is exported by listFiles, so following an instance never transitively mirrors what *it* follows — follow each origin you want. Removing an entry releases every claim that instance's mirror held here, and the content is deleted once nothing else claims it.", + "description": "Other atfs instances this one mirrors, each named by the at-uri of that instance's own dev.atfs.server record (at://{their-owner}/dev.atfs.server/{their-peer-id}). Identity rather than a URL, so a followed instance can move endpoints without breaking the follow: the follower resolves this record to learn where to poll (its endpoints) and whose name to hold the mirrored claims under (its serviceDid, or an identity derived from its peer ID when it declares none). The follower periodically walks the followed instance's dev.atfs.repo.listFiles, pins whatever is new, and releases whatever has been absent from two consecutive listings. Following is unilateral and needs no consent: listFiles is public and every pinned cid is already a DHT provider record, so an opt-in would be unenforceable. Only directly-claimed content is exported by listFiles, so following an instance never transitively mirrors what *it* follows — follow each origin you want. Removing an entry releases every claim that instance's mirror held here, and the content is deleted once nothing else claims it.", "items": { "type": "string", "format": "at-uri" diff --git a/lexicons/dev/atfs/sync/listFiles.json b/lexicons/dev/atfs/sync/listFiles.json deleted file mode 100644 index 4394a39..0000000 --- a/lexicons/dev/atfs/sync/listFiles.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "lexicon": 1, - "id": "dev.atfs.sync.listFiles", - "defs": { - "main": { - "type": "query", - "description": "Enumerate every file this instance directly claims — the whole instance, not scoped to any one account, since atfs has no per-repo notion to scope by. This exists so another atfs instance can replicate pins: a follower walks every page, diffs the resulting cid set against its own mirror of this instance, dev.atfs.repo.pinFiles whatever's new, and releases whatever's vanished. It's a poll rather than a subscription deliberately — the store keeps no event log, so there's nothing for a websocket-style firehose to replay, and a plain set-diff over current state already yields both pins and unpins with no history needed. Public and unauthenticated, unlike the repo.* calls: every cid this instance pins is already announced to the IPFS DHT as a provider record and served at /ipfs/, so nothing here is secret, and an instance that has disabled uploads (no serviceDid configured) must still be enumerable. Only DIRECTLY claimed content is listed: a file must have been uploaded here or pinned here by one of this instance's accounts. Content this instance merely mirrors from an instance it follows is deliberately absent, so a mirror never re-exports what it mirrors — which is what makes an A-follows-B-follows-A pair converge instead of echoing, lets an origin's deletions propagate outward, and keeps mirroring non-transitive (follow each origin you actually want). The instance still serves mirrored content at /ipfs/ and announces it to the DHT; it just doesn't advertise it here. A file mid-GC (its pin list has emptied but the bytes haven't been swept yet) or not yet indexed (a large blob whose UnixFS DAG hasn't finished building) is omitted from every page too — both are transient states that can appear or disappear between one poll and the next, so a follower should expect the set it sees to shift slightly poll to poll even with no new uploads, and should never treat a single absence as a deletion.", - "parameters": { - "type": "params", - "properties": { - "limit": { - "type": "integer", - "minimum": 1, - "maximum": 1000, - "default": 500, - "description": "Maximum number of files to return in this page." - }, - "cursor": { - "type": "string", - "description": "Opaque resume token from a previous call's response. Omit to start from the beginning." - } - } - }, - "output": { - "encoding": "application/json", - "schema": { - "type": "object", - "required": ["files"], - "properties": { - "cursor": { - "type": "string", - "description": "Present only when this page filled up to the requested limit, meaning more files may follow. Its absence marks the final page." - }, - "files": { - "type": "array", - "items": { - "type": "ref", - "ref": "dev.atfs.file" - } - } - } - } - } - } - } -}