collection discovery: retiring the hardcoded allowlist #

Status: design note, not built. Written 2026-08-18 after the birdlover.selfhosted.social incident.

the problem #

wantedCollection (services/src/ingest.zig) decides which firehose collections cause us to discover an actor. It is a hardcoded allowlist:

"app.bsky.actor.profile", "app.bsky.feed.post", "app.bsky.feed.like",
"app.bsky.graph.follow", "sh.tangled.actor.profile"   + "site.standard.*", "place.birds.*"

Two consequences, both structural rather than incidental:

Onboarding an app is a code change and a deploy. Until someone notices and ships, that app's users are discoverable only via #identity.
#identity becomes a single point of failure for those users. That is exactly how birdlover went missing: a birds.place account writes nothing but place.birds.*, so the commit path was blind to it, and the one #identity event it ever produced was dropped by a resolver failure. Two independent gates, both closed, one actor invisible with no signal anywhere.

The allowlist is not wrong about what it lets in. It is wrong as a mechanism: it fails closed, silently, for every atproto app that isn't bsky or tangled — and the set of such apps is the whole point of the network.

the idea: published lexicons as the signal #

Ingest activity from any collection whose NSID authority has a published lexicon. The resolution path already exists in the spec, and this repo already sits on the publishing side of it (scripts/publish-lexicon.py):

collection NSID          place.birds.sighting
  → authority             birds.place              (NSID domain authority, reversed)
  → DNS TXT               _lexicon.birds.place  →  did=did:plc:...
  → that repo's record    com.atproto.lexicon.schema, rkey = the NSID

If that record exists, someone deliberately published a schema for the thing they're writing. That is a meaningful filter:

It costs a legitimate app nothing — publishing lexicons is already the norm for anyone who wants their NSIDs to resolve.
It costs a spammer a domain, a DNS record, a DID, and a published schema, all of which are attributable and revocable.
It fails open for new apps: build an app, publish your lexicon, get discovered. No deploy on our side.

the honest objection #

The user's own framing, and it's right: a spammer can publish lexicons first. Nothing stops a determined actor from doing the setup work to clear the bar. So this is not a spam filter — it is a cost floor and an attribution hook. Its real value is:

it converts "unknown collection" from a silent drop into a cheap lookup,
it gives us a stable key to act on when abuse does appear (the authority domain/DID), so a bad app is one domain-block rather than a lexical whack-a-mole,
it moves the default from deny-unknown to allow-attributable.

Expect to still need reputation on top. The point is to stop the allowlist from deciding who exists.

constraints this must respect #

The serving invariant. Resolution must never run per-event on the ingest path. The firehose is ~4–5 discovered actors/sec against a much larger event rate; a DNS + repo fetch per commit is not viable.
Therefore: cache by authority, not by NSID. Authorities are few and change rarely. An in-process map of authority → {published, checked_at} with a negative-cache TTL keeps the hot path to a hash lookup. Unknown authority → admit the actor optimistically or queue the authority for out-of-band resolution; do not block the reader.
Failure must fail open, and loudly. The whole lesson of this incident is that a lookup failure silently deciding "not interesting" is how actors disappear. A resolution failure should still record the DID — same rule the identity resolver now follows.
Keep a deny-list. Allow-by-default plus a deny-list for known-spam collections is easier to operate than allow-by-enumeration, and it fails in the direction we can see.

the same bug one layer down: what we read, not just who we admit #

handleOp extracts displayName from exactly two collections — app.bsky.actor.profile and sh.tangled.actor.profile. Every other collection, place.birds.profile included, falls to the bare-DID branch. So a birds.place user who fills in a display name has it carried past us on the firehose and dropped, and then renders in search as a bare handle. We are not missing the data; we are discarding it.

Note what this is NOT an argument for: birds.place should not write an app.bsky.actor.profile to work around it. That would be an app writing into a namespace it does not own, on scopes it never requested. The app is behaving correctly; the index is the thing that is wrong.

The complication, and why this is not just endsWith(".profile"): display names sourced from bsky and tangled inherit moderation we already lean on. Accepting displayName from any app's profile record opens a write path for arbitrary apps to set names in our index. The current two-collection list is accidentally acting as a spam control, and removing it without putting authority/reputation in its place trades one failure for a worse one.

So: same authority question as above, applied to extraction rather than admission. Whatever answers "whose records make an actor real" should also answer "whose records may name one."

open questions #

Do we admit the actor on first sight of an unresolved authority and reconcile later, or hold it? Holding reintroduces the failure mode we just fixed.
Where does the authority cache live — in-process per ingester, or a shared KV the worker can also read for /stats attribution?
Is there a cheaper bulk source for "authorities with published lexicons" than resolving them one at a time? (Same shape as the PLC bulk-identity problem; see docs/discovery-and-identity.md.)
What does this do to corpus size? Admitting every actor of every published-lexicon app is strictly more than today. Needs a measured estimate before it ships, per the serving invariant.

until then #

wantedCollection stays hardcoded and carries a TODO pointing here. Every entry added by hand is a reminder that the mechanism is the bug, not the list.