diff --git a/blog/posts/bobbin.md b/blog/posts/bobbin.md new file mode 100644 index 00000000..4ebc647f --- /dev/null +++ b/blog/posts/bobbin.md @@ -0,0 +1,200 @@ +--- +atroot: true +template: +slug: bobbin +title: Introducing Bobbin +subtitle: A diskless, API-only AppView for Tangled +date: 2026-07-13 +authors: + - name: Lewis + email: lewis@tangled.org + handle: oyster.cafe +draft: true +--- + +Hello! It is I, Lewis, with my first Tangled post. +At Tangled my job involves a lot of Research & Development, by +which I mean fucking around and finding out. Let me tell you about one +such finding out, which turned into this new Tangled API. + +Bobbin is a read-only, API-only +[AppView](https://atproto.com/guides/glossary#app-view) +for Tangled. It serves the read side of the `sh.tangled.*` +[lexicons](https://atproto.com/specs/lexicon) +over [XRPC](https://atproto.com/specs/xrpc), so a +client can render git repositories, issues, pulls, +comments, stars, & follows, without having to ping N +[PDSes](https://atproto.com/guides/glossary#pds-personal-data-server) +for their data. It has no permanent storage. Everything it knows is only +ever in RAM and gets backfilled from upstream every restart. + +> I, along with many others, find the term "AppView" to be too +> abstract or perhaps even unfortunately-named. In atproto, datasets +> are distributed across Personal Data Servers (PDSes) that users each +> own, so an AppView is a single API through which one can query +> a dataset, in contrast to all downstream clients each maintaining a +> connection with every PDS. An AppView is one of the many programs in +> use to reduce the strain on PDSes, so that a PDS is relatively cheap and +> resource-efficient to run. + +Let me list some problems that I find Tangled's AppView to have, such +that the journey of discovery that led me to Bobbin makes sense: + +1. Tangled's AppView has no API; it doubles as a web client via SSR. +2. It is brittle in the sense that it's not able to backfill +[atproto](https://atproto.com) data easily. +3. Its architecture from the ground up is inherently built for the kind +of web program that assumes *it* is the source of truth of data. +4. Because of all of the above, and the fact that there *is* some +private data such as email->user [DID](https://atproto.com/specs/did) +ties, there ought to only be one AppView for Tangled. This means that +latency to Tangled is vaguely a function of distance from the server +in Stockholm. +5. It is hungry for RAM during a RAM crisis. + +Flipping to the inverse of all of these items would yield me my ideal +AppView. An API-only, easily-backfillable, easily-distributable, +easy-on-RAM program. + +Why "API-only"? Once it's easy to retrieve Tangled data via generic +XRPC API, more programs can be built on top of it; programs such +as alternative frontends, third-party clients, one's own targeted +dashboards, in my case it was to enable a good CLI (though I suppose +that just comes under alternative frontends). The existing Tangled +AppView being its own web client means there's nothing extendable to +build on, unless one is a scraping wizard; one has to figure out their +own way to go grab the Tangled dataset from the world of PDSes in the +wild. + +While I was mulling how to go about creating such a program for fun, I +have also been having a crisis about where to store state, after I've +woken up in cold sweats wondering if Tranquil PDS' homegrown embedded +DB could fail in new mystical ways. How could I avoid this next program +having to write any durable, fallible state? Besides, AppViews are +downstream of PDSes, which means the more we try and hold onto state in +an AppView, the more chance there is that it'll diverge from reality. +All it takes to have an AppView be wildly incorrect is missing a single +record, so it is of upmost importance to have an AppView be able to +reconcile what it thinks it knows with what's actually true. + +Being the simpleton that I am, I think that there is no better way +to make a program whose only job is to cache and serve records than +to make it never touch disk in the first place. I would want it to +simply backfill all of its data from scratch on every restart. In that +case, reconciliation with reality is just a restart away and happens +automatically! Was that feasible? What if such a backfill took a whole +day every time? That would sure make deployments slow, would it anyhow +be worth it? + +My first tests of this concept gave promising results, I +threw together a little server that could in fact backfill +all of Tangled's atproto dataset in around 15 minutes, using +[Hydrant](https://tangled.org/did:plc:6v3ul2ptnqctyxwkz5ti4amn) as its +aggregator. I decided that an acceptable backfill time would be under +5 minutes or less, so that it would be feasible to switch out running +instances on a blue/green deploy within a reasonable warming timeframe. + +> Hydrant is an aggregator for arbitrary atproto datasets that works +> by tailing the live [firehose](https://atproto.com/specs/event-stream), +> backfilling every [repo](https://atproto.com/guides/glossary#data-repo) +> on the network by pulling their +> [CARs](https://atproto.com/specs/repository#car-file-serialization), +> and serving it back over a websocket. All Bobbin has to do is open a +> connection, ask for cursor 0, and (re)build its index off the resulting +> barrage. +> +> Massive gratitude for +> [Dawn](https://tangled.org/did:plc:dfl62fgb7wtjj3fcbb72naae) for having +> made Hydrant! Then for hearing my struggles and going and optimizing it +> also for fun! + +Once I reached backfill in 5 minutes, I moved the goalpost for fun, that +the acceptable backfill time should now be 90 seconds or less. This is +around the number we are approaching today, depending on how warm the +upstream Hydrant instance is; ranging from 30 seconds in best-case +scenario to 20 minutes in a disaster where Hydrant is also pulling +records from PDSes from scratch. + +> Yes, you too can spin these up & order the entirety of Tangled in 20 +> minutes or less, or you get a refund. (Of €0.) + +There's a little conundrum that I encountered: what to have the +program serve while the backfill is still warming? Do I bother +having the program serve requests while aggregate/linked data is +possibly incorrect? I think not. So do I have a program serving +nothing in the backfill time that is possibly 20 minutes at +worst? I probably could have done that, but instead we connect to +[Slingshot](https://tangled.org/did:plc:c7mc2fn47ihdihul4vjwsuy3/tree/main/slingshot) +for point-lookups at least and immediately get accurate +data in the meantime, even if the program itself hasn't technically seen +that data yet. + +> Slingshot is a wonderful edge cache for atproto records +> & identities. Bobbin asks it for single record and identity +> lookups, so that a freshly-born Bobbin can answer individual +> queries accurately before its own index has warmed up. Slingshot +> is part of the [microcosm](https://www.microcosm.blue/) project, +> a bundle of community-run atproto infrastructure. Thanks +> [Fig](https://tangled.org/did:plc:hdhoaan3xa3jiuq4fg4mefid) for making +> it! + +The program that was turning into Bobbin was able to hold all of +Tangled's atproto dataset in around 200MB of RAM, which was good, but +some compression led to 100MB. There is likely more I could compress +that is yet to be done. "Wow, such a small amount of data for the entire +social layer of a whole platform!" you remark, but on the contrary +remember that this is the *smallest* the dataset will likely ever be +from today onwards. Space-saving is of the essence. + +Bobbin uses Slingshot as a record/identity resolver in real time for +anything it doesn't already hold in RAM or hasn't necessarily received +from Hydrant. I love Fig's work, don't get me wrong, but my lean on +Slingshot instead of just writing the pings myself was a cop-out to save +time; if there's one major architectural change yet to come to Bobbin, +it's inlining that away such that Bobbin never has to ping Slingshot and +can figure out faster ways of getting the same data. + +One would be tempted to throw in an embedded instance of Hydrant to +their AppView program itself, and have the serving of a particular +atproto dataset be just a thin wrapper over the Hydrant data; but +I want it to be easy to spin up possibly hundreds of instances of my +program. Having hundreds of Hydrants all pinging PDSes all the time +would be quite wasteful. One Hydrant instance can serve hundreds of +Bobbins without a sweat. + +Since we don't have a Hydrant/relay equivalent for Knot-related data, +Bobbin proxies Knot-related requests straight through to a given knot. I +think a knotstream aggregator is in order, perhaps even just a plugin to +Hydrant that has it serving both. I think Bobbin and Knot Mirror should +merge into one service, but that's just my opinion. + +With a "stateless" AppView, we don't have to worry about databases and +their woes: migrations, production massaging when things are slightly +off compared to the "true" dataset, et cetera. Any time we want to +change/add functionality, all we have to do is simply restart +Bobbin and the operation is done. Another benefit is that Bobbin lends +itself to "cloud-native deployments" as the corpos say. Tangled's own +flagship instance runs on a CloudFlare worker. + +All this being said, I'm not necessarily married to the idea that Bobbin +never touches disk, I think that one cute improvement could be to simply +dump RAM-stored-data contents as a snapshot once every hour and backfill +the difference at startup; but for now I think there's headroom to +continue down this path, considering Bobbin can healthily run with 200MB +given to it at time of writing. Additionally, my original requirement of +being easy on RAM is still fulfilled for now, but the moment it becomes +too heavy is another point at which Bobbin will offload parts of itself +to disk dynamically during runtime. How do I define "too heavy", you +ask? I look inside my heart, which tells me that 1GB of RAM is wasteful, +10GB is unacceptable. Bobbin also must serve users, whose each API +request puts more pressure on the RAM beyond just storing the Tangled +dataset. + +There you have it! Now there's this fun AppView with no disk storage, +sub-90-second backfill, that serves API requests as a little engine +that could. With it, anyone can build programs using the Tangled dataset +more easily. + +There is an instance of Bobbin on +[api.tangled.org](https://api.tangled.org), and hopefully you'd find it +easy to host your own.