pds #

ATProto Personal Data Server storage.

The AT Protocol is the network behind Bluesky. Each user's data lives there in a repository kept by a Personal Data Server (PDS): signed records in a Merkle Search Tree (MST), and binary blobs such as images. To back up, migrate or inspect a repository you normally go through a running PDS. pds reads and writes the repository storage itself instead. The tree lives in a SQLite blockstore, the blobs in a directory next to it, and named refs point at commits, with head pointing at the current one. Repositories go in and out as CAR files, the format PDSes use to exchange them.

A few terms recur below. A CID (content identifier) names a block by the hash of its bytes, with a prefix that says which hash function and which encoding were used. Here the encoding is DAG-CBOR, a deterministic subset of CBOR in which a link to another block is a tagged CID. A CAR file starts with a header that names the root CIDs, followed by the blocks, each with its CID. The account that owns a repository is named by a DID (decentralised identifier, W3C DID Core) such as did:web:example.com.

Installation #

Install with opam:

$ opam install nox-pds

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install nox-pds

Usage #

Create a repo, put and get records #

let put_and_get ~record_bytes =
  Eio_main.run @@ fun env ->
  Eio.Switch.run @@ fun sw ->
  let fs = Eio.Stdenv.fs env in
  let repo =
    Pds.v ~sw
      Eio.Path.(fs / "my-repo")
      ~did:(Atp.Did.of_string_exn "did:web:example.com")
  in
  Pds.put repo ~collection:"app.bsky.feed.post" ~rkey:"abc123" record_bytes;
  (match Pds.find repo ~collection:"app.bsky.feed.post" ~rkey:"abc123" with
  | Some data -> Fmt.pr "record: %d bytes@." (String.length data)
  | None -> Fmt.pr "not found@.");
  Pds.close repo

List a collection #

let list_posts repo =
  Pds.list repo ~collection:"app.bsky.feed.post"
  |> List.iter (fun (rkey, cid) ->
         Fmt.pr "%s -> %a@." rkey Atp.Cid.pp cid)

Blobs #

let store_image repo image_bytes =
  Pds.put_blob repo ~mime_type:"image/png" image_bytes

Import / export #

let backup repo = Pds.export_car repo
let restore repo car = Pds.import_car repo car

API #

Repository #

• Pds.v ~sw path ~did creates a repository, and Pds.open_ ~sw path opens an existing one.
• Pds.did t is the DID that owns the repository.
• Pds.close t releases the database handle.

Records #

• Pds.put t ~collection ~rkey data writes a record and moves head, both in one transaction.
• Pds.find t ~collection ~rkey returns the record's bytes, if it exists.
• Pds.delete t ~collection ~rkey removes a record.
• Pds.list t ~collection returns the (rkey, cid) pairs of a collection.

Blobs #

• Pds.put_blob t ~mime_type data stores a blob and returns its Atp.Blob_ref.t.
• Pds.blob t cid returns the blob stored under cid, if there is one.

Repository state #

• Pds.head t is the CID of the current commit, and Pds.set_head t cid moves it.
• Pds.checkout t is the MST at head.
• Pds.blockstore t gives access to the blockstore underneath.

Named refs #

A named ref works like a git branch. Pds.ref t name reads one, set_ref t name cid moves it, delete_ref t name removes it, and Pds.list_refs t lists them all.

CAR #

• Pds.import_car t data imports every block of a CAR file and returns how many it imported. Either all blocks go in or none do, and head stays where it was.
• Pds.export_car t writes every block reachable from head as a CAR file.

Storage layout #

<repo>/
├── pds.db              # SQLite database
│   ├── blocks table    # CID → DAG-CBOR bytes
│   ├── refs table      # name → CID (branches)
│   └── meta table      # did, version, etc.
└── blobs/              # Large binary data
    ├── ba/             # First 2 chars of CID
    │   └── bafyrei...  # Full CID as filename
    └── ...

Related Work #

• ocaml-atp provides the ATProto pieces this package is built on: MST, CID, DAG-CBOR and CAR.
• ocaml-sqlite provides the SQLite key-value store under the blockstore.
• Bluesky PDS is the reference implementation, written in TypeScript.

Licence #

ISC. See LICENSE.md.