diff --git a/HACKING.md b/HACKING.md index 4c90b9ab..f36821aa 100644 --- a/HACKING.md +++ b/HACKING.md @@ -1,23 +1,28 @@ ## git repo contents -Commands (run with, eg, `go run ./cmd/bigsky`): +Run with, eg, `go run ./cmd/bigsky`): - `cmd/bigsky`: BGS+indexer daemon - `cmd/palomar`: search indexer and query servcie (OpenSearch) - `cmd/gosky`: client CLI for talking to a PDS - `cmd/lexgen`: codegen tool for lexicons (Lexicon JSON to Go package) -- `cmd/laputa`: PDS daemon +- `cmd/laputa`: partial PDS daemon (not usable or under development) - `cmd/stress`: connects to local/default PDS and creates a ton of random posts - `cmd/beemo`: slack bot for moderation reporting (Bluesky Moderation Observer) - `cmd/fakermaker`: helper to generate fake accounts and content for testing +- `cmd/supercollider`: event stream load generation tool +- `cmd/sonar`: event stream monitoring tool - `gen`: dev tool to run CBOR type codegen Packages: - `api`: mostly output of lexgen (codegen) for lexicons: structs, CBOR marshaling. some higher-level code, and a PLC client (may rename) - - `api/atprot`: generated types for `com.atproto` lexicon + - `api/atproto`: generated types for `com.atproto` lexicon - `api/bsky`: generated types for `app.bsky` lexicon +- `atproto/crypto`: crytographic helpers (signing, key generation and serialization) +- `atproto/syntax`: string types and parsers for identifiers, datetimes, etc +- `atproto/identity`: DID and handle resolution - `bgs`: server implementation for crawling, etc - `carstore`: library for storing repo data in CAR files on disk, plus a metadata SQL db - `events`: types, codegen CBOR helpers, and persistence for event feeds @@ -35,26 +40,22 @@ Packages: - `util`: a few common definitions (may rename) - `xrpc`: XRPC client (not server) helpers -Other: -- `testscripts`: shell scripts that run gosky (CLI client) against local PDS - - -## jargon +## Jargon - BGS: Big Graph Service (or Server), which centrals crawls/consumes content from "all" PDSs and re-broadcasts as a firehose - PDS: Personal Data Server (or Service), which stores user atproto repositories and acts as a user agent in the network - CLI: Command Line Tool - CBOR: a binary serialization format, smilar to JSON -- PLC: "placeholder" DID provider -- DID: Decentralized IDentifier, a flexible W3C specification for persistent identifiers in URI form (eg, "did:plc:abcd1234") +- PLC: "placeholder" DID provider, see +- DID: Decentralized IDentifier, a flexible W3C specification for persistent identifiers in URI form (eg, `did:plc:abcd1234`) - XRPC: atproto convention for HTTP GET and POST endpoints specified by namespaced Lexicon schemas - CAR: simple file format for storing binary content-addressed blocks/blobs, sort of like .tar files - CID: content identifier for binary blobs, basically a flexible encoding of hash values - MST: Merkle Search Tree, a key/value map data structure using content addressed nodes -## lexicon and CBOR marshaling code generation +## Lexicon and CBOR code generation `gen/main.go` has a list of types internal to packages in this repo which need CBOR helper codegen. If you edit those types, or update the listed types/packages, re-run codegen like: @@ -64,26 +65,22 @@ Other: # then generate go run ./gen -To run codegen for new or updated Lexicons, using lexgen, first place (or git -checkout) the JSON lexicon files at `../atproto/`. -Then, in *this* repository (indigo), run commands like: +To run codegen for new or updated Lexicons, using lexgen, first place (or git checkout) the JSON lexicon files at `../atproto/`. Then, in *this* repository (indigo), run commands like: go run ./cmd/lexgen/ --package bsky --prefix app.bsky --outdir api/bsky ../atproto/lexicons/app/bsky/ go run ./cmd/lexgen/ --package atproto --prefix com.atproto --outdir api/atproto ../atproto/lexicons/com/atproto/ You may want to delete all the codegen files before re-generating, to detect deleted files. -It can require some manual munging between the lexgen step and a later `go run ./gen` to make sure things compile at least temporarily; otherwise the `gen` will not run. -In some cases, you might also need to add new types to ./gen/main.go. +It can require some manual munging between the lexgen step and a later `go run ./gen` to make sure things compile at least temporarily; otherwise the `gen` will not run. In some cases, you might also need to add new types to `./gen/main.go`. -To generate server stubs and handlers, push them in a temporary directory -first, then merge changes in to the actual PDS code: +To generate server stubs and handlers, push them in a temporary directory first, then merge changes in to the actual PDS code: mkdir tmppds go run ./cmd/lexgen/ --package pds --gen-server --types-import com.atproto:github.com/bluesky-social/indigo/api/atproto --types-import app.bsky:github.com/bluesky-social/indigo/api/bsky --outdir tmppds --gen-handlers ../atproto/lexicons -## tips and tricks +## Tips and Tricks When debugging websocket streams, the `websocat` tool (rust) can be helpful. CBOR binary is sort of mangled in to text by default. Eg: @@ -103,43 +100,21 @@ Set the log level to be more verbose, using an env variable: GOLOG_LOG_LEVEL=info go run ./cmd/pds -## gosky basic usage +## `gosky` basic usage Running against local typescript PDS in `dev-env` mode: # as "alice" user go run ./cmd/gosky/ --pds http://localhost:2583 createSession alice.test hunter2 > bsky.auth -The `bsky.auth` file is the default place that `gosky` and other client -commands will look for auth info. - - -## slack report bot basic usage - -You need an admin token, slack webhook URL, and auth file (see gosky above). -The auth file isn't actually used, only the admin token. - - # configure a slack webhook - export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T028K87/B04NBDB/oWbsHasdf23r2d - - # example pulling admin token out of `pass` password manager - export ATP_AUTH_ADMIN_PASSWORD=`pass bsky/pds-admin-staging | head -n1` - - # example just setting admin token directly - export ATP_AUTH_ADMIN_PASSWORD="someinsecurething123" +The `bsky.auth` file is the default place that `gosky` and other client commands will look for auth info. - # run the bot - GOLOG_LOG_LEVEL=debug go run ./cmd/beemo/ --pds https://pds.staging.example.com --auth bsky.auth notify-reports -## integrated development +## Integrated Development -Sometimes it is helpful to run a PLC, PDS, BGS, labelmaker, and other -components, all locally on your laptop, across languages. This section -describes one setup for this. +Sometimes it is helpful to run a PLC, PDS, BGS, labelmaker, and other components, all locally on your laptop, across languages. This section describes one setup for this. -First, you need PostgreSQL running locally. This could be via docker, or the -following commands assume some kind of debian/ubuntu setup with a postgres -server package installed and running. +First, you need PostgreSQL running locally. This could be via docker, or the following commands assume some kind of debian/ubuntu setup with a postgres server package installed and running. Create a user and databases for PLC+PDS: diff --git a/README.md b/README.md index 72e7417e..2d72b755 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,38 @@ ![photo](https://static.bnewbold.net/tmp/indigo_serac.jpeg) -indigo: golang code for Bluesky's atproto services -================================================== +indigo: atproto libraries and services in golang +================================================ Some Bluesky software is developed in Typescript, and lives in the [bluesky-social/atproto](https://github.com/bluesky-social/atproto) repository. Some is developed in Go, and lives here. -

+Services implemented in this repository: + +* **`bigsky`** ([README](./cmd/bigsky/README.md)): "Big Graph Service" (BGS) reference implementation, running at `bsky.network` +* **`palomar`** ([README](./cmd/palomar/README.md)): fulltext search service for -Everything in this repository is an work in progress. Features and "Lexicons" may be removed or updated, software interfaces broken, etc. -We are developing in the open, but not ready to accept or review significant contributions. Keep checking back! +## Development Quickstart + +All the packages in this repository are under active development. Features and software interfaces have not stabilized and may break or be removed.

+First, you will need the Go toolchain installed. We develop using the latest stable version of the language. + +The Makefile provides wrapper commands for basic development: + + make build + make test + make fmt + make lint + +Individual commands can be run like: + + go run ./cmd/bigsky + +The [HACKING](./HACKING.md) file has a list of commands and packages in this repository and some other development tips. + ## What is atproto? @@ -21,25 +40,20 @@ We are developing in the open, but not ready to accept or review significant con The Authenticated Transfer Protocol ("ATP" or "atproto") is a decentralized social media protocol, developed by [Bluesky PBC](https://blueskyweb.xyz). Learn more at: -- [Protocol Documentation](https://atproto.com/docs) -- [Overview Guide](https://atproto.com/guides/overview) 👈 Good place to start +- [Overview and Guides](https://atproto.com/guides/overview) 👈 Best starting point +- [Github Discussions](https://github.com/bluesky-social/atproto/discussions) 👈 Great place to ask questions +- [Protocol Specifications](https://atproto.com/specs/atp) - [Blogpost on self-authenticating data structures](https://blueskyweb.xyz/blog/3-6-2022-a-self-authenticating-social-protocol) +The Bluesky Social application encompasses a set of schemas and APIs built in the overall AT Protocol framework. The namespace for these "Lexicons" is `app.bsky.*`. -## Development -First, you will need the Go toolchain installed. We develop using the latest stable version of the language. +## Contributions -The Makefile provides wrapper commands for basic development: +We are working in the open, but not ready to actively collaborate on larger contributions to this codebase. - make build - make test - make fmt - make lint +Please at least open an issue ahead of time, *before* starting any non-trivial work that you hope to get reviewed or merged to this repo. -Individual commands can be run like: - - go run ./cmd/bigsky ## Are you a developer interested in building on atproto? diff --git a/cmd/beemo/README.md b/cmd/beemo/README.md new file mode 100644 index 00000000..4908395c --- /dev/null +++ b/cmd/beemo/README.md @@ -0,0 +1,17 @@ + +## beemo: Slack notification bot for moderation reports + +You need an admin token, slack webhook URL, and auth file (see gosky docs). +The auth file isn't actually used, only the admin token. + + # configure a slack webhook + export SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T028K87/B04NBDB/oWbsHasdf23r2d + + # example pulling admin token out of `pass` password manager + export ATP_AUTH_ADMIN_PASSWORD=`pass bsky/pds-admin-staging | head -n1` + + # example just setting admin token directly + export ATP_AUTH_ADMIN_PASSWORD="someinsecurething123" + + # run the bot + GOLOG_LOG_LEVEL=debug go run ./cmd/beemo/ --pds https://pds.staging.example.com --auth bsky.auth notify-reports