Provenance #

Provenance provides relative ATProto reputation through public vouches. It contains two Go services:

• appview: a participant index, trust graph, tag index, and private thresholds.
• ui: ATProto OAuth, account autocomplete, vouch management, and trust queries.

SQLite stores the data. sqlc generates the SQL adapter. Storage interfaces separate the services from SQLite.

Local development #

Builds require CGO and a C compiler for the native SQLite driver. The Nix development environment provides the compiler and enables CGO.

1. Enter the development environment:

   nix develop .
   

2. Generate a service key:

   export PROVENANCE_SERVICE_KEY="$(head -c 32 /dev/urandom | base64)"
   

3. Generate an encryption key:

   export PROVENANCE_ENCRYPTION_KEY="$(head -c 32 /dev/urandom | base64)"
   

4. Build the services:

   9 mk build
   

5. Start the appview:

   ./bin/appview
   

6. In another terminal with the same keys, start the UI:

   ./bin/ui
   

7. Open http://127.0.0.1:8000.

The PDS receives your password. Provenance never receives your password. The PDS must support the ATProto localhost OAuth exception.

Browser views #

• /: publish a vouch with labeled color choices: Avoid, Caution, Neutral, Trust, or Strong trust.
• /check: check a person's trust relative to your account and inspect the vouch path.
• /reputation: see the signed sum and mean of qualifying direct vouches, relative to your account.

Both features require login. The login page also supports account suggestions. Type a name or handle, then select a suggestion with the pointer or arrow keys and Enter. You can enter a full handle or DID without a suggestion. Search suggestions come from Bluesky. Direct DID suggestions use ATProto identity resolution. Visible vouches show Bluesky display names and verified handles when available. If a profile lookup fails, the list retains the verified handle or account identifier. Vouches remain public. The separate block choice overrides indirect vouches for the same topic. Topic options and private trust settings remain available in expandable sections. Use Refresh list to load indexed vouches after a publish or removal.

UI errors #

UI errors include a safe message, an error code, and a reference ID in request_id. The response also includes the reference in X-Request-ID. Find the corresponding UI request failed log entry with that reference. Logs include the operation, error code, and upstream HTTP status when available. Logs omit query strings, credentials, and upstream error bodies. Login errors show a recovery page instead of raw JSON. Account search failures do not prevent direct handle or DID entry. After a network error during publication, check the account before you repeat the request. An absent response does not prove that the PDS rejected the write. Login opts your account into the index and imports your existing vouches. The appview reads existing records once, then consumes Microcosm Jetstream events. UI writes do not trigger PDS reads. The UI reports that the index awaits the event stream.

CAUTION: Keep the encryption key. A different key makes existing OAuth sessions unreadable. Keep the service key and encryption key outside the repository.

Configuration #

Variable	Default	Purpose
PROVENANCE_SERVICE_KEY	Required	Private authentication between the UI and appview
PROVENANCE_EXTERNAL_WORKERS	4	External HTTP workers per process, from 1 to 64
PROVENANCE_ENCRYPTION_KEY	Required for UI	Base64-encoded 32-byte AES key
PROVENANCE_PUBLIC_ORIGIN	http://127.0.0.1:8000	Browser origin and OAuth callback origin
PROVENANCE_UI_ADDR	127.0.0.1:8000	UI listen address
PROVENANCE_UI_DB	data/ui.db	Encrypted OAuth and browser sessions
PROVENANCE_APPVIEW_ADDR	127.0.0.1:8001	Appview listen address
PROVENANCE_APPVIEW_URL	http://127.0.0.1:8001	Appview address for the UI
PROVENANCE_APPVIEW_DB	data/appview.db	Vouches, participant index, and private thresholds
PROVENANCE_JETSTREAM_URL	wss://jetstream.fire.hose.cam/subscribe	Microcosm Jetstream stream endpoint
PROVENANCE_SLINGSHOT_URL	https://slingshot.microcosm.blue	Record cache for missing event payloads
PROVENANCE_HOP_PENALTY	0.1	Penalty for each edge after the first edge
PROVENANCE_TRUST_CACHE_TTL	24h	Positive Go duration for cached trust results and paths
ENABLE_SOFT_VOUCH	false	Use participant Bluesky follows and blocks in trust and reputation
ENABLE_SOFT_VOUCH_ALL_AUTHORS	false	Index soft assertions from all observed stream authors. Requires ENABLE_SOFT_VOUCH=true
ENABLE_SOFT_VOUCH_LISTS	false	Use public blocked lists as soft blocks. Has no effect without ENABLE_SOFT_VOUCH=true

External request controls #

Each process shares one bounded HTTP worker queue across repository, identity, OAuth, and Microcosm clients. PROVENANCE_EXTERNAL_WORKERS defaults to four workers. Each worker retains its slot until the response ends or closes. The queue holds at most eight pending requests per worker. Further callers wait for space, subject to their request deadlines. WebSocket handshakes use the queue. An established stream does not occupy a fetch worker. The NixOS appview and UI options each expose externalWorkers. The development service exposes the same option.

Safe reads without request bodies permit at most three attempts for transient transport errors and HTTP 429, 500, 502, 503, or 504. Retries use exponential backoff with jitter. Retry waits and response reads count toward the HTTP client's timeout. The queue does not automatically retry writes. OAuth nonce exchanges and token refreshes retain the SDK's behavior. All HTTP requests identify the application as provenance/0.1 (https://provenance.ngp.computer).

Retryable failures create shared per-origin cooldowns. Retry-After accepts seconds or an HTTP date and sets the minimum cooldown. Cooldowns also apply to new requests, other clients in the process, and writes. A request deadline can end a wait without another attempt. Stream reconnects use jitter and reset their backoff only after at least one event and one minute of connection uptime.

Normal snapshots reuse verified identities in a bounded cache with a five-minute lifetime. They do not purge identities for each collection. Actual indexed identity events invalidate the appview's cached identity. Ignored record events do not invalidate identities. OAuth login and explicit participant recovery still force fresh identity resolution. Without an indexed identity event or explicit recovery, a PDS change can remain cached for five minutes. The UI has its own identity cache and does not receive appview identity events.

These controls bound HTTP concurrency, not requests per second. Multiple processes or replicas have independent limits and cooldowns. Cooldowns do not survive restarts. DNS identity lookups use SDK verification and timeouts, not the HTTP queue. Broad referenced-list snapshots remain a follow-up (tsk-27). Shared account-search caches and endpoint throttles remain a follow-up (tsk-28).

Soft vouches #

Soft vouches are optional. A Bluesky follow scores 0.5. A Bluesky block scores -0.5, not the hard-block score -3. These scores affect trust paths, reputation author eligibility, reputation sums, and reputation means. Explicit vouches take precedence, including zero scores and hard blocks. Otherwise, a Bluesky block overrides a follow. Duplicate records never multiply a pair's contribution. Soft vouches apply to default queries and topic fallback queries, but not tag-only queries. An applicable explicit topic or default assertion overrides a soft assertion. The public vouch schema and write endpoints retain integer-only scores.

The participant mode imports follows and blocks from each participant's PDS at bootstrap or explicit sync. Existing participants receive this import at startup when the feature first becomes active. The all-author mode also indexes subsequent stream records from nonparticipants. It does not enroll those authors or import their repositories. Both modes subscribe to the follow and block collections and use Slingshot for absent event payloads. Disabled mode does not fetch, subscribe to, ingest, or calculate soft assertions.

SQLite stores soft assertions separately, with their collection, original record key, fixed score, replay timestamps, and account state. Path responses include the incoming assertion's source. The UI shows the Bluesky source for soft path assertions. Cache keys separate feature modes. Cache dependencies include the author, collection, and original record key. The existing TTL policy still applies to new edges, precedence changes, and account status changes. Each feature mode uses a separate stream cursor. Replay remains limited by the stream provider's retention. After a long outage or a disabled period, use explicit participant sync to recover current records. The all-author mode has no repository-wide recovery for nonparticipants.

CAUTION: All-author mode can greatly increase database size and stream traffic. Graph limits still apply. Large graphs return resource-limit errors rather than partial scores. NixOS and native development services expose enableSoftVouch and softVouchAllAuthors, both false by default.

Blocked lists #

With ENABLE_SOFT_VOUCH_LISTS=true, each member of a public blocked moderation list contributes -0.5 from the subscriber. The flag has no effect when ENABLE_SOFT_VOUCH=false. Positive lists and private list mutes remain outside this feature. Individual follow records already cover accounts that a user follows through a list's bulk-follow action. Explicit vouches still take precedence. Overlapping lists and individual soft blocks never multiply a pair's contribution. Blocked-list assertions apply only to default queries and topic fallback queries. Participant mode imports public listblock records and their referenced lists at bootstrap or explicit sync. All-author mode also accepts observed listblock events from nonparticipants. List owners do not need to participate. Their list definitions and memberships come from their current PDS. Snapshots fetch each owner's list collections once and reject incomplete pages or forged repository URIs. Each collection permits at most 10,000 records. A participant snapshot permits at most 100 distinct blocked lists. If an owner exceeds these limits or its PDS fails, the snapshot fails without replacing the index. Stream updates cover listblock records, list definitions, and memberships, including missing-payload hydration. The index retains referenced lists and replay tombstones after subscribers unblock them. No automatic cleanup exists. List-enabled modes have separate cache keys and stream cursors. Selected path dependencies include the subscription, list definition, and membership record. Path responses include source: "app.bsky.graph.listblock" and list_uri. The UI shows the blocked-list source. The existing TTL policy still applies to new members, new subscriptions, and account status changes. NixOS and native development services expose softVouchLists, false by default.

CAUTION: List owners can change members and therefore change subscribers' trust graphs. List subscriptions add stream traffic, database records, and PDS requests. Replay gaps still require explicit participant sync.

Index capacity #

The soft record and replay-version tables use WITHOUT ROWID to avoid duplicate primary-key storage. Schema version 8 also stores integer identity keys in soft accounts, soft records, replay versions, list records, and cache dependencies. A persistent dictionary stores each full DID once. Reads return full DIDs to services, algorithms, and public APIs. The dictionary preserves exact DIDs, including did:web values. Keys remain stable across restarts. The adapter resolves keys through SQLite inside write transactions. It does not retain a network-wide identity map in RAM. Participant configuration and explicit vouch tables retain their text keys. The migration rebuilds the high-volume tables atomically. It preserves records, replay tombstones, list provenance, and cached path dependencies. The migration needs temporary disk space for the original tables, rebuilt tables, indexes, and WAL. Before an upgrade, stop all processes that use the database. Then create a backup. SQLite reuses freed pages, but the database file does not automatically shrink after the migration. After a backup and a service stop, an operator can use SQLite VACUUM to reclaim unused pages. VACUUM also needs temporary disk space. No automatic compaction or tombstone cleanup exists.

The scale benchmark uses 100,000 follows, 1,000 authors, realistic DID lengths, live replay versions, and the production indexes. On the same fixture, database pages decrease from the original 644 to 332 bytes per edge (48%). The identity-key pass reduces the previous 497 bytes per edge to 332 (33%). Graph-load and selection allocations decrease from 162 MB to 118 MB (27%). Identity keys reduce disk use, not these allocations. The algorithm input still contains full DIDs. These figures exclude WAL files, deleted-record tombstones, cached results, and list data. Allocation totals are not peak RAM measurements. They are fixture measurements, not capacity guarantees for the Bluesky network.

Run the reproducible benchmarks:

go test ./internal/storage ./internal/trust -run '^$' -bench 'Benchmark.*Scale' -benchtime=1x -benchmem

CAUTION: All-author ingestion does not make global trust queries viable at Bluesky scale. Uncached queries still load the global graph and reject inputs that exceed the default one-million-edge limit. Traversal now stops at 10 hops, but this cap does not restrict the initial database graph load. Concurrent uncached queries can each allocate their own graph and traversal state. Trust cache rows, replay tombstones, and referenced list data still grow without automatic cleanup. The all-author mode also lacks a historical network-wide bootstrap. It indexes observed stream events, not all existing follows and blocks.

API #

Trust algorithms #

appview.New uses algorithms.Optimistic, the current bottleneck-and-hop-penalty algorithm. Use appview.NewWithAlgorithm to inject another trust.Algorithm implementation in cmd/appview/main.go. A nil algorithm uses the default implementation.

service := appview.NewWithAlgorithm(store, atproto.NewRepository(), cfg.Limits, myAlgorithm)

The interface has two methods:

Describe() trust.Descriptor
Calculate(context.Context, trust.Input) (trust.Result, error)

Describe declares the algorithm name and its account-list parameters. Calculate receives a context and an input with topic-selected edges, source DID, target DID, resolved account lists, and resource limits. Implementations and shared traversal helpers reside in internal/trust/algorithms. Shared interfaces, input types, results, and topic selection remain in internal/trust. The service handles identity resolution, topic selection, and the graph input limit before the algorithm runs. Self queries receive no edges. The algorithm returns trust.Result or an error. Public trust, authenticated trust, and trusted checks use the same injected trust algorithm. Private threshold checks remain in the service.

Alternative algorithms must preserve self score 3, direct block score -3, and unknown score null. Calculated scores must stay within [-2, 2]. Algorithms must respect context cancellation and traversal limits. Trust paths can contain at most 10 hops (trust.MaxDepth). Longer paths do not contribute. If no path within this cap exists, the built-in algorithms return unknown trust. Reputation author eligibility uses the same cap. Cache keys include the cap, so old longer paths cannot bypass it. The service rejects longer paths from custom algorithms. Custom algorithms must also honor the cap when they omit path details. Implementations must support concurrent requests. If an algorithm returns a path, its per-node scores must describe that path. The service does not enforce these algorithm-specific rules.

The algorithm variable in cmd/appview/main.go selects the active implementation. It defaults to algorithms.Optimistic{}. Change it to algorithms.ShortestPath{} to select ShortestPath, then restart the service. The HTTP layer uses algorithm metadata to validate trust options and generate /v1/openapi.json at startup. The schema includes endpoint parameters, response types, request bodies, and authentication requirements.

ShortestPath selects the path with the fewest hops after it removes ignored accounts and their edges. Among paths with the same hop count, it selects the highest score with the existing hop penalty. Only ShortestPath accepts the repeated ignored query parameter. Public reputation, authenticated reputation, trusted checks, and the UI reputation API support this parameter when ShortestPath is active. Optimistic rejects it, including an empty value, with HTTP 400. Ignored lists accept at most 100 entries. They cannot contain the source or target, including aliases for those identities. An unresolved ignored handle rejects the query. The service normalizes all query handles, resolves them under one five-second deadline, and removes duplicate DIDs. Algorithms receive only DIDs and never resolve identities themselves.

Graph scenario tests #

internal/trust/trusttest provides a graph harness without databases or network access. Scenarios use named accounts, directed edges, expected outcomes, and optional account lists or resource limits. The harness converts names to fixture DIDs. It preserves explicit DIDs. It checks status, score, path, incoming assertions, optional prefix scores, and input immutability.

Add scenarios to internal/trust/algorithms/scenarios_test.go, or use the harness in a new test:

trusttest.Run(t, algorithms.Optimistic{}, []trusttest.Scenario{{
    Name: "friend of a friend",
    Source: "alice",
    Target: "bob",
    Edges: []trusttest.Edge{
        {From: "alice", To: "carol", Score: 2},
        {From: "carol", To: "bob", Score: 1},
    },
    Want: trusttest.Outcome{
        Status: trust.CalculatedStatus,
        Score: .9,
        Path: []string{"alice", "carol", "bob"},
        Scores: []float64{3, 2, .9},
    },
}})

Scenario.Input() builds a fresh graph input for direct experiments. Accounts uses algorithm parameter names, such as algorithms.IgnoredParameter for ShortestPath. Limits overrides the default limits and hop penalty. Context supports cancellation scenarios. Outcome.Error accepts an expected error, such as trust.ErrLimit.

Both algorithms run the common suite, including chains, diamonds, cycles, negative scores, blocks, disconnected accounts, and equal-score ties. The suite repeats these scenarios with reversed edges to check stable results. Separate scenarios show competing-path differences and ShortestPath account exclusions.

Run the suite:

go test ./internal/trust/...
go test ./internal/trust/algorithms -run 'TestCommonScenarios/ShortestPath' -v

Relative reputation #

CAUTION: This release changes the endpoint names and response meanings. Use /v1/trust and /v1/me/trust for the previous path-based calculation. The Check trust tab retains path-based trust and path details. The additional Check reputation tab shows absolute_score as the signed sum and mean_score as the arithmetic mean. Reputation uses your private threshold for author trust. Topic options apply to both assertions and author trust paths. Zero assertions count in the mean. Empty sets return zero, which does not prove consensus. The /v1/reputation and /v1/me/reputation endpoints return {"score": 4, "absolute_score": 4, "mean_score": 2}.

DirectSum sums each target vouch whose author has source-relative trust at least equal to the threshold. Negative vouches subtract from the sum. Blocks do not contribute. Unknown or blocked authors never qualify. The source can contribute its own direct vouch, because its self trust is 3. absolute_score is the signed sum, not its mathematical absolute value. score remains an alias for existing clients. mean_score is the sum divided by the number of qualifying non-block vouches. Zero scores count in this divisor. An empty set returns 0 for both scores. The sum has no score clamp and no single path. Topic selection applies to both incoming vouches and author trust paths. Fallback selects one vouch per author-target pair, including explicit zero scores. With ShortestPath, ignored accounts cannot act as authorities or intermediaries.

Public queries default to threshold 1. They never use or disclose the source's private threshold. Authenticated queries default to the authenticated user's stored threshold. Both endpoints accept threshold to override the default, within [-2, 2]. They accept handles or DIDs and reject include_path, because an aggregate has no single path.

internal/reputation defines a separate algorithm interface. The existing trust.Algorithm interface remains unchanged. Reputation input supplies direct vouches, a threshold, and a cached trust reader rather than raw storage access. appview.Options independently selects the trust algorithm, reputation algorithm, and cache TTL. cmd/appview/main.go selects DirectSum by default. The OpenAPI document identifies both active algorithms and leaves the reputation score unbounded.

internal/reputation/reputationtest provides named graph scenarios for algorithm experiments. The suite covers both trust algorithms, thresholds, positive and negative sums, blocks, unknown authors, cycles, cancellation, and limits.

go test ./internal/reputation/...
curl --get 'http://127.0.0.1:8001/v1/reputation' \
  --data-urlencode 'source=alice.bsky.social' \
  --data-urlencode 'target=bob.bsky.social' \
  --data-urlencode 'threshold=1.5'

Trust cache #

SQLite stores full trust results, including unknown and self results, paths, assertions, scores, and comments. Cache keys include resolved DIDs, algorithm metadata, topic mode, tag, traversal limits, and normalized account options. Private thresholds do not affect cache keys. Reputation sums reuse cached author trust but do not cache the sum itself. Warm trust reads require no graph scan. Cold aggregate queries share one lazy graph snapshot across author calculations.

The system-wide TTL defaults to 24 hours. PROVENANCE_TRUST_CACHE_TTL accepts a positive duration, such as 24h or 30m. NixOS and native service modules expose trustCacheTTL. A changed TTL applies to newly calculated entries. Reads reject entries at their expiry time. Expired entries remain on disk until a calculation replaces them. The expiry index supports future range-based cleanup without a full cache scan.

Record updates and deletions invalidate only caches whose selected paths use that author's exact record key. Snapshots preserve caches for unchanged records. Dependency rows have an indexed author-record lookup and cascade with cache deletion. Revisions prevent a calculation from publishing a stale cache entry after a concurrent graph mutation. Repeated concurrent mutations produce a resource-limit error rather than a partial or stale result.

New paths, new blocks, unrelated updates, and account status changes do not invalidate existing paths. Unknown results also remain cached until expiry because they have no path dependencies. These rules can delay new information for the full TTL. A record removal still invalidates paths that use that record. If an algorithm's calculation rules change, change its descriptor name to prevent reuse of old cache entries.

With ShortestPath active:

curl --get 'http://127.0.0.1:8001/v1/trust' \
  --data-urlencode 'source=alice.bsky.social' \
  --data-urlencode 'target=bob.bsky.social' \
  --data-urlencode 'ignored=carol.bsky.social' \
  --data-urlencode 'ignored=did:plc:z5jscut23oj6ukqvgjqvkogb' \
  --data-urlencode 'include_path=true'

Public appview endpoints:

• GET /healthz
• GET /v1/openapi.json
• GET /v1/trust?source=DID_OR_HANDLE&target=DID_OR_HANDLE&mode=default
• GET /v1/reputation?source=DID_OR_HANDLE&target=DID_OR_HANDLE&threshold=1
• GET /v1/tags?prefix=work
• GET /v1/vouches?author=DID&offset=0

The public trust endpoint accepts a DID or full account handle for each account, without authentication. Handles resolve to DIDs before the service calculates trust from the local index. The endpoint does not add accounts to the index or fetch their vouches. It returns a score and status, but no private threshold or trusted flag. Use include_path=true to include the path and per-node scores. Use tag and mode for topic-specific queries, as with DID queries.

curl --get 'http://127.0.0.1:8001/v1/trust' \
  --data-urlencode 'source=alice.bsky.social' \
  --data-urlencode 'target=did:plc:z5jscut23oj6ukqvgjqvkogb' \
  --data-urlencode 'include_path=true'

Authenticated appview endpoints:

• POST /v1/participants: opt in the authenticated identity.
• POST /v1/sync: explicitly recover the authenticated identity's records through a PDS snapshot.
• GET /v1/me: read the authenticated identity's private threshold.
• PUT /v1/me/threshold: set {"threshold": 1.2}.
• GET /v1/me/trust?target=DID_OR_HANDLE: calculate relative trust with the private threshold.
• GET /v1/me/reputation?target=DID_OR_HANDLE: sum assertions from authors who meet the private trust threshold.
• GET /v1/trusted?target=DID_OR_HANDLE: return only {"trusted": true} or {"trusted": false}.

Internal authenticated requests require both headers:

Authorization: Bearer <PROVENANCE_SERVICE_KEY>
X-Provenance-DID: <authenticated DID>

The UI derives the DID from its authenticated session. The service key grants access to every participant's private configuration. Do not expose that key to browsers or untrusted clients. This release does not provide third-party API credentials.

Trust and reputation queries accept mode=default, mode=tag, or mode=fallback. The last two modes require tag. Fallback applies to each author-target pair and preserves explicit zero scores and blocks. Tags are case-sensitive and contain at most 128 Unicode code points. For trust queries, add include_path=true to request a path for an untrusted result. Public trust queries omit paths unless the caller requests them. When a response includes path, it also includes path_scores in the same order:

[
  {"did": "did:plc:source", "assertion": null, "score": 3},
  {"did": "did:plc:intermediate", "assertion": 2, "score": 2, "comment": "Reference text from the source."},
  {"did": "did:plc:target", "assertion": 1, "score": 0.9}
]

assertion is the incoming vouch score from the previous node. score is the cumulative score along this path, with the hop penalty and clamp. It is not an independent best-path calculation for that node. The source has no incoming assertion and uses the self score of 3. Direct blocks retain score -3. Unknown results contain neither path field.

Vouch records accept an optional comment string for public reference text. Comments contain at most 300 Unicode code points. The authenticated PUT /api/vouch endpoint accepts this field with subject, tag, and score. Each path_scores node includes the comment from its incoming vouch, when that comment is not empty. The source node has no incoming comment. Existing records without comments remain valid. Comments do not affect scores, path selection, tags, or record keys. Clients must treat comments as untrusted text, not HTML.

The score equals the maximum path bottleneck minus its additional-hop penalty, clamped to [-2, 2]. A direct block by the querying user returns -3 and overrides all paths. Other blocks participate in graph calculations as -3. Self queries return 3. Unknown queries return a null score. The default private threshold is 1.

Tests and SQL #

9 mk generate
9 mk test
go vet ./...

The tests cover graph semantics, tag fallback, SQLite transactions, OAuth state protection, and authenticated HTTP behavior. Stream tests cover replay, reconnects, record deletion, account lifecycle events, bootstrap races, and schema upgrades.

Nix packages #

Local Nix commands use Git-tracked files. An ignored, untracked .env does not enter the flake source. If you create a source file, run git add before a Nix build.

CAUTION: Do not use path:. with secrets in the repository. That source mode copies ignored files before flake filters run.

Build both services:

nix build .

Build one service:

nix build .#appview
nix build .#ui

The default package contains bin/appview and bin/ui. The individual packages contain only their service binary. The flake exposes Linux packages for x86-64 and ARM64, plus Darwin packages for ARM64. The pinned unstable Nixpkgs release does not support x86-64 Darwin. Use the environment variables from Local development to start the packaged binaries. For example, run ./result/bin/ui after the UI package build.

Run package, module, and NixOS service tests:

nix flake check .

The Linux service test starts both services with runtime credentials and tests database permissions and service restarts. It does not contact public Microcosm instances or perform account login.

Native development services #

Start both services with one command:

nix run .#services

Open http://127.0.0.1:8000. Use the process-compose interface to inspect logs and stop the services. For a terminal without the interface, use:

nix run .#services -- -t=false

Press Ctrl-C to stop the services. The launcher starts the appview first and waits for its HTTP readiness probe before the UI starts. Both services bind to 127.0.0.1 by default. The default public origin is http://127.0.0.1:8000. To bind both services to all IPv4 interfaces, use:

PROVENANCE_DEV_BIND_HOST=0.0.0.0 nix run .#services

PROVENANCE_DEV_BIND_HOST accepts 127.0.0.1 or 0.0.0.0. It changes only the listen addresses, not the public origin or internal appview URL. Readiness probes still use loopback. The override exposes both services to the network. Restrict access with your firewall. The launcher does not require Docker, Podman, or root access. It uses services-flake and process-compose-flake without a flake-parts conversion.

The first start creates keys and databases under ./data/provenance-dev. Later starts retain those files. Keys remain outside the Nix store. The directory uses mode 0700. Keys and databases use mode 0600. The services run as your current user, without container isolation. The launcher disables the process-compose HTTP control server and automatic dotenv imports.

To use another state directory, set PROVENANCE_DEV_DATA_DIR:

PROVENANCE_DEV_DATA_DIR="$HOME/.local/state/provenance-dev" nix run .#services

The launcher manages keys, database paths, ports, and the local appview URL. Use PROVENANCE_DEV_BIND_HOST to select the bind host. The launcher ignores direct overrides for PROVENANCE_APPVIEW_ADDR and PROVENANCE_UI_ADDR. PROVENANCE_PUBLIC_ORIGIN, PROVENANCE_JETSTREAM_URL, PROVENANCE_SLINGSHOT_URL, and PROVENANCE_HOP_PENALTY still accept environment overrides. Public Microcosm instances remain the defaults. Account login requires a real PDS. Vouches remain public. Use a dedicated test account.

The flake exports processComposeModules.provenance for other process-compose configurations. Import that module with the default module from services-flake. Enable services.provenance.<name> and configure package, dataDir, appviewPort, or uiPort. Each instance needs separate ports and a separate data directory. Leave PROVENANCE_DEV_DATA_DIR unset for a configuration with multiple instances.

nix flake check . includes a native launcher check for HTTP readiness and private file permissions. The check uses temporary state and does not perform account login or publish vouches.

Git hooks #

Enter the development shell to install the Git pre-commit hook:

nix develop .

The hook configuration uses cachix/git-hooks.nix with prek, not the Python pre-commit runner. The shell creates an ignored .pre-commit-config.yaml symlink to the generated Nix configuration. The hook uses Nix-store tool paths, so it also works outside the shell.

Commit checks cover Go/Nix formatting, JSON/YAML syntax, merge conflicts, private keys, final newlines, and trailing whitespace. The sqlc check rejects generated SQL adapters that differ from the schema and queries. Formatting hooks can change files. Review those changes before you stage them again.

Run all commit checks:

nix fmt

Alternatively, run prek run --all-files inside the development shell. The same checks run in the Nix sandbox through nix flake check .. Go package checks use the manual stage because they need a writable module cache:

nix develop . --command prek run --all-files --hook-stage manual

The manual stage runs go vet ./... and go test -race ./.... It can download Go modules on the first run. The package build still runs the Go tests offline in the Nix sandbox.

NixOS deployment #

Add Provenance to your system flake inputs. For a local test, use a repository path:

inputs.provenance.url = "path:/home/noah/repos/provenance";

Import inputs.provenance.nixosModules.default into the host's NixOS module list. Use nixosModules.appview or nixosModules.ui for a host with one service.

Create secret files outside the Nix store before you start the services:

sudo install -d -m 0700 /var/lib/provenance-secrets
sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /var/lib/provenance-secrets/service-key'
sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /var/lib/provenance-secrets/encryption-key'

Do not regenerate keys on each deployment. The UI encryption key protects stored sessions. Keep it with the UI database backup. The UI and appview require the same service key. An existing secret manager can supply the files instead. Use quoted absolute path strings, not Nix path literals or builtins.readFile, for secret options.

Configure both services in a host module:

{ inputs, pkgs, ... }:
{
  services.provenance = {
    appview = {
      enable = true;
      package = inputs.provenance.packages.${pkgs.stdenv.hostPlatform.system}.appview;
      serviceKeyFile = "/var/lib/provenance-secrets/service-key";
      listenAddress = "127.0.0.1:8001";
      jetstreamURL = "wss://jetstream.fire.hose.cam/subscribe";
      slingshotURL = "https://slingshot.microcosm.blue";
      hopPenalty = 0.1;
      trustCacheTTL = "24h";
    };
    ui = {
      enable = true;
      package = inputs.provenance.packages.${pkgs.stdenv.hostPlatform.system}.ui;
      serviceKeyFile = "/var/lib/provenance-secrets/service-key";
      encryptionKeyFile = "/var/lib/provenance-secrets/encryption-key";
      listenAddress = "127.0.0.1:8000";
      publicOrigin = "https://provenance.example.com";
      appviewURL = "http://127.0.0.1:8001";
    };
  };
}

Pass inputs through your system flake's specialArgs for this example. Explicit package options use the Provenance flake's package set. Without these options, each module builds its package with the host's pkgs.

For a local browser test, set ui.publicOrigin = "http://127.0.0.1:8000". For public deployment, place an HTTPS reverse proxy before the UI. Set publicOrigin to the browser-visible origin, not the internal listen address. The modules do not configure a proxy or open firewall ports. Keep the appview on loopback unless a remote UI requires access. Use HTTPS for a remote appviewURL because requests include the shared service key. If you change the appview listen address, also change ui.appviewURL.

Both services expose stateDirectory and databaseFile options. The defaults store databases in /var/lib/provenance-appview/appview.db and /var/lib/provenance-ui/ui.db. stateDirectory is a directory name, not an absolute path. databaseFile is a filename within that directory. Use separate state directories for the services.

Systemd provides dynamic service users, private state directories, restart policies, and filesystem restrictions. Systemd reads secret files as the service manager and passes private credential copies to each service. The units contain secret paths, not secret contents. Application configuration checks reject invalid URLs and key contents at startup.

Inspect the services:

systemctl status provenance-appview provenance-ui
journalctl -u provenance-appview -u provenance-ui

Microcosm indexing #

The default public services come from the Microcosm ecosystem. Configuration can select other public instances or self-hosted instances:

export PROVENANCE_JETSTREAM_URL=wss://jetstream2.fr.hose.cam/subscribe
export PROVENANCE_SLINGSHOT_URL=https://records.example.com

For local instances, use loopback ws:// and http:// URLs. Other instances require wss:// and https://. The endpoints are operator-controlled and can point to private self-hosted networks. Do not accept endpoint configuration from untrusted clients.

The stream uses Jetstream's JSON time_us protocol and filters the vouch collection. The appview checks participation before record lookup or index updates. Valid event payloads need no extra record lookup. For missing payloads, Slingshot must return the event's exact record URI and CID. Unavailable hydration stops cursor advancement for that event.

Each event update and its cursor share one SQLite transaction. Reconnects resume from the saved cursor with a five-second overlap. Record versions and deletion tombstones prevent duplicate or older events from restoring obsolete assertions. Inactive accounts disappear from public graph queries and vouch lists. Reactivation restores retained records. Account deletion removes the account's authored records. Malformed assertion updates remove the previous assertion, not the stream cursor.

Initial backfill and explicit recovery still read the PDS. Slingshot has no listRecords endpoint. Constellation exposes backlinks, not complete outgoing record lists. Spacedust's current documentation lists missing replay and delete events, so it cannot maintain this index alone.

The stream provider defines its replay retention. The legacy Jetstream protocol does not report a reliable replay gap to this client. If an outage exceeds retention, invoke POST /v1/sync for each affected participant. Recovery replaces that participant's index and preserves the private threshold. The appview never schedules periodic PDS recovery. Bootstrap watermarks assume that the appview clock and stream timestamps are close to UTC. Keep the host clock synchronized.

Lexicon tools #

The development shell includes ATProto goat v0.2.5, not the unrelated ASCII-art tool with the same name.

nix develop .
9 mk lex-check

To create another record schema, choose a new NSID:

9 mk lex-new NSID=computer.ngp.provenance.example

This command creates a template. Edit its fields before publication. It refuses to replace an existing schema file. The vouch schema already exists at lexicons/computer/ngp/provenance/vouch.json.

To publish the vouch schema:

1. Choose the ATProto account that will own the schema.

2. Add this DNS TXT record, with that account's DID:

   _lexicon.provenance.ngp.computer. TXT "did=did:plc:YOUR_SCHEMA_ACCOUNT"
   

3. Run goat lex check-dns after DNS propagation.

4. Supply GOAT_USERNAME and GOAT_PASSWORD through your secret manager. Use an app password for the schema account.

5. Run 9 mk lex-publish.

6. Retrieve the published record:

   goat get at://YOUR_SCHEMA_ACCOUNT_DID/com.atproto.lexicon.schema/computer.ngp.provenance.vouch
   

The publish command retains the DNS authority check and publishes only the vouch schema. It does not update an existing schema by default. Before an update, run goat lex breaking and review the changes. Then use goat lex publish --update lexicons/computer/ngp/provenance/vouch.json. Do not use --skip-dns-check for public publication. Keep passwords out of command arguments and repository files. goat account login stores credentials in plaintext. Prefer environment credentials for publication. Provenance does not publish schemas during startup or builds.

See the ATProto publication guide for the DNS authority and record format.

Deployment limits #

This project is an initial implementation, not an audited production service. The vouch collection is computer.ngp.provenance.vouch, under the provenance.ngp.computer namespace authority. After an upgrade from the provisional collection, log out and log in to grant the new collection permissions. Old records do not move automatically. Republish vouches in the new collection. Existing appview databases can still contain old assertions until an explicit recovery replaces each participant's snapshot. Invoke POST /v1/sync for each affected participant after the namespace change. Keep the old records and database backups until you complete the transition.

The UI uses a public OAuth client with encrypted persistent sessions. Refresh tokens have server-defined lifetimes. Browser sessions expire after 14 days. Repository operations serialize within one UI process to prevent refresh races. Do not run multiple UI replicas against the same session database.

Jetstream is a trusted JSON source, not a repository-signature verification layer. The index can be stale during source outages or stream lag. The health endpoint reports HTTP availability, not stream freshness. Failed or incomplete recovery snapshots preserve the previous index. Each bootstrap snapshot supports at most 10,000 vouches. The pending-bootstrap pass supports at most 10,000 participants. The version ledger retains record tombstones. Monitor database growth. Do not run multiple appview ingestors against one database. Trust queries permit at most 100,000 states and 1,000,000 edge visits. Resource-limit errors never return a partial score.

For public deployment, use HTTPS and a reverse proxy with request limits and rate limits. Restrict the appview's authenticated endpoints to the UI service network. Do not use a proxy that bypasses the ATProto adapter's SSRF protection. Back up both databases and the encryption key. The databases use private file permissions and SQLite WAL mode. The initial migration uses a transaction and records its schema version. The services reject databases with newer schema versions.

The design describes the service boundaries and trust rules. The security review lists trust assumptions and operational risks. The lexicon describes public vouch records. The tsk queue records the implementation roadmap.