diff --git a/.gitignore b/.gitignore index eb5342e..529c34d 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ *.db-shm *.db-wal .env +/.pre-commit-config.yaml diff --git a/README.md b/README.md index 0d5041c..d6be4c9 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,205 @@ go vet ./... The tests cover graph semantics, tag fallback, SQLite transactions, OAuth state protection, and authenticated HTTP behavior. Stream tests cover replay, reconnects, record deletion, account lifecycle events, bootstrap races, and schema upgrades. +## Nix packages + +Build both services: + +```sh +nix build path:. +``` + +Build one service: + +```sh +nix build path:.#appview +nix build path:.#ui +``` + +The default package contains `bin/appview` and `bin/ui`. +The individual packages contain only their service binary. +The flake exposes Linux packages for x86-64 and ARM64, plus Darwin packages for ARM64. +The pinned unstable Nixpkgs release does not support x86-64 Darwin. +Use the environment variables from [Local development](#local-development) to start the packaged binaries. +For example, run `./result/bin/ui` after the UI package build. + +Run package, module, and NixOS service tests: + +```sh +nix flake check path:. +``` + +The Linux service test starts both services with runtime credentials and tests database permissions and service restarts. +It does not contact public Microcosm instances or perform account login. + +## Native development services + +Start both services with one command: + +```sh +nix run path:.#services +``` + +Open `http://127.0.0.1:8080`. +Use the process-compose interface to inspect logs and stop the services. +For a terminal without the interface, use: + +```sh +nix run path:.#services -- -t=false +``` + +Press `Ctrl-C` to stop the services. +The launcher starts the appview first and waits for its HTTP readiness probe before the UI starts. +Both services listen only on loopback. +The launcher does not require Docker, Podman, or root access. +It uses `services-flake` and `process-compose-flake` without a flake-parts conversion. + +The first start creates keys and databases under `./data/provenance-dev`. +Later starts retain those files. Keys remain outside the Nix store. +The directory uses mode `0700`. Keys and databases use mode `0600`. +The services run as your current user, without container isolation. +The launcher disables the process-compose HTTP control server and automatic dotenv imports. + +To use another state directory, set `PROVENANCE_DEV_DATA_DIR`: + +```sh +PROVENANCE_DEV_DATA_DIR="$HOME/.local/state/provenance-dev" nix run path:.#services +``` + +The launcher manages keys, database paths, listen addresses, and the local appview URL. +It ignores environment overrides for those values. +`PROVENANCE_PUBLIC_ORIGIN`, `PROVENANCE_JETSTREAM_URL`, `PROVENANCE_SLINGSHOT_URL`, and `PROVENANCE_HOP_PENALTY` still accept environment overrides. +Public Microcosm instances remain the defaults. +Account login requires a real PDS. Vouches remain public. +Use a dedicated test account. + +The flake exports `processComposeModules.provenance` for other process-compose configurations. +Import that module with the default module from `services-flake`. +Enable `services.provenance.` and configure `package`, `dataDir`, `appviewPort`, or `uiPort`. +Each instance needs separate ports and a separate data directory. +Leave `PROVENANCE_DEV_DATA_DIR` unset for a configuration with multiple instances. + +`nix flake check path:.` includes a native launcher check for HTTP readiness and private file permissions. +The check uses temporary state and does not perform account login or publish vouches. + +## Git hooks + +Enter the development shell to install the Git pre-commit hook: + +```sh +nix develop path:. +``` + +The hook configuration uses `cachix/git-hooks.nix` with `prek`, not the Python pre-commit runner. +The shell creates an ignored `.pre-commit-config.yaml` symlink to the generated Nix configuration. +The hook uses Nix-store tool paths, so it also works outside the shell. + +Commit checks cover Go/Nix formatting, JSON/YAML syntax, merge conflicts, private keys, final newlines, and trailing whitespace. +The sqlc check rejects generated SQL adapters that differ from the schema and queries. +Formatting hooks can change files. Review those changes before you stage them again. + +Run all commit checks: + +```sh +nix fmt +``` + +Alternatively, run `prek run --all-files` inside the development shell. +The same checks run in the Nix sandbox through `nix flake check path:.`. +Go package checks use the manual stage because they need a writable module cache: + +```sh +nix develop path:. --command prek run --all-files --hook-stage manual +``` + +The manual stage runs `go vet ./...` and `go test -race ./...`. +It can download Go modules on the first run. +The package build still runs the Go tests offline in the Nix sandbox. + +## NixOS deployment + +Add Provenance to your system flake inputs. +For a local test, use a repository path: + +```nix +inputs.provenance.url = "path:/home/noah/repos/provenance"; +``` + +Import `inputs.provenance.nixosModules.default` into the host's NixOS module list. +Use `nixosModules.appview` or `nixosModules.ui` for a host with one service. + +Create secret files outside the Nix store before you start the services: + +```sh +sudo install -d -m 0700 /var/lib/provenance-secrets +sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /var/lib/provenance-secrets/service-key' +sudo sh -c 'umask 077; head -c 32 /dev/urandom | base64 > /var/lib/provenance-secrets/encryption-key' +``` + +Do not regenerate keys on each deployment. +The UI encryption key protects stored sessions. Keep it with the UI database backup. +The UI and appview require the same service key. +An existing secret manager can supply the files instead. +Use quoted absolute path strings, not Nix path literals or `builtins.readFile`, for secret options. + +Configure both services in a host module: + +```nix +{ inputs, pkgs, ... }: +{ + services.provenance = { + appview = { + enable = true; + package = inputs.provenance.packages.${pkgs.stdenv.hostPlatform.system}.appview; + serviceKeyFile = "/var/lib/provenance-secrets/service-key"; + listenAddress = "127.0.0.1:8081"; + jetstreamURL = "wss://jetstream.fire.hose.cam/subscribe"; + slingshotURL = "https://slingshot.microcosm.blue"; + hopPenalty = 0.1; + }; + ui = { + enable = true; + package = inputs.provenance.packages.${pkgs.stdenv.hostPlatform.system}.ui; + serviceKeyFile = "/var/lib/provenance-secrets/service-key"; + encryptionKeyFile = "/var/lib/provenance-secrets/encryption-key"; + listenAddress = "127.0.0.1:8080"; + publicOrigin = "https://provenance.example.com"; + appviewURL = "http://127.0.0.1:8081"; + }; + }; +} +``` + +Pass `inputs` through your system flake's `specialArgs` for this example. +Explicit package options use the Provenance flake's package set. +Without these options, each module builds its package with the host's `pkgs`. + +For a local browser test, set `ui.publicOrigin = "http://127.0.0.1:8080"`. +For public deployment, place an HTTPS reverse proxy before the UI. +Set `publicOrigin` to the browser-visible origin, not the internal listen address. +The modules do not configure a proxy or open firewall ports. +Keep the appview on loopback unless a remote UI requires access. +Use HTTPS for a remote `appviewURL` because requests include the shared service key. +If you change the appview listen address, also change `ui.appviewURL`. + +Both services expose `stateDirectory` and `databaseFile` options. +The defaults store databases in `/var/lib/provenance-appview/appview.db` and `/var/lib/provenance-ui/ui.db`. +`stateDirectory` is a directory name, not an absolute path. +`databaseFile` is a filename within that directory. +Use separate state directories for the services. + +Systemd provides dynamic service users, private state directories, restart policies, and filesystem restrictions. +Systemd reads secret files as the service manager and passes private credential copies to each service. +The units contain secret paths, not secret contents. +Application configuration checks reject invalid URLs and key contents at startup. + +Inspect the services: + +```sh +systemctl status provenance-appview provenance-ui +journalctl -u provenance-appview -u provenance-ui +``` + ## Microcosm indexing The default public services come from the [Microcosm ecosystem](https://microcosm.blue/). diff --git a/flake.lock b/flake.lock index 0b43205..989f17a 100644 --- a/flake.lock +++ b/flake.lock @@ -1,5 +1,42 @@ { "nodes": { + "flake-compat": { + "flake": false, + "locked": { + "lastModified": 1767039857, + "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=", + "owner": "NixOS", + "repo": "flake-compat", + "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab", + "type": "github" + }, + "original": { + "owner": "NixOS", + "repo": "flake-compat", + "type": "github" + } + }, + "git-hooks": { + "inputs": { + "flake-compat": "flake-compat", + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1790500375, + "narHash": "sha256-XN3sDtn8TU9hAc9xqZ+SqRB/HHm2wjxM6aSWYLJU+oo=", + "owner": "cachix", + "repo": "git-hooks.nix", + "rev": "a0e4241b51206fbcbf52fd322eb5f0cd80f153c4", + "type": "github" + }, + "original": { + "owner": "cachix", + "repo": "git-hooks.nix", + "type": "github" + } + }, "nixpkgs": { "locked": { "lastModified": 1791260994, @@ -16,9 +53,43 @@ "type": "github" } }, + "process-compose-flake": { + "locked": { + "lastModified": 1791313691, + "narHash": "sha256-9Bp/38NrvbjUGZAZTUiYWfbXW/y6B1rSBJHEgRsDK4g=", + "owner": "Platonic-Systems", + "repo": "process-compose-flake", + "rev": "142d4b084026e91d9ab5822639d308aca5fb1b2b", + "type": "github" + }, + "original": { + "owner": "Platonic-Systems", + "repo": "process-compose-flake", + "type": "github" + } + }, "root": { "inputs": { - "nixpkgs": "nixpkgs" + "git-hooks": "git-hooks", + "nixpkgs": "nixpkgs", + "process-compose-flake": "process-compose-flake", + "services-flake": "services-flake" + } + }, + "services-flake": { + "locked": { + "lastModified": 1791367800, + "narHash": "sha256-vUC7++qixRr0JAULJjE7bVDO5Ci23WW/oBcUSv0evVo=", + "owner": "juspay", + "repo": "services-flake", + "rev": "03189b6f0ef5083ffce220e912e13e8cd4907108", + "type": "github" + }, + "original": { + "owner": "juspay", + "ref": "0.5.0", + "repo": "services-flake", + "type": "github" } } }, diff --git a/flake.nix b/flake.nix index dd62071..c7b7224 100644 --- a/flake.nix +++ b/flake.nix @@ -1,18 +1,141 @@ { - description = "Provenance development environment"; + description = "Provenance services, NixOS modules, and development environment"; inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + inputs.services-flake.url = "github:juspay/services-flake/0.5.0"; + inputs.process-compose-flake.url = "github:Platonic-Systems/process-compose-flake"; + inputs.git-hooks = { + url = "github:cachix/git-hooks.nix"; + inputs.nixpkgs.follows = "nixpkgs"; + }; - outputs = { nixpkgs, ... }: + outputs = + { + self, + nixpkgs, + services-flake, + process-compose-flake, + git-hooks, + ... + }: let - systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ]; - in { - devShells = nixpkgs.lib.genAttrs systems (system: - let pkgs = import nixpkgs { inherit system; }; - in { + systems = [ + "x86_64-linux" + "aarch64-linux" + "aarch64-darwin" + ]; + forSystems = nixpkgs.lib.genAttrs systems; + devServices = + system: + import ./nix/dev-services.nix { + pkgs = import nixpkgs { inherit system; }; + package = self.packages.${system}.provenance; + inherit services-flake process-compose-flake; + }; + hooks = + system: + git-hooks.lib.${system}.run ( + import ./nix/hooks.nix { + pkgs = import nixpkgs { inherit system; }; + } + ); + in + { + packages = forSystems ( + system: + let + pkgs = import nixpkgs { inherit system; }; + provenance = pkgs.callPackage ./nix/package.nix { }; + in + { + default = provenance; + inherit provenance; + appview = provenance.override { + pname = "provenance-appview"; + subPackages = [ "cmd/appview" ]; + }; + ui = provenance.override { + pname = "provenance-ui"; + subPackages = [ "cmd/ui" ]; + }; + services = (devServices system).outputs.package; + } + ); + + apps = forSystems (system: { + services = { + type = "app"; + meta.description = "Start the native Provenance development services"; + program = "${self.packages.${system}.services}/bin/provenance-services"; + }; + }); + + processComposeModules.provenance = services-flake.lib.multiService ./nix/services/provenance.nix; + + nixosModules = { + appview = import ./nix/modules/appview.nix; + ui = import ./nix/modules/ui.nix; + default = { + imports = [ + self.nixosModules.appview + self.nixosModules.ui + ]; + }; + }; + + checks = forSystems ( + system: + let + pkgs = import nixpkgs { inherit system; }; + in + { + package = self.packages.${system}.default; + dev-services = (devServices system).outputs.check; + pre-commit = hooks system; + } + // nixpkgs.lib.optionalAttrs pkgs.stdenv.hostPlatform.isLinux { + modules = import ./nix/tests/modules.nix { + inherit + self + nixpkgs + pkgs + system + ; + }; + services = pkgs.testers.runNixOSTest (import ./nix/tests/services.nix { inherit self system; }); + } + ); + + devShells = forSystems ( + system: + let + pkgs = import nixpkgs { inherit system; }; + in + { default = pkgs.mkShell { - packages = [ pkgs.go pkgs.sqlc pkgs.sqlite ]; + packages = [ + pkgs.go + pkgs.sqlc + pkgs.sqlite + ] + ++ (hooks system).enabledPackages; + shellHook = (hooks system).shellHook; }; - }); + } + ); + + formatter = forSystems ( + system: + let + pkgs = import nixpkgs { inherit system; }; + config = (hooks system).config; + in + pkgs.writeShellApplication { + name = "provenance-hooks"; + text = '' + exec ${pkgs.lib.getExe config.package} run --all-files --config ${config.configFile} "$@" + ''; + } + ); }; } diff --git a/nix/dev-services.nix b/nix/dev-services.nix new file mode 100644 index 0000000..5739850 --- /dev/null +++ b/nix/dev-services.nix @@ -0,0 +1,49 @@ +{ + pkgs, + package, + services-flake, + process-compose-flake, +}: +let + compose = import process-compose-flake.lib { inherit pkgs; }; +in +(compose.evalModules { + name = "provenance-services"; + modules = [ + services-flake.processComposeModules.default + (services-flake.lib.multiService ./services/provenance.nix) + { + services.provenance.provenance = { + enable = true; + inherit package; + }; + + # No unauthenticated process-control listener or implicit dotenv secrets. + cli.options.no-server = true; + cli.environment.PC_DISABLE_DOTENV = true; + cli.options.ordered-shutdown = true; + cli.preHook = '' + export PROVENANCE_DEV_DATA_DIR="''${PROVENANCE_DEV_DATA_DIR:-./data/provenance-dev}" + ''; + + settings.processes.test = { + command = pkgs.writeShellApplication { + name = "provenance-services-test"; + runtimeInputs = [ + pkgs.curl + pkgs.coreutils + ]; + text = '' + curl --fail http://127.0.0.1:8081/healthz + curl --fail http://127.0.0.1:8080/oauth-client-metadata.json + test "$(stat -c %a "$PROVENANCE_DEV_DATA_DIR/appview.db")" = 600 + test "$(stat -c %a "$PROVENANCE_DEV_DATA_DIR/ui.db")" = 600 + test "$(stat -c %a "$PROVENANCE_DEV_DATA_DIR/service-key")" = 600 + test "$(stat -c %a "$PROVENANCE_DEV_DATA_DIR/encryption-key")" = 600 + ''; + }; + depends_on.provenance-ui.condition = "process_healthy"; + }; + } + ]; +}).config diff --git a/nix/hooks.nix b/nix/hooks.nix new file mode 100644 index 0000000..eae089b --- /dev/null +++ b/nix/hooks.nix @@ -0,0 +1,49 @@ +{ pkgs }: +{ + src = pkgs.lib.cleanSource ../.; + package = pkgs.prek; + default_stages = [ "pre-commit" ]; + + hooks = { + gofmt = { + enable = true; + package = pkgs.go; + }; + nixfmt = { + enable = true; + package = pkgs.nixfmt; + }; + check-json.enable = true; + check-yaml.enable = true; + check-merge-conflicts.enable = true; + detect-private-keys.enable = true; + end-of-file-fixer.enable = true; + trim-trailing-whitespace.enable = true; + + sqlc-diff = { + enable = true; + name = "Check generated SQL adapter"; + entry = "${pkgs.sqlc}/bin/sqlc diff"; + files = "^(sqlc\\.yaml|internal/storage/.*\\.(sql|go))$"; + pass_filenames = false; + }; + + # Go checks need a writable module cache. Run them explicitly, outside the Nix sandbox. + go-vet = { + enable = true; + name = "Go vet"; + entry = "${pkgs.go}/bin/go vet ./..."; + files = "(\\.go$|^go\\.(mod|sum)$)"; + pass_filenames = false; + stages = [ "manual" ]; + }; + go-race = { + enable = true; + name = "Go race tests"; + entry = "${pkgs.go}/bin/go test -race ./..."; + files = "(\\.go$|^go\\.(mod|sum)$)"; + pass_filenames = false; + stages = [ "manual" ]; + }; + }; +} diff --git a/nix/modules/appview.nix b/nix/modules/appview.nix new file mode 100644 index 0000000..377963b --- /dev/null +++ b/nix/modules/appview.nix @@ -0,0 +1,75 @@ +{ + config, + lib, + pkgs, + ... +}: +let + cfg = config.services.provenance.appview; + common = import ./common.nix { inherit lib; }; +in +{ + options.services.provenance.appview = { + enable = lib.mkEnableOption "the Provenance appview"; + + package = lib.mkOption { + type = lib.types.package; + default = pkgs.callPackage ../package.nix { + pname = "provenance-appview"; + subPackages = [ "cmd/appview" ]; + }; + defaultText = lib.literalExpression "the Provenance appview package"; + description = "Package that provides bin/appview."; + }; + + listenAddress = lib.mkOption { + type = lib.types.str; + default = "127.0.0.1:8081"; + description = "HTTP listen address in host:port form. Use brackets for IPv6 hosts."; + }; + + stateDirectory = common.stateOption "provenance-appview"; + databaseFile = common.databaseOption "appview.db"; + serviceKeyFile = common.secretOption "Absolute runtime path to the shared service key. Keep the file outside the Nix store."; + + jetstreamURL = lib.mkOption { + type = lib.types.str; + default = "wss://jetstream.fire.hose.cam/subscribe"; + description = "Jetstream endpoint. Public Microcosm instances and self-hosted instances are supported."; + }; + + slingshotURL = lib.mkOption { + type = lib.types.str; + default = "https://slingshot.microcosm.blue"; + description = "Slingshot origin for missing event payloads."; + }; + + hopPenalty = lib.mkOption { + type = lib.types.addCheck (lib.types.either lib.types.float lib.types.int) (value: value >= 0); + default = 0.1; + description = "Trust penalty for each edge after the first edge. The value must be nonnegative."; + }; + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + assertion = common.validSecret cfg.serviceKeyFile; + message = "services.provenance.appview.serviceKeyFile must name an absolute runtime file outside the Nix store."; + } + ]; + + systemd.services.provenance-appview = common.unit { + inherit (cfg) package stateDirectory; + binary = "appview"; + credentials.PROVENANCE_SERVICE_KEY = cfg.serviceKeyFile; + environment = { + PROVENANCE_APPVIEW_ADDR = cfg.listenAddress; + PROVENANCE_APPVIEW_DB = "/var/lib/${cfg.stateDirectory}/${cfg.databaseFile}"; + PROVENANCE_JETSTREAM_URL = cfg.jetstreamURL; + PROVENANCE_SLINGSHOT_URL = cfg.slingshotURL; + PROVENANCE_HOP_PENALTY = toString cfg.hopPenalty; + }; + }; + }; +} diff --git a/nix/modules/common.nix b/nix/modules/common.nix new file mode 100644 index 0000000..002e7a6 --- /dev/null +++ b/nix/modules/common.nix @@ -0,0 +1,96 @@ +{ lib }: +let + validSecret = + path: + path != null + && lib.hasPrefix "/" path + && !(lib.hasPrefix "/nix/store/" path) + && !(lib.hasInfix "\n" path) + && !(lib.hasInfix "\r" path); +in +{ + inherit validSecret; + + secretOption = + description: + lib.mkOption { + type = lib.types.nullOr lib.types.str; + default = null; + inherit description; + example = "/run/secrets/provenance/service-key"; + }; + + stateOption = + name: + lib.mkOption { + type = lib.types.strMatching "[a-zA-Z0-9_-]+"; + default = name; + description = "Name of the private systemd state directory under /var/lib."; + }; + + databaseOption = + name: + lib.mkOption { + type = lib.types.strMatching "[a-zA-Z0-9_-][a-zA-Z0-9._-]*"; + default = name; + description = "SQLite filename within the managed state directory."; + }; + + # Load only credential paths into the unit. Secret contents enter the process at runtime. + unit = + { + package, + binary, + stateDirectory, + environment, + credentials, + }: + { + wantedBy = [ "multi-user.target" ]; + wants = [ "network-online.target" ]; + after = [ "network-online.target" ]; + inherit environment; + script = + lib.concatMapStringsSep "\n" (name: '' + export ${name}="$(< "$CREDENTIALS_DIRECTORY/${name}")" + '') (builtins.attrNames credentials) + + '' + exec ${package}/bin/${binary} + ''; + + serviceConfig = { + LoadCredential = lib.mapAttrsToList (name: path: "${name}:${path}") ( + lib.filterAttrs (_: path: path != null) credentials + ); + DynamicUser = true; + StateDirectory = stateDirectory; + StateDirectoryMode = "0700"; + WorkingDirectory = "/var/lib/${stateDirectory}"; + UMask = "0077"; + Restart = "on-failure"; + RestartSec = 5; + TimeoutStopSec = 15; + NoNewPrivileges = true; + ProtectSystem = "strict"; + ProtectHome = true; + PrivateTmp = true; + PrivateDevices = true; + ProtectClock = true; + ProtectKernelTunables = true; + ProtectKernelModules = true; + ProtectControlGroups = true; + RestrictSUIDSGID = true; + RestrictRealtime = true; + LockPersonality = true; + MemoryDenyWriteExecute = true; + CapabilityBoundingSet = ""; + AmbientCapabilities = ""; + RestrictAddressFamilies = [ + "AF_INET" + "AF_INET6" + "AF_UNIX" + "AF_NETLINK" + ]; + }; + }; +} diff --git a/nix/modules/ui.nix b/nix/modules/ui.nix new file mode 100644 index 0000000..04a2d09 --- /dev/null +++ b/nix/modules/ui.nix @@ -0,0 +1,86 @@ +{ + config, + lib, + pkgs, + ... +}: +let + cfg = config.services.provenance.ui; + common = import ./common.nix { inherit lib; }; + localAppview = config.services.provenance.appview.enable or false; +in +{ + options.services.provenance.ui = { + enable = lib.mkEnableOption "the Provenance UI"; + + package = lib.mkOption { + type = lib.types.package; + default = pkgs.callPackage ../package.nix { + pname = "provenance-ui"; + subPackages = [ "cmd/ui" ]; + }; + defaultText = lib.literalExpression "the Provenance UI package"; + description = "Package that provides bin/ui."; + }; + + listenAddress = lib.mkOption { + type = lib.types.str; + default = "127.0.0.1:8080"; + description = "HTTP listen address in host:port form. Use brackets for IPv6 hosts."; + }; + + stateDirectory = common.stateOption "provenance-ui"; + databaseFile = common.databaseOption "ui.db"; + serviceKeyFile = common.secretOption "Absolute runtime path to the appview service key. Keep the file outside the Nix store."; + encryptionKeyFile = common.secretOption "Absolute runtime path to the base64-encoded 32-byte encryption key. Keep the file outside the Nix store."; + + publicOrigin = lib.mkOption { + type = lib.types.str; + default = "http://127.0.0.1:8080"; + example = "https://provenance.example.com"; + description = "Browser origin and OAuth callback origin. Use HTTPS except for loopback development."; + }; + + appviewURL = lib.mkOption { + type = lib.types.str; + default = "http://127.0.0.1:8081"; + description = "Appview origin. Use HTTPS for remote appviews or HTTP on loopback."; + }; + }; + + config = lib.mkIf cfg.enable { + assertions = [ + { + assertion = common.validSecret cfg.serviceKeyFile; + message = "services.provenance.ui.serviceKeyFile must name an absolute runtime file outside the Nix store."; + } + { + assertion = common.validSecret cfg.encryptionKeyFile; + message = "services.provenance.ui.encryptionKeyFile must name an absolute runtime file outside the Nix store."; + } + { + assertion = + !localAppview || cfg.stateDirectory != config.services.provenance.appview.stateDirectory; + message = "The Provenance UI and appview require separate state directories."; + } + ]; + + systemd.services.provenance-ui = lib.mkMerge [ + (common.unit { + inherit (cfg) package stateDirectory; + binary = "ui"; + credentials = { + PROVENANCE_SERVICE_KEY = cfg.serviceKeyFile; + PROVENANCE_ENCRYPTION_KEY = cfg.encryptionKeyFile; + }; + environment = { + PROVENANCE_UI_ADDR = cfg.listenAddress; + PROVENANCE_UI_DB = "/var/lib/${cfg.stateDirectory}/${cfg.databaseFile}"; + PROVENANCE_PUBLIC_ORIGIN = cfg.publicOrigin; + PROVENANCE_APPVIEW_URL = cfg.appviewURL; + }; + }) + (lib.mkIf localAppview { after = [ "provenance-appview.service" ]; }) + ]; + }; +} diff --git a/nix/package.nix b/nix/package.nix new file mode 100644 index 0000000..247487e --- /dev/null +++ b/nix/package.nix @@ -0,0 +1,40 @@ +{ + lib, + buildGoModule, + pname ? "provenance", + subPackages ? [ + "cmd/appview" + "cmd/ui" + ], +}: + +buildGoModule { + inherit pname subPackages; + version = "0.1.0"; + + src = lib.fileset.toSource { + root = ../.; + fileset = lib.fileset.unions [ + ../go.mod + ../go.sum + ../cmd + ../internal + ]; + }; + + vendorHash = "sha256-AREnluE97NFAXDW5Vb8+rhqscWqxsNirtQCH8nbLQ9E="; + env.CGO_ENABLED = "0"; + + # Include library tests even when the package contains only one service binary. + checkPhase = '' + runHook preCheck + go test ./... + runHook postCheck + ''; + + meta = { + description = "ATProto relative reputation services"; + platforms = lib.platforms.unix; + mainProgram = if subPackages == [ "cmd/ui" ] then "ui" else "appview"; + }; +} diff --git a/nix/services/provenance.nix b/nix/services/provenance.nix new file mode 100644 index 0000000..fe80041 --- /dev/null +++ b/nix/services/provenance.nix @@ -0,0 +1,117 @@ +{ + config, + name, + lib, + pkgs, + ... +}: +let + cfg = config; + setupName = "${name}-setup"; + appviewName = "${name}-appview"; + uiName = "${name}-ui"; + setup = pkgs.writeShellApplication { + name = "provenance-dev-setup"; + runtimeInputs = [ pkgs.coreutils ]; + text = '' + umask 077 + state="''${PROVENANCE_DEV_DATA_DIR:-}" + if [ -z "$state" ]; then + state=${lib.escapeShellArg cfg.dataDir} + fi + mkdir -p "$state" + chmod 700 "$state" + + # Retain keys across restarts. Atomic rename prevents partial credential files. + for key in service-key encryption-key; do + if [ -L "$state/$key" ]; then + echo "Development key files must not be symbolic links." >&2 + exit 1 + fi + if [ -e "$state/$key" ]; then + chmod 600 "$state/$key" + continue + fi + temporary=$(mktemp "$state/.key.XXXXXX") + trap 'rm -f "$temporary"' EXIT + head -c 32 /dev/urandom | base64 > "$temporary" + mv -n "$temporary" "$state/$key" + rm -f "$temporary" + trap - EXIT + done + ''; + }; + start = + binary: + pkgs.writeShellApplication { + name = "provenance-dev-${binary}"; + runtimeInputs = [ pkgs.coreutils ]; + text = '' + state="''${PROVENANCE_DEV_DATA_DIR:-}" + if [ -z "$state" ]; then + state=${lib.escapeShellArg cfg.dataDir} + fi + PROVENANCE_SERVICE_KEY=$(cat "$state/service-key") + export PROVENANCE_SERVICE_KEY + ${lib.optionalString (binary == "ui") '' + PROVENANCE_ENCRYPTION_KEY=$(cat "$state/encryption-key") + export PROVENANCE_ENCRYPTION_KEY + ''} + export PROVENANCE_APPVIEW_ADDR="127.0.0.1:${toString cfg.appviewPort}" + export PROVENANCE_UI_ADDR="127.0.0.1:${toString cfg.uiPort}" + export PROVENANCE_APPVIEW_URL="http://127.0.0.1:${toString cfg.appviewPort}" + export PROVENANCE_PUBLIC_ORIGIN="''${PROVENANCE_PUBLIC_ORIGIN:-http://127.0.0.1:${toString cfg.uiPort}}" + export PROVENANCE_APPVIEW_DB="$state/appview.db" + export PROVENANCE_UI_DB="$state/ui.db" + exec ${cfg.package}/bin/${binary} + ''; + }; + probe = port: path: { + http_get = { + host = "127.0.0.1"; + inherit port path; + }; + initial_delay_seconds = 1; + period_seconds = 1; + timeout_seconds = 1; + failure_threshold = 30; + }; +in +{ + options = { + package = lib.mkOption { + type = lib.types.package; + default = pkgs.callPackage ../package.nix { }; + description = "Package that provides the appview and UI binaries."; + }; + appviewPort = lib.mkOption { + type = lib.types.port; + default = 8081; + description = "Loopback port for the development appview."; + }; + uiPort = lib.mkOption { + type = lib.types.port; + default = 8080; + description = "Loopback port for the development UI."; + }; + }; + + config.outputs.settings.processes = { + ${setupName} = { + command = setup; + availability.restart = "no"; + }; + ${appviewName} = { + command = start "appview"; + depends_on.${setupName}.condition = "process_completed_successfully"; + readiness_probe = probe cfg.appviewPort "/healthz"; + availability.restart = "on_failure"; + }; + ${uiName} = { + command = start "ui"; + depends_on.${appviewName}.condition = "process_healthy"; + readiness_probe = probe cfg.uiPort "/oauth-client-metadata.json"; + availability.restart = "on_failure"; + }; + }; +} diff --git a/nix/tests/modules.nix b/nix/tests/modules.nix new file mode 100644 index 0000000..7b78eea --- /dev/null +++ b/nix/tests/modules.nix @@ -0,0 +1,134 @@ +{ + self, + nixpkgs, + pkgs, + system, +}: +let + inherit (nixpkgs) lib; + evaluate = + modules: + (lib.nixosSystem { + inherit system; + modules = modules ++ [ + { + system.stateVersion = "26.05"; + boot.loader.grub.enable = false; + fileSystems."/" = { + device = "/dev/root"; + fsType = "ext4"; + }; + } + ]; + }).config; + valid = cfg: lib.all (item: item.assertion) cfg.assertions; + appviewConfig.services.provenance.appview = { + enable = true; + serviceKeyFile = "/run/secrets/provenance/service-key"; + listenAddress = "[::1]:9091"; + stateDirectory = "test-appview"; + databaseFile = "graph.sqlite"; + jetstreamURL = "wss://stream.example.com/subscribe"; + slingshotURL = "https://records.example.com"; + hopPenalty = 0.25; + }; + uiConfig.services.provenance.ui = { + enable = true; + serviceKeyFile = "/run/secrets/provenance/service-key"; + encryptionKeyFile = "/run/secrets/provenance/encryption-key"; + listenAddress = "[::1]:9090"; + stateDirectory = "test-ui"; + databaseFile = "sessions.sqlite"; + publicOrigin = "https://provenance.example.com"; + appviewURL = "https://appview.example.com"; + }; + disabled = evaluate [ self.nixosModules.default ]; + appview = evaluate [ + self.nixosModules.appview + appviewConfig + ]; + ui = evaluate [ + self.nixosModules.ui + uiConfig + ]; + combined = evaluate [ + self.nixosModules.default + appviewConfig + uiConfig + ]; + missingSecrets = evaluate [ + self.nixosModules.default + { + services.provenance.appview.enable = true; + services.provenance.ui.enable = true; + } + ]; + storeSecret = evaluate [ + self.nixosModules.appview + { + services.provenance.appview = { + enable = true; + serviceKeyFile = "/nix/store/not-a-runtime-secret"; + }; + } + ]; + sharedState = evaluate [ + self.nixosModules.default + appviewConfig + uiConfig + { + services.provenance.ui.stateDirectory = lib.mkForce "test-appview"; + } + ]; + appUnit = appview.systemd.services.provenance-appview; + uiUnit = ui.systemd.services.provenance-ui; + fixture = pkgs.runCommand "provenance-package-fixture" { } "mkdir -p $out/bin"; + override = evaluate [ + self.nixosModules.ui + uiConfig + { + services.provenance.ui.package = fixture; + } + ]; + invalidPenalty = + builtins.tryEval + (evaluate [ + self.nixosModules.appview + { + services.provenance.appview.hopPenalty = -0.1; + } + ]).services.provenance.appview.hopPenalty; +in +assert !(disabled.systemd.services ? provenance-appview); +assert !(disabled.systemd.services ? provenance-ui); +assert valid appview && valid ui && valid combined; +assert !(appview.systemd.services ? provenance-ui); +assert !(ui.systemd.services ? provenance-appview); +assert !(valid missingSecrets) && !(valid storeSecret); +assert !(valid sharedState); +assert !invalidPenalty.success; +assert appUnit.environment.PROVENANCE_APPVIEW_ADDR == "[::1]:9091"; +assert appUnit.environment.PROVENANCE_APPVIEW_DB == "/var/lib/test-appview/graph.sqlite"; +assert appUnit.environment.PROVENANCE_JETSTREAM_URL == "wss://stream.example.com/subscribe"; +assert appUnit.environment.PROVENANCE_SLINGSHOT_URL == "https://records.example.com"; +assert appUnit.environment.PROVENANCE_HOP_PENALTY == toString 0.25; +assert !(appUnit.environment ? PROVENANCE_SERVICE_KEY); +assert !(uiUnit.environment ? PROVENANCE_ENCRYPTION_KEY); +assert + appUnit.serviceConfig.LoadCredential + == [ "PROVENANCE_SERVICE_KEY:/run/secrets/provenance/service-key" ]; +assert builtins.length uiUnit.serviceConfig.LoadCredential == 2; +assert appUnit.serviceConfig.DynamicUser && uiUnit.serviceConfig.DynamicUser; +assert appUnit.serviceConfig.ProtectSystem == "strict"; +assert uiUnit.environment.PROVENANCE_PUBLIC_ORIGIN == "https://provenance.example.com"; +assert uiUnit.environment.PROVENANCE_UI_ADDR == "[::1]:9090"; +assert uiUnit.environment.PROVENANCE_UI_DB == "/var/lib/test-ui/sessions.sqlite"; +assert uiUnit.environment.PROVENANCE_APPVIEW_URL == "https://appview.example.com"; +assert lib.elem "provenance-appview.service" combined.systemd.services.provenance-ui.after; +assert lib.hasInfix (builtins.unsafeDiscardStringContext ( + toString fixture +)) override.systemd.services.provenance-ui.script; +pkgs.runCommand "provenance-module-checks" { } '' + mkdir -p "$out" + echo "NixOS module checks passed." > "$out/result" +'' diff --git a/nix/tests/services.nix b/nix/tests/services.nix new file mode 100644 index 0000000..64d3434 --- /dev/null +++ b/nix/tests/services.nix @@ -0,0 +1,64 @@ +{ self, system }: +{ + name = "provenance-services"; + + nodes.machine = { pkgs, ... }: { + imports = [ self.nixosModules.default ]; + environment.systemPackages = [ pkgs.curl ]; + + # Create test-only keys at runtime so that credentials follow the production path. + systemd.services.provenance-test-secrets = { + before = [ + "provenance-appview.service" + "provenance-ui.service" + ]; + requiredBy = [ + "provenance-appview.service" + "provenance-ui.service" + ]; + serviceConfig = { + Type = "oneshot"; + RemainAfterExit = true; + UMask = "0077"; + }; + script = '' + mkdir -p /run/provenance-test + head -c 32 /dev/urandom | base64 > /run/provenance-test/service-key + head -c 32 /dev/urandom | base64 > /run/provenance-test/encryption-key + ''; + }; + + services.provenance = { + appview = { + enable = true; + package = self.packages.${system}.appview; + serviceKeyFile = "/run/provenance-test/service-key"; + jetstreamURL = "ws://127.0.0.1:9999/subscribe"; + slingshotURL = "http://127.0.0.1:9999"; + }; + ui = { + enable = true; + package = self.packages.${system}.ui; + serviceKeyFile = "/run/provenance-test/service-key"; + encryptionKeyFile = "/run/provenance-test/encryption-key"; + }; + }; + }; + + testScript = '' + start_all() + machine.wait_for_unit("provenance-appview.service") + machine.wait_for_unit("provenance-ui.service") + machine.wait_for_open_port(8081) + machine.wait_for_open_port(8080) + machine.succeed("curl --fail http://127.0.0.1:8081/healthz") + machine.succeed("curl --fail http://127.0.0.1:8080/oauth-client-metadata.json") + machine.succeed("curl --fail http://127.0.0.1:8080/ | grep Provenance") + machine.succeed("test $(stat -c %a /var/lib/provenance-appview/appview.db) = 600") + machine.succeed("test $(stat -c %a /var/lib/provenance-ui/ui.db) = 600") + machine.succeed("systemctl restart provenance-appview provenance-ui") + machine.wait_for_open_port(8081) + machine.wait_for_open_port(8080) + machine.succeed("curl --fail http://127.0.0.1:8080/") + ''; +}