diff --git a/DATA-FLOW.md b/DATA-FLOW.md index 57b097ffd..d20d146f8 100644 --- a/DATA-FLOW.md +++ b/DATA-FLOW.md @@ -1,6 +1,6 @@ # what solstone sends to your AI provider — and what it doesn't -solstone is local-first. your observers, your audio and screen, and your journal all stay on your machine, in plain files you own. this doc is the plain answer to the question a privacy-motivated owner should be able to *find* rather than *ask*: when solstone uses an AI model, what actually leaves your machine, who it goes to, and under whose terms. +solstone is local-first. sol on your devices, your audio and screen, and your journal all stay on your machine, in plain files you own. this doc is the plain answer to the question a privacy-motivated owner should be able to *find* rather than *ask*: when solstone uses an AI model, what actually leaves your machine, who it goes to, and under whose terms. short version: with a local model, nothing leaves. with a hosted provider, only the specific task's text goes — straight from your machine to that provider, under your own key and your own account. sol pbc is never in that path and never sees it. diff --git a/INSTALL.md b/INSTALL.md index e740ff9fd..72ad12224 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -108,7 +108,7 @@ what actually leaves your machine differs sharply between these paths: with the ## install sol on your devices -your journal needs sol alongside it — sol takes in your day on each device and keeps it in your journal. each platform ships its own package (the engineering docs call these observers); install one for each machine you want sol on. +your journal needs sol alongside it — sol takes in your day on each device and keeps it in your journal. each platform ships its own package; install one for each machine you want sol on. **macOS:** download the signed app bundle from https://solstone.app/observers and drag it to Applications. it pairs itself with the running journal on first launch. @@ -119,7 +119,7 @@ pipx install solstone-linux solstone-linux install-service ``` -`solstone-linux install-service` walks you through pairing the observer with your running journal. choose any name you'd like to identify this machine by. +`solstone-linux install-service` walks you through pairing this machine with your running journal. choose any name you'd like to identify this machine by. **tmux terminal sessions:** @@ -128,7 +128,7 @@ pipx install solstone-tmux solstone-tmux install-service ``` -(for observer packages, `uv tool install solstone-tmux` is also fine if you prefer uv.) +(for these packages, `uv tool install solstone-tmux` is also fine if you prefer uv.) ## migrating from a pre-split install @@ -192,7 +192,7 @@ Skipping `--journal` here silently resolves to `~/journal` and starts fresh — 2. optional: remove the installed `sol` agent skill: `sol skills uninstall`. 3. uninstall the python packages: `uv tool uninstall solstone-journal && uv tool uninstall solstone` (or `pipx uninstall solstone-journal`). 4. macOS only: drag `/Applications/solstone.app` to Trash. -5. macOS only, optional: remove observer app data and the parakeet model cache: +5. macOS only, optional: remove sol's app data and the parakeet model cache: ```bash rm -rf ~/Library/Application\ Support/solstone/ ``` -- 2.51.2 From c7349edf5ef1d7a9b273eb07e419ada4998e6ff0 Mon Sep 17 00:00:00 2001 From: Jer Miller Date: Fri, 31 Jul 2026 15:31:45 -0600 Subject: [PATCH 2/6] docs: align owner-facing terminology canon --- AGENTS.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8d486b7fa..e4d16d213 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -522,15 +522,15 @@ The live journal also carries `journal/AGENTS.md` as its runtime-facing breadcru ## 13. Owner-facing copy: the system-anatomy canon -- **Composition by register: owner-facing two parts, sol the keeper.** In owner-facing copy, name the two parts the owner has — `solstone = observers + journal` — and name sol as the keeper who lives in and tends the journal, not a third enumerated part. Never write a three-part owner-facing enumeration in owner-visible copy. The engineering/architecture register is retained and explicit: in architecture statements, technical docs, system/diagram-internal labels, code-side prose, and this repo's architecture sections, the system is `solstone = observers + sol agent + journal` — the sol agent is the running software that tends the journal. The split is by register, not contradiction: owner-facing → two parts, sol the keeper in the journal; engineering/architecture → the sol agent is the running software that tends the journal. -- **Ban surveillance verbs in branded surfaces.** Never use "capture", "watch", "record", "monitor", "track", or "collect" in template copy, settings labels, error messages, onboarding text, or README / INSTALL prose. Prefer "observe alongside", "experience along with", or "take in what you take in". -- **`capture` is code-only.** Keep it in module names such as `solstone/observe/`, function names, OS subsystem identifiers such as `com.solstone.capture`, and internal architecture diagrams. That is intentional and aligned with the canon. +- **Owner-facing composition.** solstone is the platform; sol is the app on your devices; the journal is the memory sol keeps. In owner-facing copy, use `sol`, `the journal`, and `your journal`; do not write `solstone = observers + journal`, use “observer” as a customer-facing noun, or call sol a keeper. The engineering register remains distinct: `observer`/`observe` may remain in repo names, code, protocol fields, `OBSERVE.md`, `AGENTS.md`, and technical prose. +- **Ban surveillance vocabulary in owner-facing surfaces.** Never use “capture”, “watch”, “record”, “monitor”, “track”, “collect”, “observe”, or bare “listen”/“hear” to describe sol in template copy, settings labels, error messages, onboarding text, or README / INSTALL prose. Prefer “sol experiences your day with you”, “sol takes in what you take in”, and “sol keeps it all in your journal”. +- **Technical identifiers are not renamed.** Keep `capture` and `observer`/`observe` in module names, function names, protocol fields, OS subsystem identifiers, `OBSERVE.md`, `AGENTS.md`, and internal architecture diagrams. These are engineering vocabulary, not customer-facing copy. - **Name artifacts for owners, not pipelines.** In branded prose, say "raw media", "the originals", or "observations". Never say "raw captures" or "screen captures" in owner-facing strings. Code-side artifact names stay as-is. -- **`sol` is one thing.** `sol` is the running software; there is no homunculus behind it. Use two registers for one entity: `sol` in conversation, `sol agent` in technical contexts. -- **`keeper` is a surface-specific edge case.** `voice-terminology.md` makes `keeper` the role noun for `sol` in product copy generally. The `solstone-swift` surface bans `keeper` because the mobile UX uses the owner's chosen identity, default `sol`. When writing copy for a specific surface, follow that surface's terminology covenant. +- **`sol` is one thing.** `sol` is the app and running software; there is no homunculus behind it. Use `sol` in owner-facing copy and `sol agent` in technical contexts—the same entity, not a keeper role. +- **`keeper` is retired from customer-facing copy.** Use `sol` by name and the approved verbs; `sol agent` is reserved for technical contexts. Surface-specific rules may be stricter, but none reintroduce “keeper” to owner-facing copy. - **Edit with the right mental model.** Internal architecture vocabulary in this repo stays as-is: `solstone/observe/`, the capture pipeline, and screen capture log subsystems remain correct code language. Apply the canon to owner-facing strings only: UI copy, settings text, install / README prose, error messages, and onboarding. If an owner sees it, follow the canon; if it's code or internal docs about pipelines, `capture` is fine. | Surface | Terminology rule | |---------|------------------| -| Code surfaces | `capture` is fine in code, module names, function names, subsystem ids, and internal architecture docs. | -| Branded surfaces | `capture` is banned. Use owner-facing phrasing such as "observe alongside", "experience along with", "take in what you take in", "raw media", "the originals", or "observations". | +| Code surfaces | `capture`, `observer`, and `observe` are fine in code, identifiers, protocol fields, subsystem ids, `AGENTS.md`, and internal architecture docs. | +| Branded surfaces | `capture`, `observer`, `observe`, and surveillance verbs are banned in owner-facing prose. Use “sol experiences your day with you”, “sol takes in what you take in”, “the journal”, and “your journal”. | -- 2.51.2 From 400d5eb47b048ec99ae241c5cdcb883f1c57c822 Mon Sep 17 00:00:00 2001 From: Jer Miller Date: Fri, 31 Jul 2026 16:52:24 -0600 Subject: [PATCH 3/6] fix(install): source checkouts install the published speakers-analyze helper uv sync prunes the workspace-excluded helper, leaving source checkouts with a journal doctor blocker and an EX_CONFIG supervisor exit. Add make speakers-analyze-helper to reinstall the published wheel using a pin derived at run time from packages/solstone-journal/pyproject.toml and uv pip install --no-config --no-deps. Run it from make install after the platform sync block so later uv syncs do not prune it. Update the speakers-analyze repair text to keep the packaged message unchanged while source checkouts point contributors to make speakers-analyze-helper or make install. --- AGENTS.md | 1 + CONTRIBUTING.md | 2 + Makefile | 7 +- scripts/install_speakers_analyze_helper.py | 266 ++++++++++++++++++ solstone/think/doctor.py | 4 +- .../think/speakers_analyze_installation.py | 23 +- tests/test_doctor.py | 2 +- tests/test_install_speakers_analyze_helper.py | 197 +++++++++++++ tests/test_sense.py | 4 +- ...eakers_analyze_helper_makefile_contract.py | 55 ++++ tests/test_speakers_analyze_installation.py | 40 +++ tests/test_think_utils.py | 44 +++ 12 files changed, 634 insertions(+), 11 deletions(-) create mode 100644 scripts/install_speakers_analyze_helper.py create mode 100644 tests/test_install_speakers_analyze_helper.py create mode 100644 tests/test_speakers_analyze_helper_makefile_contract.py diff --git a/AGENTS.md b/AGENTS.md index e4d16d213..7178494a5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -90,6 +90,7 @@ Verified against `Makefile`. Grouped by use. | Target | When to use | |--------|-------------| | `make install` | First setup and whenever `pyproject.toml` or `uv.lock` changes. Creates `.venv/`, syncs deps, runs `make skills`. | +| `make speakers-analyze-helper` | Reinstall the published speakers-analyze helper into `.venv` after a source-checkout `uv sync` prunes it. Normally run by `make install`; useful after manual syncs. | | `make skills` | Regenerate generated router references, then rewrite the `sol` + `journal` router skill symlinks into `journal/`. (`make install` depends on this; rarely run alone.) | | `make update` | Upgrade all deps to latest, regenerate `uv.lock`. Expect test churn. | | `make update-prices` | Refresh genai-prices model-cost data when adding a new provider model or when pricing tests fail. | diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b271dbab3..1d9b78756 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -56,6 +56,8 @@ make install `make install` creates `.venv/`, syncs dependencies from `pyproject.toml` and `uv.lock`, installs the package in editable mode, regenerates router skill references, and refreshes the `sol` + `journal` project skill symlinks into the journal. +In a source checkout, bare `uv sync` removes the published speakers-analyze helper because the workspace config prunes that package from the active dev environment; `make speakers-analyze-helper` puts the published helper back, and `make install` runs it after the final sync. Use `uv sync --inexact` when you intentionally need a prune-free sync. From inside this workspace, `uv pip install` of the helper can report success with exit code 0 while installing nothing unless `--no-config` is passed, so use the Make target instead of hand-running uv. + `.venv/bin/journal setup` runs doctor diagnostics, confirms the journal path, installs local transcription models, installs the `sol` user skill for Claude Code / Codex / Gemini when those agents are configured, installs the `sol` + `journal` router skills into the journal, creates or refreshes the source-checkout wrappers at `~/.local/bin/sol` and `~/.local/bin/journal`, and starts the background service. The default web interface listens on http://localhost:5015. Use `.venv/bin/journal setup --port 8000` to choose another port on the first run. After the first setup run, the wrapper lets you use `sol` from anywhere: diff --git a/Makefile b/Makefile index e5322e0f1..51d1e8bdb 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,7 @@ export TMPDIR := /var/tmp PYTEST_BASETEMP_INIT := BASETEMP=$$(mktemp -d /var/tmp/solstone-pytest-XXXXXX); trap 'rm -rf "$$BASETEMP"' EXIT INT TERM; PYTEST_BASETEMP_FLAG := --basetemp "$$BASETEMP" -.PHONY: install hopper-install uninstall test test-cov test-integration test-release release-checks test-performance test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging check-rust-fmt check-rust-msrv check-rust-clippy check-rust-test check-rust-ios check-rust-deny check-release-advisory-liveness check-rust-release-manifest audit openapi check-openapi check-openapi-observer-client-contract contract check-contract journal-resolution-vectors check-journal-resolution-vectors build-native-sol-grammar-oracle check-native-sol-grammar-oracle build-native-sol-root-contract check-native-sol-root-contract check-core-sdist-compile-inputs build-native-sol-journal-host-commands check-native-sol-journal-host-commands build-journal-access-rejection-inventory check-journal-access-rejection-inventory check-native-sol-python-manifest build-native-sol-inventory check-native-sol-inventory check-native-sol-architecture check-native-sol-contract-routes check-native-sol-conformance check-native-sol-coverage check-native-sol-no-python-spawn check-native-sol-compat check-native-sol-docs-links check-removed-time-parser-ready dev all sandbox sandbox-stop install-models parakeet-helper parakeet-helper-clean wheel-speakers-analyze-linux wheel-speakers-analyze-linux-x86_64 wheel-speakers-analyze-linux-aarch64 wheel-macos wheel-macos-clean verify verify-api verify-schemathesis update-api-baselines eval-schemas service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-journal-config-owner check-call-http-only check-no-legacy-chat check-channel-adapter-scrub check-brain-health-cutover check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-schema-bounds check-thin-base-install check-extras-consistency check-cogitate-prompts smoke-cogitate release release-test publish-release publish-release-test FORCE +.PHONY: install hopper-install uninstall test test-cov test-integration test-release release-checks test-performance test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging check-rust-fmt check-rust-msrv check-rust-clippy check-rust-test check-rust-ios check-rust-deny check-release-advisory-liveness check-rust-release-manifest audit openapi check-openapi check-openapi-observer-client-contract contract check-contract journal-resolution-vectors check-journal-resolution-vectors build-native-sol-grammar-oracle check-native-sol-grammar-oracle build-native-sol-root-contract check-native-sol-root-contract check-core-sdist-compile-inputs build-native-sol-journal-host-commands check-native-sol-journal-host-commands build-journal-access-rejection-inventory check-journal-access-rejection-inventory check-native-sol-python-manifest build-native-sol-inventory check-native-sol-inventory check-native-sol-architecture check-native-sol-contract-routes check-native-sol-conformance check-native-sol-coverage check-native-sol-no-python-spawn check-native-sol-compat check-native-sol-docs-links check-removed-time-parser-ready dev all sandbox sandbox-stop install-models speakers-analyze-helper parakeet-helper parakeet-helper-clean wheel-speakers-analyze-linux wheel-speakers-analyze-linux-x86_64 wheel-speakers-analyze-linux-aarch64 wheel-macos wheel-macos-clean verify verify-api verify-schemathesis update-api-baselines eval-schemas service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-journal-config-owner check-call-http-only check-no-legacy-chat check-channel-adapter-scrub check-brain-health-cutover check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-schema-bounds check-thin-base-install check-extras-consistency check-cogitate-prompts smoke-cogitate release release-test publish-release publish-release-test FORCE # Default target - install package in editable mode all: install @@ -153,6 +153,7 @@ install: .installed echo "parakeet install: unsupported host '$$OS_NAME/$$ARCH'; supported: darwin/arm64, linux/x86_64" >&2; \ exit 1; \ fi + @$(MAKE) speakers-analyze-helper || { echo 'speakers-analyze helper install failed' >&2; exit 1; } @touch .installed @$(VENV_BIN)/journal install-models || { echo "journal install-models failed" >&2; exit 1; } @@ -351,6 +352,10 @@ install-models: @test -x "$(VENV_BIN)/sol" || { echo "missing $(VENV_BIN)/sol; run make install first" >&2; exit 1; } $(VENV_BIN)/journal install-models +speakers-analyze-helper: + @test -x "$(VENV_BIN)/python" || { echo "missing $(VENV_BIN)/python; run make install first" >&2; exit 1; } + @$(VENV_BIN)/python scripts/install_speakers_analyze_helper.py + # Build the parakeet helper binary (macOS/arm64 only, requires Xcode CLT) parakeet-helper: cd solstone/observe/transcribe/parakeet_helper && swift build -c release diff --git a/scripts/install_speakers_analyze_helper.py b/scripts/install_speakers_analyze_helper.py new file mode 100644 index 000000000..299e76364 --- /dev/null +++ b/scripts/install_speakers_analyze_helper.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc +"""Install the published speakers-analyze helper into a source checkout venv.""" + +from __future__ import annotations + +import os +import shutil +import subprocess +import sys +import tomllib +from collections.abc import Callable, Mapping, Sequence +from dataclasses import dataclass +from importlib.metadata import PackageNotFoundError +from importlib.metadata import version as distribution_version +from pathlib import Path + +from packaging.markers import Marker, default_environment +from packaging.requirements import Requirement + +from solstone.think.speakers_analyze_installation import ( + HELPER_DIST_NAME, + speakers_analyze_path_for_executable, +) + +ROOT = Path(__file__).resolve().parent.parent + +Runner = Callable[..., subprocess.CompletedProcess[str]] +UvFinder = Callable[[str], str | None] +VersionReader = Callable[[str], str] +ExecutablePredicate = Callable[[Path], bool] + + +class SpeakersAnalyzeHelperInstallError(RuntimeError): + """Raised when the published helper cannot be installed or verified.""" + + +@dataclass(frozen=True) +class DerivedHelperPin: + pin: str + expected_version: str + markers: tuple[Marker, ...] + raw_requirements: tuple[str, ...] + + +def default_executable_predicate(path: Path) -> bool: + return os.access(path, os.X_OK) + + +def read_project_dependencies(pyproject_path: Path) -> list[str]: + try: + data = tomllib.loads(pyproject_path.read_text(encoding="utf-8")) + except FileNotFoundError as exc: + raise SpeakersAnalyzeHelperInstallError( + f"missing package metadata: {pyproject_path}" + ) from exc + dependencies = data.get("project", {}).get("dependencies") + if not isinstance(dependencies, list) or not all( + isinstance(item, str) for item in dependencies + ): + raise SpeakersAnalyzeHelperInstallError( + f"{pyproject_path} project.dependencies must be a list of strings" + ) + return dependencies + + +def derive_helper_pin( + dependencies: Sequence[str], *, dependency_label: str = "project.dependencies" +) -> DerivedHelperPin: + raw_requirements: list[str] = [] + pins: set[str] = set() + markers: list[Marker] = [] + + for raw in dependencies: + requirement = Requirement(raw) + if requirement.name != HELPER_DIST_NAME: + continue + raw_requirements.append(raw) + pin = f"{requirement.name}{requirement.specifier}" + pins.add(pin) + if requirement.marker is None: + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} {HELPER_DIST_NAME} pins must all be marker-gated; " + f"missing marker on {raw!r}" + ) + markers.append(requirement.marker) + + if not raw_requirements: + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} must contain {HELPER_DIST_NAME}; found none" + ) + if len(pins) != 1: + found = ", ".join(sorted(pins)) + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} {HELPER_DIST_NAME} pins must resolve to exactly " + f"one name==version; found {found}" + ) + + pin = next(iter(pins)) + expected_version = _expected_version(pin, dependency_label=dependency_label) + return DerivedHelperPin( + pin=pin, + expected_version=expected_version, + markers=tuple(markers), + raw_requirements=tuple(raw_requirements), + ) + + +def is_environment_covered( + markers: Sequence[Marker], environment: Mapping[str, str] +) -> bool: + return any(marker.evaluate(dict(environment)) for marker in markers) + + +def ensure_running_target_python(running_python: Path, target_python: Path) -> None: + expected = target_python.resolve() + actual = running_python.resolve() + if actual != expected: + raise SpeakersAnalyzeHelperInstallError( + f"speakers-analyze helper install must run under {expected}; " + f"running under {actual}" + ) + + +def install_helper( + pin: str, + *, + python: Path, + uv_executable: str, + runner: Runner = subprocess.run, +) -> None: + argv = [ + uv_executable, + "pip", + "install", + "--no-config", + "--no-deps", + "--python", + str(python), + pin, + ] + result = runner(argv, capture_output=True, text=True) + if result.returncode != 0: + detail = (result.stderr or result.stdout or "").strip() + suffix = f": {detail}" if detail else "" + raise SpeakersAnalyzeHelperInstallError( + f"speakers-analyze helper install failed: uv pip install exited " + f"{result.returncode}{suffix}" + ) + + +def assert_helper_installed( + pin: str, + *, + python: Path, + version_reader: VersionReader = distribution_version, + executable_predicate: ExecutablePredicate = default_executable_predicate, +) -> None: + expected = _expected_version(pin, dependency_label="installed helper pin") + try: + actual = version_reader(HELPER_DIST_NAME) + except PackageNotFoundError as exc: + raise SpeakersAnalyzeHelperInstallError( + f"speakers-analyze helper install failed: missing {HELPER_DIST_NAME} " + "distribution metadata" + ) from exc + if actual != expected: + raise SpeakersAnalyzeHelperInstallError( + f"speakers-analyze helper install failed: {HELPER_DIST_NAME} is " + f"{actual} but expected {expected}" + ) + + helper_path = speakers_analyze_path_for_executable(python) + if not helper_path.exists(): + raise SpeakersAnalyzeHelperInstallError( + f"speakers-analyze helper install failed: missing executable {helper_path}" + ) + if not executable_predicate(helper_path): + raise SpeakersAnalyzeHelperInstallError( + "speakers-analyze helper install failed: executable is not executable " + f"{helper_path}" + ) + + +def run_installation( + *, + repo_root: Path, + running_python: Path, + environment: Mapping[str, str], + runner: Runner = subprocess.run, + uv_finder: UvFinder = shutil.which, + version_reader: VersionReader = distribution_version, + executable_predicate: ExecutablePredicate = default_executable_predicate, +) -> int: + venv_python = repo_root / ".venv" / "bin" / "python" + ensure_running_target_python(running_python, venv_python) + + dependencies = read_project_dependencies( + repo_root / "packages" / "solstone-journal" / "pyproject.toml" + ) + helper_pin = derive_helper_pin(dependencies) + if not is_environment_covered(helper_pin.markers, environment): + print( + "speakers-analyze helper install skipped: this environment is not " + f"covered by {HELPER_DIST_NAME} markers" + ) + return 0 + + uv_executable = uv_finder("uv") + if uv_executable is None: + raise SpeakersAnalyzeHelperInstallError( + "speakers-analyze helper install failed: uv not found on PATH" + ) + install_helper( + helper_pin.pin, + python=venv_python, + uv_executable=uv_executable, + runner=runner, + ) + assert_helper_installed( + helper_pin.pin, + python=venv_python, + version_reader=version_reader, + executable_predicate=executable_predicate, + ) + helper_path = speakers_analyze_path_for_executable(venv_python) + print( + f"speakers-analyze helper ready: {helper_path} ({helper_pin.expected_version})" + ) + return 0 + + +def main(argv: Sequence[str] | None = None) -> int: + args = list(sys.argv[1:] if argv is None else argv) + if args: + print("usage: install_speakers_analyze_helper.py", file=sys.stderr) + return 2 + try: + return run_installation( + repo_root=ROOT, + running_python=Path(sys.executable), + environment=default_environment(), + ) + except SpeakersAnalyzeHelperInstallError as exc: + print(exc, file=sys.stderr) + return 1 + + +def _expected_version(pin: str, *, dependency_label: str) -> str: + requirement = Requirement(pin) + specifiers = tuple(requirement.specifier) + if ( + requirement.name != HELPER_DIST_NAME + or len(specifiers) != 1 + or specifiers[0].operator != "==" + ): + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} must pin {HELPER_DIST_NAME} with exactly one " + f"== version; found {pin!r}" + ) + return specifiers[0].version + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/solstone/think/doctor.py b/solstone/think/doctor.py index bf943c385..8662034f1 100644 --- a/solstone/think/doctor.py +++ b/solstone/think/doctor.py @@ -1412,15 +1412,15 @@ def parakeet_cpp_stt_ready_check(args: Args) -> CheckResult: def speakers_analyze_installation_check(args: Args) -> CheckResult: del args from solstone.think.speakers_analyze_installation import ( - SPEAKERS_ANALYZE_REPAIR_TEXT, check_speakers_analyze_installation, + speakers_analyze_repair_text, ) check = SPEAKERS_ANALYZE_INSTALLATION_CHECK result = check_speakers_analyze_installation() if result.ok: return make_result(check, "ok", "speakers-analyze installation ready") - return make_result(check, "fail", result.message, SPEAKERS_ANALYZE_REPAIR_TEXT) + return make_result(check, "fail", result.message, speakers_analyze_repair_text()) def _make_feature_check( diff --git a/solstone/think/speakers_analyze_installation.py b/solstone/think/speakers_analyze_installation.py index 3a7e5e70a..447b681ab 100644 --- a/solstone/think/speakers_analyze_installation.py +++ b/solstone/think/speakers_analyze_installation.py @@ -40,7 +40,7 @@ from solstone.think.model_assets import ( resolve_pyannote_segmentation_model, resolve_wespeaker_model, ) -from solstone.think.utils import get_journal +from solstone.think.utils import get_journal, is_source_checkout HELPER_DIST_NAME = "solstone-core-speakers-analyze" MODELS_DIST_NAME = "solstone-journal-models" @@ -56,10 +56,15 @@ GENERATION_FD_MIN = 3 GENERATION_FD_MAX = 1_048_576 GENERATION_TOKEN_MAX = (1 << 31) - 1 -SPEAKERS_ANALYZE_REPAIR_TEXT = ( +PACKAGED_SPEAKERS_ANALYZE_REPAIR_TEXT = ( "Repair: reinstall the journal host stack with solstone-journal, or " "solstone-journal-cuda on NVIDIA hosts, and restart the journal." ) +SOURCE_CHECKOUT_SPEAKERS_ANALYZE_REPAIR_TEXT = ( + "Repair: run make speakers-analyze-helper to install the published " + "speakers-analyze helper, then run make install to finish source-checkout " + "setup." +) SpeakersAnalyzeInstallationStatus = Literal[ "ok", @@ -89,7 +94,7 @@ class SpeakersAnalyzeInstallationResult: detail = f": {self.detail}" if self.detail else "" return ( f"Speakers-analyze installation is incomplete " - f"({self.status}{detail}). {SPEAKERS_ANALYZE_REPAIR_TEXT}" + f"({self.status}{detail}). {speakers_analyze_repair_text()}" ) @@ -118,6 +123,12 @@ def speakers_analyze_path_for_executable(executable: str | Path | None = None) - return Path(executable or sys.executable).with_name(HELPER_BINARY_NAME) +def speakers_analyze_repair_text() -> str: + if is_source_checkout(): + return SOURCE_CHECKOUT_SPEAKERS_ANALYZE_REPAIR_TEXT + return PACKAGED_SPEAKERS_ANALYZE_REPAIR_TEXT + + def _packaging_platform_tags() -> set[str]: return {tag.platform for tag in tags.sys_tags()} @@ -604,18 +615,20 @@ def _now_iso() -> str: __all__ = [ - "GENERATION_FD_ENV_KEY", "GENERATION_ENV_KEY", + "GENERATION_FD_ENV_KEY", "GENERATION_TOKEN_ENV_KEY", "HELPER_BINARY_NAME", "HELPER_DIST_NAME", "MODELS_DIST_NAME", + "PACKAGED_SPEAKERS_ANALYZE_REPAIR_TEXT", "ROOT_DIST_NAME", - "SPEAKERS_ANALYZE_REPAIR_TEXT", + "SOURCE_CHECKOUT_SPEAKERS_ANALYZE_REPAIR_TEXT", "SpeakersAnalyzeGeneration", "SpeakersAnalyzeInstallationResult", "check_speakers_analyze_installation", "enter_speakers_analyze_generation", "runtime_has_speakers_analyze_wheel_coverage", "speakers_analyze_path_for_executable", + "speakers_analyze_repair_text", ] diff --git a/tests/test_doctor.py b/tests/test_doctor.py index bd0f84b83..9a5b50856 100644 --- a/tests/test_doctor.py +++ b/tests/test_doctor.py @@ -600,7 +600,7 @@ class TestSpeakersAnalyzeInstallation: assert result.status == "fail" assert "asset-missing" in result.detail - assert result.fix == installation.SPEAKERS_ANALYZE_REPAIR_TEXT + assert result.fix == installation.speakers_analyze_repair_text() class TestHostDependencies: diff --git a/tests/test_install_speakers_analyze_helper.py b/tests/test_install_speakers_analyze_helper.py new file mode 100644 index 000000000..d89e26b2d --- /dev/null +++ b/tests/test_install_speakers_analyze_helper.py @@ -0,0 +1,197 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc + +from __future__ import annotations + +import subprocess +from pathlib import Path + +import pytest + +from scripts import install_speakers_analyze_helper as installer + + +def _write_leaf_pyproject(root: Path, dependencies: list[str]) -> None: + leaf = root / "packages" / "solstone-journal" + leaf.mkdir(parents=True) + deps = "\n".join(f' "{dependency}",' for dependency in dependencies) + (leaf / "pyproject.toml").write_text( + f"[project]\ndependencies = [\n{deps}\n]\n", + encoding="utf-8", + ) + + +def _venv_python(root: Path) -> Path: + return root / ".venv" / "bin" / "python" + + +def _linux_x86_64_env() -> dict[str, str]: + return {"sys_platform": "linux", "platform_machine": "x86_64"} + + +def _covered_pin(version: str = "7.8.9") -> str: + return ( + f"{installer.HELPER_DIST_NAME}=={version}; " + "sys_platform == 'linux' and platform_machine == 'x86_64'" + ) + + +def test_derive_helper_pin_rejects_multiple_distinct_versions() -> None: + dependencies = [ + _covered_pin("7.8.9"), + ( + f"{installer.HELPER_DIST_NAME}==8.0.0; " + "sys_platform == 'darwin' and platform_machine == 'arm64'" + ), + ] + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.derive_helper_pin(dependencies) + + message = str(exc_info.value) + assert "exactly one name==version" in message + assert f"{installer.HELPER_DIST_NAME}==7.8.9" in message + assert f"{installer.HELPER_DIST_NAME}==8.0.0" in message + + +def test_derive_helper_pin_rejects_zero_helper_pins() -> None: + with pytest.raises( + installer.SpeakersAnalyzeHelperInstallError, + match=f"must contain {installer.HELPER_DIST_NAME}; found none", + ): + installer.derive_helper_pin(["onnxruntime>=1.25.0"]) + + +def test_uncovered_environment_skips_without_uv(tmp_path: Path, capsys) -> None: + root = tmp_path / "repo" + _write_leaf_pyproject( + root, + [ + ( + f"{installer.HELPER_DIST_NAME}==7.8.9; " + "sys_platform == 'darwin' and platform_machine == 'arm64'" + ) + ], + ) + + def fail_runner( + *_args: object, **_kwargs: object + ) -> subprocess.CompletedProcess[str]: + raise AssertionError("uv must not run for uncovered environments") + + def fail_uv_finder(_name: str) -> str | None: + raise AssertionError("uv must not be located for uncovered environments") + + rc = installer.run_installation( + repo_root=root, + running_python=_venv_python(root), + environment=_linux_x86_64_env(), + runner=fail_runner, + uv_finder=fail_uv_finder, + ) + + captured = capsys.readouterr() + assert rc == 0 + assert "not covered" in captured.out + + +def test_install_uses_no_config_no_deps_target_python_and_derived_pin( + tmp_path: Path, +) -> None: + root = tmp_path / "repo" + _write_leaf_pyproject(root, [_covered_pin()]) + python = _venv_python(root) + python.parent.mkdir(parents=True) + helper = installer.speakers_analyze_path_for_executable(python) + helper.write_text("#!/bin/sh\n", encoding="utf-8") + helper.chmod(0o755) + calls: list[list[str]] = [] + + def fake_runner( + argv: list[str], **_kwargs: object + ) -> subprocess.CompletedProcess[str]: + calls.append(argv) + return subprocess.CompletedProcess(argv, 0, "", "") + + rc = installer.run_installation( + repo_root=root, + running_python=python, + environment=_linux_x86_64_env(), + runner=fake_runner, + uv_finder=lambda _name: "/usr/bin/uv", + version_reader=lambda dist_name: "7.8.9", + ) + + assert rc == 0 + assert calls == [ + [ + "/usr/bin/uv", + "pip", + "install", + "--no-config", + "--no-deps", + "--python", + str(python), + f"{installer.HELPER_DIST_NAME}==7.8.9", + ] + ] + + +def test_missing_uv_fails_loudly(tmp_path: Path) -> None: + root = tmp_path / "repo" + _write_leaf_pyproject(root, [_covered_pin()]) + python = _venv_python(root) + + with pytest.raises( + installer.SpeakersAnalyzeHelperInstallError, + match="uv not found on PATH", + ): + installer.run_installation( + repo_root=root, + running_python=python, + environment=_linux_x86_64_env(), + uv_finder=lambda _name: None, + ) + + +def test_assert_helper_installed_reports_version_mismatch(tmp_path: Path) -> None: + with pytest.raises( + installer.SpeakersAnalyzeHelperInstallError, + match=f"{installer.HELPER_DIST_NAME} is 0.0.1 but expected 7.8.9", + ): + installer.assert_helper_installed( + f"{installer.HELPER_DIST_NAME}==7.8.9", + python=_venv_python(tmp_path), + version_reader=lambda _dist_name: "0.0.1", + ) + + +def test_assert_helper_installed_reports_missing_binary(tmp_path: Path) -> None: + with pytest.raises( + installer.SpeakersAnalyzeHelperInstallError, + match="missing executable", + ): + installer.assert_helper_installed( + f"{installer.HELPER_DIST_NAME}==7.8.9", + python=_venv_python(tmp_path), + version_reader=lambda _dist_name: "7.8.9", + ) + + +def test_assert_helper_installed_reports_non_executable_binary(tmp_path: Path) -> None: + python = _venv_python(tmp_path) + python.parent.mkdir(parents=True) + helper = installer.speakers_analyze_path_for_executable(python) + helper.write_text("#!/bin/sh\n", encoding="utf-8") + helper.chmod(0o644) + + with pytest.raises( + installer.SpeakersAnalyzeHelperInstallError, + match="executable is not executable", + ): + installer.assert_helper_installed( + f"{installer.HELPER_DIST_NAME}==7.8.9", + python=python, + version_reader=lambda _dist_name: "7.8.9", + executable_predicate=lambda _path: False, + ) diff --git a/tests/test_sense.py b/tests/test_sense.py index 73baaf26d..7fdb510be 100644 --- a/tests/test_sense.py +++ b/tests/test_sense.py @@ -2874,12 +2874,12 @@ def test_main_speakers_analyze_failure_prints_canonical_message_once( ): from solstone.observe import sense from solstone.think.speakers_analyze_installation import ( - SPEAKERS_ANALYZE_REPAIR_TEXT, + speakers_analyze_repair_text, ) message = ( "Speakers-analyze installation is incomplete " - f"(asset-missing: wespeaker). {SPEAKERS_ANALYZE_REPAIR_TEXT}" + f"(asset-missing: wespeaker). {speakers_analyze_repair_text()}" ) monkeypatch.setenv("SOLSTONE_JOURNAL", str(tmp_path)) diff --git a/tests/test_speakers_analyze_helper_makefile_contract.py b/tests/test_speakers_analyze_helper_makefile_contract.py new file mode 100644 index 000000000..920501203 --- /dev/null +++ b/tests/test_speakers_analyze_helper_makefile_contract.py @@ -0,0 +1,55 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc + +from __future__ import annotations + +import re +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[1] +MAKEFILE = REPO_ROOT / "Makefile" + + +def _makefile_text() -> str: + return MAKEFILE.read_text(encoding="utf-8") + + +def _recipe_body(makefile: str, target: str) -> str: + match = re.search( + rf"^{re.escape(target)}:[^\n]*\n(?P(?:\t[^\n]*\n)+)", + makefile, + re.MULTILINE, + ) + assert match is not None, f"Makefile missing {target} recipe" + return match.group("body") + + +def test_speakers_analyze_helper_target_is_phony_and_defined() -> None: + makefile = _makefile_text() + phony = next( + line for line in makefile.splitlines() if line.startswith(".PHONY:") + ).split() + + assert "speakers-analyze-helper" in phony + body = _recipe_body(makefile, "speakers-analyze-helper") + assert 'test -x "$(VENV_BIN)/python"' in body + assert "$(VENV_BIN)/python scripts/install_speakers_analyze_helper.py" in body + + +def test_install_invokes_helper_after_platform_block_before_install_models() -> None: + body = _recipe_body(_makefile_text(), "install") + + platform_block_end = body.index("\tfi\n") + helper_index = body.index("$(MAKE) speakers-analyze-helper") + install_models_index = body.index("$(VENV_BIN)/journal install-models") + + assert platform_block_end < helper_index < install_models_index + + helper_line = next( + line for line in body.splitlines() if "$(MAKE) speakers-analyze-helper" in line + ) + assert helper_line.startswith("\t@$(MAKE) speakers-analyze-helper") + command = helper_line[1:] + assert not command.startswith("-") + assert not command.startswith("@-") + assert "|| true" not in helper_line diff --git a/tests/test_speakers_analyze_installation.py b/tests/test_speakers_analyze_installation.py index a784b9f18..04d92a2bb 100644 --- a/tests/test_speakers_analyze_installation.py +++ b/tests/test_speakers_analyze_installation.py @@ -154,6 +154,46 @@ def test_helper_path_is_sibling_of_python_executable(tmp_path: Path): ) +def test_packaged_repair_text_preserves_existing_plain_prose( + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setattr(installation, "is_source_checkout", lambda: False) + + assert installation.speakers_analyze_repair_text() == ( + "Repair: reinstall the journal host stack with solstone-journal, or " + "solstone-journal-cuda on NVIDIA hosts, and restart the journal." + ) + + +def test_source_checkout_repair_text_names_make_targets( + monkeypatch: pytest.MonkeyPatch, +): + monkeypatch.setattr(installation, "is_source_checkout", lambda: True) + + text = installation.speakers_analyze_repair_text() + + assert "make speakers-analyze-helper" in text + assert "make install" in text + assert "`" not in text + + +def test_result_message_resolves_repair_text_at_access_time( + monkeypatch: pytest.MonkeyPatch, +): + result = installation.SpeakersAnalyzeInstallationResult( + "asset-missing", "wespeaker" + ) + monkeypatch.setattr(installation, "is_source_checkout", lambda: False) + packaged = result.message + + monkeypatch.setattr(installation, "is_source_checkout", lambda: True) + source = result.message + + assert installation.PACKAGED_SPEAKERS_ANALYZE_REPAIR_TEXT in packaged + assert installation.SOURCE_CHECKOUT_SPEAKERS_ANALYZE_REPAIR_TEXT in source + assert packaged != source + + def test_missing_helper_distribution_metadata(tmp_path: Path): def version_reader(dist_name: str) -> str: if dist_name == installation.ROOT_DIST_NAME: diff --git a/tests/test_think_utils.py b/tests/test_think_utils.py index 3f159aec2..66ccec026 100644 --- a/tests/test_think_utils.py +++ b/tests/test_think_utils.py @@ -24,6 +24,7 @@ from solstone.think.utils import ( get_journal, get_journal_info, get_project_root, + is_source_checkout, iter_segments, resolve_segment_dir, segment_key, @@ -1168,6 +1169,49 @@ class TestJournalResolution: assert isinstance(excinfo.value.error, PermissionError) +class TestSourceCheckoutDetection: + def fake_utils_file(self, root: Path) -> str: + utils_file = root / "solstone" / "think" / "utils.py" + utils_file.parent.mkdir(parents=True) + utils_file.touch() + return str(utils_file) + + def test_is_source_checkout_is_cwd_independent_when_git_present( + self, monkeypatch, tmp_path + ): + import solstone.think.utils as utils + + root = tmp_path / "repo" + root.mkdir() + (root / "pyproject.toml").write_text("[project]\n", encoding="utf-8") + (root / ".git").mkdir() + unrelated = tmp_path / "elsewhere" + unrelated.mkdir() + monkeypatch.setattr(utils, "__file__", self.fake_utils_file(root)) + + monkeypatch.chdir(root) + assert is_source_checkout() + monkeypatch.chdir(unrelated) + assert is_source_checkout() + + def test_is_source_checkout_is_cwd_independent_without_git( + self, monkeypatch, tmp_path + ): + import solstone.think.utils as utils + + root = tmp_path / "repo" + root.mkdir() + (root / "pyproject.toml").write_text("[project]\n", encoding="utf-8") + unrelated = tmp_path / "elsewhere" + unrelated.mkdir() + monkeypatch.setattr(utils, "__file__", self.fake_utils_file(root)) + + monkeypatch.chdir(root) + assert not is_source_checkout() + monkeypatch.chdir(unrelated) + assert not is_source_checkout() + + class TestGetJournalInfoConfigBranch: def write_config(self, home: Path, content: str) -> Path: cfg = home / ".config" / "solstone" / "config.toml" -- 2.51.2 From dad8b4a155a912dd222c192191947c8b5a88c832 Mon Sep 17 00:00:00 2001 From: Jer Miller Date: Fri, 31 Jul 2026 16:59:57 -0600 Subject: [PATCH 4/6] fix(install): speakers-analyze helper installer fails with specific messages Malformed requirement strings in the derived pin now raise the installer error instead of surfacing an InvalidRequirement traceback during make install. The malformed-metadata error names what project.dependencies actually contained, and the interpreter guard that binds marker evaluation to the target venv now has negative test coverage. --- scripts/install_speakers_analyze_helper.py | 29 ++++++-- tests/test_install_speakers_analyze_helper.py | 67 +++++++++++++++++++ 2 files changed, 89 insertions(+), 7 deletions(-) diff --git a/scripts/install_speakers_analyze_helper.py b/scripts/install_speakers_analyze_helper.py index 299e76364..d24d023c6 100644 --- a/scripts/install_speakers_analyze_helper.py +++ b/scripts/install_speakers_analyze_helper.py @@ -17,7 +17,7 @@ from importlib.metadata import version as distribution_version from pathlib import Path from packaging.markers import Marker, default_environment -from packaging.requirements import Requirement +from packaging.requirements import InvalidRequirement, Requirement from solstone.think.speakers_analyze_installation import ( HELPER_DIST_NAME, @@ -56,12 +56,17 @@ def read_project_dependencies(pyproject_path: Path) -> list[str]: f"missing package metadata: {pyproject_path}" ) from exc dependencies = data.get("project", {}).get("dependencies") - if not isinstance(dependencies, list) or not all( - isinstance(item, str) for item in dependencies - ): + if not isinstance(dependencies, list): raise SpeakersAnalyzeHelperInstallError( - f"{pyproject_path} project.dependencies must be a list of strings" + f"{pyproject_path} project.dependencies must be a list of strings; " + f"found {type(dependencies).__name__}: {dependencies!r}" ) + for item in dependencies: + if not isinstance(item, str): + raise SpeakersAnalyzeHelperInstallError( + f"{pyproject_path} project.dependencies must be a list of strings; " + f"found non-string entry {type(item).__name__}: {item!r}" + ) return dependencies @@ -73,7 +78,12 @@ def derive_helper_pin( markers: list[Marker] = [] for raw in dependencies: - requirement = Requirement(raw) + try: + requirement = Requirement(raw) + except InvalidRequirement as exc: + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} contains invalid requirement {raw!r}" + ) from exc if requirement.name != HELPER_DIST_NAME: continue raw_requirements.append(raw) @@ -248,7 +258,12 @@ def main(argv: Sequence[str] | None = None) -> int: def _expected_version(pin: str, *, dependency_label: str) -> str: - requirement = Requirement(pin) + try: + requirement = Requirement(pin) + except InvalidRequirement as exc: + raise SpeakersAnalyzeHelperInstallError( + f"{dependency_label} contains invalid requirement {pin!r}" + ) from exc specifiers = tuple(requirement.specifier) if ( requirement.name != HELPER_DIST_NAME diff --git a/tests/test_install_speakers_analyze_helper.py b/tests/test_install_speakers_analyze_helper.py index d89e26b2d..6599732c8 100644 --- a/tests/test_install_speakers_analyze_helper.py +++ b/tests/test_install_speakers_analyze_helper.py @@ -62,6 +62,44 @@ def test_derive_helper_pin_rejects_zero_helper_pins() -> None: installer.derive_helper_pin(["onnxruntime>=1.25.0"]) +def test_derive_helper_pin_reports_invalid_requirement() -> None: + raw = f"{installer.HELPER_DIST_NAME} ==" + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.derive_helper_pin([raw]) + + message = str(exc_info.value) + assert "project.dependencies contains invalid requirement" in message + assert repr(raw) in message + + +def test_read_project_dependencies_reports_missing_dependencies(tmp_path: Path) -> None: + pyproject = tmp_path / "pyproject.toml" + pyproject.write_text('[project]\nname = "fixture"\n', encoding="utf-8") + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.read_project_dependencies(pyproject) + + message = str(exc_info.value) + assert "project.dependencies must be a list of strings" in message + assert "found NoneType: None" in message + + +def test_read_project_dependencies_reports_non_string_entry(tmp_path: Path) -> None: + pyproject = tmp_path / "pyproject.toml" + pyproject.write_text( + '[project]\ndependencies = ["onnxruntime>=1.25.0", 42]\n', + encoding="utf-8", + ) + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.read_project_dependencies(pyproject) + + message = str(exc_info.value) + assert "project.dependencies must be a list of strings" in message + assert "found non-string entry int: 42" in message + + def test_uncovered_environment_skips_without_uv(tmp_path: Path, capsys) -> None: root = tmp_path / "repo" _write_leaf_pyproject( @@ -137,6 +175,18 @@ def test_install_uses_no_config_no_deps_target_python_and_derived_pin( ] +def test_running_python_mismatch_reports_expected_and_actual(tmp_path: Path) -> None: + target_python = _venv_python(tmp_path / "repo") + running_python = tmp_path / "other" / "python" + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.ensure_running_target_python(running_python, target_python) + + message = str(exc_info.value) + assert str(target_python.resolve()) in message + assert str(running_python.resolve()) in message + + def test_missing_uv_fails_loudly(tmp_path: Path) -> None: root = tmp_path / "repo" _write_leaf_pyproject(root, [_covered_pin()]) @@ -166,6 +216,23 @@ def test_assert_helper_installed_reports_version_mismatch(tmp_path: Path) -> Non ) +def test_assert_helper_installed_reports_invalid_pin_requirement( + tmp_path: Path, +) -> None: + pin = f"{installer.HELPER_DIST_NAME} ==" + + with pytest.raises(installer.SpeakersAnalyzeHelperInstallError) as exc_info: + installer.assert_helper_installed( + pin, + python=_venv_python(tmp_path), + version_reader=lambda _dist_name: "7.8.9", + ) + + message = str(exc_info.value) + assert "installed helper pin contains invalid requirement" in message + assert repr(pin) in message + + def test_assert_helper_installed_reports_missing_binary(tmp_path: Path) -> None: with pytest.raises( installer.SpeakersAnalyzeHelperInstallError, -- 2.51.2 From a788551edbffa6232f64c284fd0d525b06f5dddf Mon Sep 17 00:00:00 2001 From: Jer Miller Date: Fri, 31 Jul 2026 17:23:26 -0600 Subject: [PATCH 5/6] feat(rust): guard the workspace-owned SPL tag pin Add a repository guard that derives the approved SPL source and tag from core/Cargo.toml [workspace.dependencies], keeping the tag value out of the guard as a literal. Bind both core/Cargo.lock SPL records to that derived tag and to one shared resolved commit. Require member manifests to inherit via workspace = true, resolving Cargo package identity through the package field so aliases and unlisted manifests cannot evade the check. Reject local-route overrides by Cargo package identity, including workspace patch entries, legacy replace entries, Cargo source replacement, and tracked in-tree SPL package copies. The existing pin, lockfile, and cargo-deny source rail are unchanged. The guard is wired into install-checks, so make ci and make verify both cover it. --- Makefile | 9 +- docs/PORTING.md | 1 + scripts/check_spl_dependency_pin.py | 605 ++++++++++++++++++++++ tests/test_spl_dependency_pin_guard.py | 662 +++++++++++++++++++++++++ 4 files changed, 1276 insertions(+), 1 deletion(-) create mode 100644 scripts/check_spl_dependency_pin.py create mode 100644 tests/test_spl_dependency_pin_guard.py diff --git a/Makefile b/Makefile index 51d1e8bdb..d9ea98881 100644 --- a/Makefile +++ b/Makefile @@ -14,7 +14,7 @@ export TMPDIR := /var/tmp PYTEST_BASETEMP_INIT := BASETEMP=$$(mktemp -d /var/tmp/solstone-pytest-XXXXXX); trap 'rm -rf "$$BASETEMP"' EXIT INT TERM; PYTEST_BASETEMP_FLAG := --basetemp "$$BASETEMP" -.PHONY: install hopper-install uninstall test test-cov test-integration test-release release-checks test-performance test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging check-rust-fmt check-rust-msrv check-rust-clippy check-rust-test check-rust-ios check-rust-deny check-release-advisory-liveness check-rust-release-manifest audit openapi check-openapi check-openapi-observer-client-contract contract check-contract journal-resolution-vectors check-journal-resolution-vectors build-native-sol-grammar-oracle check-native-sol-grammar-oracle build-native-sol-root-contract check-native-sol-root-contract check-core-sdist-compile-inputs build-native-sol-journal-host-commands check-native-sol-journal-host-commands build-journal-access-rejection-inventory check-journal-access-rejection-inventory check-native-sol-python-manifest build-native-sol-inventory check-native-sol-inventory check-native-sol-architecture check-native-sol-contract-routes check-native-sol-conformance check-native-sol-coverage check-native-sol-no-python-spawn check-native-sol-compat check-native-sol-docs-links check-removed-time-parser-ready dev all sandbox sandbox-stop install-models speakers-analyze-helper parakeet-helper parakeet-helper-clean wheel-speakers-analyze-linux wheel-speakers-analyze-linux-x86_64 wheel-speakers-analyze-linux-aarch64 wheel-macos wheel-macos-clean verify verify-api verify-schemathesis update-api-baselines eval-schemas service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-journal-config-owner check-call-http-only check-no-legacy-chat check-channel-adapter-scrub check-brain-health-cutover check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-schema-bounds check-thin-base-install check-extras-consistency check-cogitate-prompts smoke-cogitate release release-test publish-release publish-release-test FORCE +.PHONY: install hopper-install uninstall test test-cov test-integration test-release release-checks test-performance test-app test-only format format-check install-checks ci clean clean-install coverage watch versions update update-prices preflight pre-commit skills render-packaging check-rust-fmt check-rust-msrv check-rust-clippy check-rust-test check-rust-ios check-rust-deny check-release-advisory-liveness check-rust-release-manifest check-spl-dependency-pin audit openapi check-openapi check-openapi-observer-client-contract contract check-contract journal-resolution-vectors check-journal-resolution-vectors build-native-sol-grammar-oracle check-native-sol-grammar-oracle build-native-sol-root-contract check-native-sol-root-contract check-core-sdist-compile-inputs build-native-sol-journal-host-commands check-native-sol-journal-host-commands build-journal-access-rejection-inventory check-journal-access-rejection-inventory check-native-sol-python-manifest build-native-sol-inventory check-native-sol-inventory check-native-sol-architecture check-native-sol-contract-routes check-native-sol-conformance check-native-sol-coverage check-native-sol-no-python-spawn check-native-sol-compat check-native-sol-docs-links check-removed-time-parser-ready dev all sandbox sandbox-stop install-models speakers-analyze-helper parakeet-helper parakeet-helper-clean wheel-speakers-analyze-linux wheel-speakers-analyze-linux-x86_64 wheel-speakers-analyze-linux-aarch64 wheel-macos wheel-macos-clean verify verify-api verify-schemathesis update-api-baselines eval-schemas service-logs check-layer-hygiene check-api-conventions check-journal-io-access check-journal-io-mechanic check-journal-config-owner check-call-http-only check-no-legacy-chat check-channel-adapter-scrub check-brain-health-cutover check-tools-http-only check-access-imports-clean check-convey-bind-imports-clean check-schema-bounds check-thin-base-install check-extras-consistency check-cogitate-prompts smoke-cogitate release release-test publish-release publish-release-test FORCE # Default target - install package in editable mode all: install @@ -599,6 +599,9 @@ install-checks: .installed @echo "=== Running rust release-manifest check ===" @$(MAKE) check-rust-release-manifest @echo "" + @echo "=== Running SPL dependency-pin check ===" + @$(MAKE) check-spl-dependency-pin + @echo "" @echo "=== Checking conversion-wave retirements ===" @$(MAKE) check-conversion-retirements @echo "=== Running access-imports-clean check ===" @@ -799,6 +802,10 @@ check-schema-bounds: .installed check-rust-release-manifest: .installed $(VENV_BIN)/python scripts/check_rust_release_manifest.py +# SPL git dependency pin guard +check-spl-dependency-pin: + python3 scripts/check_spl_dependency_pin.py + # Conversion-wave Python and package retirement gate check-conversion-retirements: python3 scripts/check_conversion_retirements.py diff --git a/docs/PORTING.md b/docs/PORTING.md index be5f8c15b..307386cd3 100644 --- a/docs/PORTING.md +++ b/docs/PORTING.md @@ -96,6 +96,7 @@ for the helper release lanes. | Rust lint | `make check-rust-clippy` | GNU-host check | Runs the existing clippy `-D warnings` gate. | | Rust tests | `make check-rust-test` | GNU-host check | Runs workspace Rust tests on the GNU host. | | Rust dependency policy | `make check-rust-deny` | GNU-host check | Locked, offline bans/licenses/sources policy over the supported cargo-deny graph. | +| SPL dependency pin | `make check-spl-dependency-pin` | GNU-host check | Verifies the Rust core workspace resolves `spl-core` and `spl-transport` only through the workspace-owned `spl-rust` tag pin, with member manifests inheriting it, lockfile binding intact, and local patch/source replacement routes rejected. | | Rust advisories | `make audit` | GNU-host check | Verifies a signed advisory mirror packet, materializes its bundle locally, then performs a locked offline advisory check without refreshing or mutating the operator inputs. | | iOS canary | `make check-rust-ios` | iOS cross-target canary | Cross-target drift evidence for eligible library crates; explicitly excludes `solstone-core-indexer-store` because the native SQLite store is not yet in the iOS gate, and `solstone-core-speakers-analyze` plus `solstone-core-speakers-onnx` because the analyzer transitively depends on ONNX Runtime host-only native linkage. | | Core sdist compile inputs | `make check-core-sdist-compile-inputs` | Packaging-source check | Verifies shipping Rust compile-time inputs are discovered and covered by the normalized `solstone-core` sdist injection set. | diff --git a/scripts/check_spl_dependency_pin.py b/scripts/check_spl_dependency_pin.py new file mode 100644 index 000000000..e3b208498 --- /dev/null +++ b/scripts/check_spl_dependency_pin.py @@ -0,0 +1,605 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc + +"""Guard the Rust core workspace-owned SPL dependency pin.""" + +from __future__ import annotations + +import argparse +import re +import subprocess +import sys +import tomllib +from collections.abc import Iterator, Mapping +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parent.parent + +APPROVED_SOURCE_URL = "https://github.com/solpbc/spl-rust" +SPL_PACKAGES = ("spl-core", "spl-transport") +SPL_PACKAGE_SET = set(SPL_PACKAGES) + +DEPENDENCY_KINDS = ("dependencies", "dev-dependencies", "build-dependencies") +MEMBER_OVERRIDE_KEYS = ("git", "tag", "rev", "branch", "version", "path") +WORKSPACE_SELECTOR_KEYS = ("rev", "branch", "version", "path") +CONFIG_REPLACEMENT_KEYS = ("replace-with", "directory", "local-registry", "registry") +CONFIG_PATHS = ( + ".cargo/config.toml", + ".cargo/config", + "core/.cargo/config.toml", + "core/.cargo/config", +) +LOCK_COMMIT_RE = re.compile(r"^[0-9a-f]{40}$") + +W001_WORKSPACE_DEPENDENCY_MISSING = ( + "spl-pin W001 workspace dependency missing: {package} is absent from " + "core/Cargo.toml [workspace.dependencies]; repair by declaring {package} " + f"from {APPROVED_SOURCE_URL} with the shared tag." +) +W002_WORKSPACE_DEPENDENCY_TABLE = ( + "spl-pin W002 workspace dependency must be a table: {package} is a bare " + "version string in core/Cargo.toml [workspace.dependencies]; repair by " + f"using a git dependency table with URL {APPROVED_SOURCE_URL} and the " + "shared tag." +) +W003_WORKSPACE_SOURCE_URL = ( + "spl-pin W003 workspace source URL mismatch: {package} uses git source " + f"{{found}}; repair by using {APPROVED_SOURCE_URL}." +) +W004_WORKSPACE_SELECTOR_TAG_ONLY = ( + "spl-pin W004 workspace selector must be tag-only: {package} declares " + "{keys}; repair by removing rev/branch/version/path selectors and using " + "only tag in the workspace dependency entry." +) +W005_WORKSPACE_TAG_EMPTY = ( + "spl-pin W005 workspace tag is empty: {package} has an empty tag value; " + "repair by setting the shared non-empty SPL tag in core/Cargo.toml." +) +W006_WORKSPACE_TAGS_SPLIT = ( + "spl-pin W006 workspace SPL tags split: spl-core and spl-transport use " + "different tags; repair by setting both workspace entries to the same tag." +) +W007_WORKSPACE_ALIAS = ( + "spl-pin W007 workspace dependency alias touches SPL package: {dependency} " + "resolves to {package}; repair by declaring SPL packages only under " + "canonical keys spl-core and spl-transport without package aliases." +) + +M001_MEMBER_OVERRIDE = ( + "spl-pin M001 member SPL dependency overrides workspace pin: {manifest} " + "{table}.{dependency} resolves to {package} and declares {keys}; repair by " + "removing git/tag/rev/branch/version/path and using workspace = true for " + "the SPL dependency." +) + +L001_LOCK_PACKAGE_MISSING = ( + "spl-pin L001 lockfile package missing: core/Cargo.lock has no [[package]] " + "record for {package}; repair by regenerating the lockfile from the " + "workspace SPL tag pin." +) +L002_LOCK_PACKAGE_DUPLICATED = ( + "spl-pin L002 lockfile package duplicated: core/Cargo.lock has {count} " + "[[package]] records for {package}; repair by resolving the graph to a " + "single {package} record from the workspace SPL tag pin." +) +L003_LOCK_SOURCE_MISSING = ( + "spl-pin L003 lockfile source missing: {package} has no source key in " + "core/Cargo.lock; repair by resolving it from the approved spl-rust git " + "tag, not a path/workspace package." +) +L004_LOCK_SOURCE_NOT_GIT = ( + "spl-pin L004 lockfile source is not git: {package} source is {source}; " + f"repair by resolving it from {APPROVED_SOURCE_URL} with the workspace tag." +) +L005_LOCK_GIT_URL = ( + "spl-pin L005 lockfile git URL mismatch: {package} source URL is {url}; " + f"repair by resolving it from {APPROVED_SOURCE_URL}." +) +L006_LOCK_SELECTOR_TAG = ( + "spl-pin L006 lockfile selector must be tag: {package} source selector is " + "{selector}; repair by resolving it with tag from the workspace SPL pin." +) +L007_LOCK_TAG_WORKSPACE = ( + "spl-pin L007 lockfile tag disagrees with workspace: {package} lock tag is " + "{lock_tag}; repair by regenerating core/Cargo.lock so it matches the " + "workspace SPL tag." +) +L008_LOCK_COMMIT_INVALID = ( + "spl-pin L008 lockfile commit fragment invalid: {package} source commit is " + "{commit}; repair by regenerating core/Cargo.lock with a git source ending " + "in a 40-character lowercase hex commit." +) +L009_LOCK_COMMITS_SPLIT = ( + "spl-pin L009 lockfile SPL commits split: spl-core and spl-transport " + "resolve to different commits; repair by regenerating core/Cargo.lock so " + "both packages resolve to the same spl-rust commit." +) + +R001_WORKSPACE_PATCH_SOURCE = ( + "spl-pin R001 workspace patch rewrites approved source: core/Cargo.toml " + f"[patch.{{source}}] targets {APPROVED_SOURCE_URL}; repair by removing the " + "SPL patch route and using the workspace tag pin." +) +R002_WORKSPACE_PATCH_PACKAGE = ( + "spl-pin R002 workspace patch supplies SPL package: core/Cargo.toml " + "[patch.{source}] entry {dependency} resolves to {package}; repair by " + "removing the SPL patch entry and using the workspace tag pin." +) +R003_REPLACE_PACKAGE = ( + "spl-pin R003 legacy replace targets SPL package: core/Cargo.toml " + "[replace] key {replace_key} names {package}; repair by removing the SPL " + "replace entry and using the workspace tag pin." +) +R004_CONFIG_SOURCE_REPLACEMENT = ( + "spl-pin R004 Cargo source replacement rewrites approved source: " + f"{{config_path}} [source.{{source_name}}] targets {APPROVED_SOURCE_URL} " + "and declares {keys}; repair by removing the source replacement route and " + "using the workspace tag pin." +) +R005_CONFIG_PATCH_SOURCE = ( + "spl-pin R005 Cargo config patch rewrites approved source: {config_path} " + f"[patch.{{source}}] targets {APPROVED_SOURCE_URL}; repair by removing the " + "SPL patch route and using the workspace tag pin." +) +R006_CONFIG_PATCH_PACKAGE = ( + "spl-pin R006 Cargo config patch supplies SPL package: {config_path} " + "[patch.{source}] entry {dependency} resolves to {package}; repair by " + "removing the SPL patch entry and using the workspace tag pin." +) +R007_IN_TREE_PACKAGE_COPY = ( + "spl-pin R007 tracked in-tree SPL package copy: {manifest} declares package " + "name {package}; repair by removing the in-tree SPL implementation copy and " + "depending on the approved spl-rust tag pin." +) + + +def load_toml(path: Path) -> dict[str, Any]: + with path.open("rb") as handle: + return tomllib.load(handle) + + +def as_mapping(value: object) -> Mapping[str, Any] | None: + if isinstance(value, Mapping): + return value + return None + + +def format_value(value: object) -> str: + if value is None: + return "" + if isinstance(value, str): + return value + return repr(value) + + +def format_keys(keys: list[str]) -> str: + return ", ".join(keys) + + +def tracked_cargo_manifests(root: Path) -> list[Path]: + result = subprocess.run( + ["git", "ls-files", "--", "*Cargo.toml"], + cwd=root, + check=True, + capture_output=True, + text=True, + ) + return [root / line for line in result.stdout.splitlines() if line] + + +def dependency_identity(name: str, spec: object) -> str: + table = as_mapping(spec) + if table is None: + return name + package = table.get("package") + if isinstance(package, str): + return package + return name + + +def selector_keys(spec: object) -> list[str]: + if isinstance(spec, str): + return ["version"] + table = as_mapping(spec) + if table is None: + return [] + return [key for key in MEMBER_OVERRIDE_KEYS if key in table] + + +def iter_dependency_tables( + manifest: Mapping[str, Any], +) -> Iterator[tuple[str, Mapping[str, Any]]]: + for kind in DEPENDENCY_KINDS: + table = as_mapping(manifest.get(kind)) + if table is not None: + yield kind, table + + target = as_mapping(manifest.get("target")) + if target is None: + return + for cfg in sorted(target): + target_table = as_mapping(target[cfg]) + if target_table is None: + continue + for kind in DEPENDENCY_KINDS: + table = as_mapping(target_table.get(kind)) + if table is not None: + yield f"target.{cfg}.{kind}", table + + +def workspace_selector_findings(spec: Mapping[str, Any]) -> list[str]: + keys: list[str] = [] + if "tag" not in spec: + keys.append("missing tag") + keys.extend(key for key in WORKSPACE_SELECTOR_KEYS if key in spec) + return keys + + +def collect_workspace_findings(root: Path) -> tuple[list[str], str | None]: + manifest = load_toml(root / "core" / "Cargo.toml") + workspace = as_mapping(manifest.get("workspace")) or {} + dependencies = as_mapping(workspace.get("dependencies")) or {} + findings: list[str] = [] + tags: dict[str, str] = {} + package_valid = dict.fromkeys(SPL_PACKAGES, True) + + alias_findings: list[str] = [] + for dependency in sorted(dependencies): + spec = dependencies[dependency] + identity = dependency_identity(dependency, spec) + if dependency != identity and ( + dependency in SPL_PACKAGE_SET or identity in SPL_PACKAGE_SET + ): + alias_findings.append( + W007_WORKSPACE_ALIAS.format( + dependency=dependency, + package=identity, + ) + ) + if dependency in SPL_PACKAGE_SET: + package_valid[dependency] = False + + for package in SPL_PACKAGES: + spec = dependencies.get(package) + if spec is None: + findings.append(W001_WORKSPACE_DEPENDENCY_MISSING.format(package=package)) + package_valid[package] = False + continue + if isinstance(spec, str): + findings.append(W002_WORKSPACE_DEPENDENCY_TABLE.format(package=package)) + package_valid[package] = False + continue + + table = as_mapping(spec) + if table is None: + findings.append(W002_WORKSPACE_DEPENDENCY_TABLE.format(package=package)) + package_valid[package] = False + continue + + git_source = table.get("git") + if git_source != APPROVED_SOURCE_URL: + findings.append( + W003_WORKSPACE_SOURCE_URL.format( + package=package, + found=format_value(git_source), + ) + ) + package_valid[package] = False + + selector_problems = workspace_selector_findings(table) + if selector_problems: + findings.append( + W004_WORKSPACE_SELECTOR_TAG_ONLY.format( + package=package, + keys=format_keys(selector_problems), + ) + ) + package_valid[package] = False + + tag = table.get("tag") + if "tag" in table and (not isinstance(tag, str) or not tag): + findings.append(W005_WORKSPACE_TAG_EMPTY.format(package=package)) + package_valid[package] = False + elif isinstance(tag, str) and tag: + tags[package] = tag + + if all(package_valid.values()) and len(set(tags.values())) == 1: + workspace_tag = next(iter(tags.values())) + else: + workspace_tag = None + + if len(tags) == len(SPL_PACKAGES) and len(set(tags.values())) > 1: + findings.append(W006_WORKSPACE_TAGS_SPLIT) + workspace_tag = None + + findings.extend(alias_findings) + return findings, workspace_tag + + +def collect_member_findings(root: Path) -> list[str]: + findings: list[str] = [] + core_root = root / "core" + workspace_manifest = core_root / "Cargo.toml" + for manifest_path in sorted(tracked_cargo_manifests(root)): + if manifest_path == workspace_manifest: + continue + if not manifest_path.is_relative_to(core_root): + continue + manifest = load_toml(manifest_path) + rel_manifest = manifest_path.relative_to(root).as_posix() + for table_name, dependencies in iter_dependency_tables(manifest): + for dependency in sorted(dependencies): + spec = dependencies[dependency] + identity = dependency_identity(dependency, spec) + if identity not in SPL_PACKAGE_SET: + continue + keys = selector_keys(spec) + if not keys: + continue + findings.append( + M001_MEMBER_OVERRIDE.format( + manifest=rel_manifest, + table=table_name, + dependency=dependency, + package=identity, + keys=format_keys(keys), + ) + ) + return findings + + +def lock_packages(lock: Mapping[str, Any], package: str) -> list[Mapping[str, Any]]: + packages = lock.get("package") + if not isinstance(packages, list): + return [] + records: list[Mapping[str, Any]] = [] + for record in packages: + table = as_mapping(record) + if table is not None and table.get("name") == package: + records.append(table) + return records + + +def parse_git_lock_source( + source: str, +) -> tuple[str, str | None, str | None, str | None]: + body = source.removeprefix("git+") + before_hash, separator, commit = body.partition("#") + commit_value = commit if separator else None + url, query_separator, query = before_hash.partition("?") + if not query_separator: + return url, None, None, commit_value + selector, value_separator, value = query.partition("=") + return url, selector, value if value_separator else "", commit_value + + +def collect_lockfile_findings(root: Path, workspace_tag: str | None) -> list[str]: + lock = load_toml(root / "core" / "Cargo.lock") + findings: list[str] = [] + commits: dict[str, str] = {} + + for package in SPL_PACKAGES: + records = lock_packages(lock, package) + if not records: + findings.append(L001_LOCK_PACKAGE_MISSING.format(package=package)) + continue + if len(records) > 1: + findings.append( + L002_LOCK_PACKAGE_DUPLICATED.format( + package=package, + count=len(records), + ) + ) + continue + + source = records[0].get("source") + if source is None: + findings.append(L003_LOCK_SOURCE_MISSING.format(package=package)) + continue + if not isinstance(source, str) or not source.startswith("git+"): + findings.append( + L004_LOCK_SOURCE_NOT_GIT.format( + package=package, + source=format_value(source), + ) + ) + continue + + url, selector, selector_value, commit = parse_git_lock_source(source) + if url != APPROVED_SOURCE_URL: + findings.append(L005_LOCK_GIT_URL.format(package=package, url=url)) + if selector != "tag": + findings.append( + L006_LOCK_SELECTOR_TAG.format( + package=package, + selector=selector or "", + ) + ) + elif workspace_tag is not None and selector_value != workspace_tag: + findings.append( + L007_LOCK_TAG_WORKSPACE.format( + package=package, + lock_tag=selector_value, + ) + ) + + if commit is None or not LOCK_COMMIT_RE.fullmatch(commit): + findings.append( + L008_LOCK_COMMIT_INVALID.format( + package=package, + commit=commit or "", + ) + ) + else: + commits[package] = commit + + if len(commits) == len(SPL_PACKAGES) and len(set(commits.values())) > 1: + findings.append(L009_LOCK_COMMITS_SPLIT) + + return findings + + +def collect_patch_findings( + patch: Mapping[str, Any], + *, + source_message: str, + package_message: str, + config_path: str | None = None, +) -> list[str]: + findings: list[str] = [] + for source in sorted(patch): + entries = as_mapping(patch[source]) + if source == APPROVED_SOURCE_URL: + if config_path is None: + findings.append(source_message.format(source=source)) + else: + findings.append( + source_message.format(config_path=config_path, source=source) + ) + if entries is None: + continue + for dependency in sorted(entries): + spec = entries[dependency] + identity = dependency_identity(dependency, spec) + if identity not in SPL_PACKAGE_SET: + continue + if config_path is None: + findings.append( + package_message.format( + source=source, + dependency=dependency, + package=identity, + ) + ) + else: + findings.append( + package_message.format( + config_path=config_path, + source=source, + dependency=dependency, + package=identity, + ) + ) + return findings + + +def source_name_url(name: str, table: Mapping[str, Any]) -> str | None: + git = table.get("git") + if isinstance(git, str): + return git + if name.startswith("https://") or name.startswith("http://"): + return name + return None + + +def collect_config_findings(root: Path, config_path: Path) -> list[str]: + config = load_toml(config_path) + rel_config = config_path.relative_to(root).as_posix() + findings: list[str] = [] + + patch = as_mapping(config.get("patch")) + if patch is not None: + findings.extend( + collect_patch_findings( + patch, + source_message=R005_CONFIG_PATCH_SOURCE, + package_message=R006_CONFIG_PATCH_PACKAGE, + config_path=rel_config, + ) + ) + + sources = as_mapping(config.get("source")) + if sources is None: + return findings + for source_name in sorted(sources): + source_table = as_mapping(sources[source_name]) + if source_table is None: + continue + url = source_name_url(source_name, source_table) + keys = [key for key in CONFIG_REPLACEMENT_KEYS if key in source_table] + if url == APPROVED_SOURCE_URL and keys: + findings.append( + R004_CONFIG_SOURCE_REPLACEMENT.format( + config_path=rel_config, + source_name=source_name, + keys=format_keys(keys), + ) + ) + return findings + + +def collect_local_route_findings(root: Path) -> list[str]: + findings: list[str] = [] + workspace_manifest = load_toml(root / "core" / "Cargo.toml") + + patch = as_mapping(workspace_manifest.get("patch")) + if patch is not None: + findings.extend( + collect_patch_findings( + patch, + source_message=R001_WORKSPACE_PATCH_SOURCE, + package_message=R002_WORKSPACE_PATCH_PACKAGE, + ) + ) + + replace = as_mapping(workspace_manifest.get("replace")) + if replace is not None: + for replace_key in sorted(replace): + package = str(replace_key).split(":", 1)[0] + if package in SPL_PACKAGE_SET: + findings.append( + R003_REPLACE_PACKAGE.format( + replace_key=replace_key, + package=package, + ) + ) + + for relative_config in CONFIG_PATHS: + config_path = root / relative_config + if config_path.exists(): + findings.extend(collect_config_findings(root, config_path)) + + for manifest_path in sorted(tracked_cargo_manifests(root)): + manifest = load_toml(manifest_path) + package = as_mapping(manifest.get("package")) + package_name = package.get("name") if package is not None else None + if package_name in SPL_PACKAGE_SET: + findings.append( + R007_IN_TREE_PACKAGE_COPY.format( + manifest=manifest_path.relative_to(root).as_posix(), + package=package_name, + ) + ) + + return findings + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description="SPL dependency pin guard") + parser.add_argument( + "--root", + type=Path, + default=ROOT, + help="Repository root to scan (defaults to the checkout root).", + ) + args = parser.parse_args(argv) + + root = args.root.resolve() + workspace_findings, workspace_tag = collect_workspace_findings(root) + findings = [ + *workspace_findings, + *collect_member_findings(root), + *collect_lockfile_findings(root, workspace_tag), + *collect_local_route_findings(root), + ] + + for finding in findings: + print(finding, file=sys.stderr) + + return 1 if findings else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_spl_dependency_pin_guard.py b/tests/test_spl_dependency_pin_guard.py new file mode 100644 index 000000000..a1315b417 --- /dev/null +++ b/tests/test_spl_dependency_pin_guard.py @@ -0,0 +1,662 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc +from __future__ import annotations + +import subprocess +from collections.abc import Callable +from pathlib import Path + +import pytest + +import scripts.check_spl_dependency_pin as guard + +BASE_TAG = "v9.8.7-test" +SECOND_TAG = "v9.8.8-test" +BASE_COMMIT = "0123456789abcdef0123456789abcdef01234567" +OTHER_COMMIT = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + +Mutator = Callable[[Path], None] +ExpectedMessage = Callable[[Path], str] + + +def _git(repo: Path, *args: str) -> str: + result = subprocess.run( + ["git", *args], + cwd=repo, + check=True, + capture_output=True, + text=True, + ) + return result.stdout.strip() + + +def _write(repo: Path, rel_path: str, text: str) -> None: + path = repo / rel_path + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + + +def _read(repo: Path, rel_path: str) -> str: + return (repo / rel_path).read_text(encoding="utf-8") + + +def _replace(repo: Path, rel_path: str, old: str, new: str) -> None: + path = repo / rel_path + source = path.read_text(encoding="utf-8") + assert old in source + path.write_text(source.replace(old, new), encoding="utf-8") + + +def _append(repo: Path, rel_path: str, text: str) -> None: + path = repo / rel_path + source = path.read_text(encoding="utf-8") + path.write_text(f"{source}{text}", encoding="utf-8") + + +def _workspace_spl_line(package: str, tag: str = BASE_TAG) -> str: + return f'{package} = {{ git = "{guard.APPROVED_SOURCE_URL}", tag = "{tag}" }}' + + +def _lock_source(tag: str, commit: str) -> str: + return f"git+{guard.APPROVED_SOURCE_URL}?tag={tag}#{commit}" + + +def _lock_spl_record( + package: str, tag: str = BASE_TAG, commit: str = BASE_COMMIT +) -> str: + return ( + "[[package]]\n" + f'name = "{package}"\n' + 'version = "0.1.0"\n' + f'source = "{_lock_source(tag, commit)}"\n' + "dependencies = []\n" + "\n" + ) + + +def _workspace_manifest(tag: str = BASE_TAG) -> str: + return ( + "[workspace]\n" + 'members = ["crates/member"]\n' + "\n" + "[workspace.dependencies]\n" + f"{_workspace_spl_line('spl-core', tag)}\n" + f"{_workspace_spl_line('spl-transport', tag)}\n" + ) + + +def _member_manifest() -> str: + return ( + "[package]\n" + 'name = "member"\n' + 'version = "0.1.0"\n' + 'edition = "2024"\n' + "\n" + "[dependencies]\n" + "spl-core = { workspace = true }\n" + "spl-transport = { workspace = true }\n" + ) + + +def _lockfile(tag: str = BASE_TAG, commit: str = BASE_COMMIT) -> str: + return ( + "version = 4\n" + "\n" + "[[package]]\n" + 'name = "member"\n' + 'version = "0.1.0"\n' + "dependencies = [\n" + ' "spl-core",\n' + ' "spl-transport",\n' + "]\n" + "\n" + f"{_lock_spl_record('spl-core', tag, commit)}" + f"{_lock_spl_record('spl-transport', tag, commit)}" + ) + + +def _write_valid_repo( + root: Path, + *, + tag: str = BASE_TAG, + commit: str = BASE_COMMIT, +) -> Path: + repo = root / "repo" + repo.mkdir() + _git(repo, "init") + _write(repo, "core/Cargo.toml", _workspace_manifest(tag)) + _write(repo, "core/crates/member/Cargo.toml", _member_manifest()) + _write(repo, "core/Cargo.lock", _lockfile(tag, commit)) + _git(repo, "add", ".") + return repo + + +def _run_guard(repo: Path) -> int: + return guard.main(["--root", str(repo)]) + + +def _expect_failure( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + mutator: Mutator, + expected_message: str, +) -> None: + repo = _write_valid_repo(tmp_path) + mutator(repo) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 1 + assert expected_message in captured.err + + +def test_valid_repo_passes(tmp_path: Path, capsys: pytest.CaptureFixture[str]) -> None: + repo = _write_valid_repo(tmp_path) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 0 + assert captured.err == "" + + +def test_valid_repo_passes_with_derived_second_tag( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path, tag=SECOND_TAG, commit=BASE_COMMIT) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 0 + assert captured.err == "" + + +def test_python_spl_surfaces_do_not_trip_in_tree_package_copy( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path) + _write(repo, "solstone/think/spl/foo.py", "VALUE = 1\n") + _write(repo, "tests/spl/bar.py", "VALUE = 1\n") + _write(repo, "docs/design/spl-fixture.md", "# SPL fixture\n") + _git(repo, "add", ".") + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 0 + assert captured.err == "" + + +@pytest.mark.parametrize( + ("mutator", "expected"), + [ + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + f"{_workspace_spl_line('spl-core')}\n", + "", + ), + lambda _repo: guard.W001_WORKSPACE_DEPENDENCY_MISSING.format( + package="spl-core" + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-core"), + 'spl-core = "0.1.0"', + ), + lambda _repo: guard.W002_WORKSPACE_DEPENDENCY_TABLE.format( + package="spl-core" + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + guard.APPROVED_SOURCE_URL, + "https://example.invalid/spl-rust", + ), + lambda _repo: guard.W003_WORKSPACE_SOURCE_URL.format( + package="spl-core", + found="https://example.invalid/spl-rust", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-core"), + ( + f'spl-core = {{ git = "{guard.APPROVED_SOURCE_URL}", ' + f'rev = "{BASE_COMMIT}" }}' + ), + ), + lambda _repo: guard.W004_WORKSPACE_SELECTOR_TAG_ONLY.format( + package="spl-core", + keys="missing tag, rev", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-core"), + f'spl-core = {{ git = "{guard.APPROVED_SOURCE_URL}", tag = "" }}', + ), + lambda _repo: guard.W005_WORKSPACE_TAG_EMPTY.format(package="spl-core"), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-transport"), + _workspace_spl_line("spl-transport", SECOND_TAG), + ), + lambda _repo: guard.W006_WORKSPACE_TAGS_SPLIT, + ), + ( + lambda repo: _append( + repo, + "core/Cargo.toml", + ( + f'myspl = {{ package = "spl-core", ' + f'git = "{guard.APPROVED_SOURCE_URL}", tag = "{BASE_TAG}" }}\n' + ), + ), + lambda _repo: guard.W007_WORKSPACE_ALIAS.format( + dependency="myspl", + package="spl-core", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-core"), + ( + f'spl-core = {{ package = "other", ' + f'git = "{guard.APPROVED_SOURCE_URL}", tag = "{BASE_TAG}" }}' + ), + ), + lambda _repo: guard.W007_WORKSPACE_ALIAS.format( + dependency="spl-core", + package="other", + ), + ), + ], +) +def test_workspace_properties_fail( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + mutator: Mutator, + expected: ExpectedMessage, +) -> None: + _expect_failure(tmp_path, capsys, mutator, expected(tmp_path)) + + +def test_absent_workspace_tag_reports_w004_without_w005( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path) + _replace( + repo, + "core/Cargo.toml", + _workspace_spl_line("spl-core"), + f'spl-core = {{ git = "{guard.APPROVED_SOURCE_URL}", rev = "{BASE_COMMIT}" }}', + ) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 1 + assert ( + guard.W004_WORKSPACE_SELECTOR_TAG_ONLY.format( + package="spl-core", + keys="missing tag, rev", + ) + in captured.err + ) + assert guard.W005_WORKSPACE_TAG_EMPTY.format(package="spl-core") not in captured.err + + +@pytest.mark.parametrize( + ("mutator", "expected"), + [ + ( + lambda repo: _replace( + repo, + "core/crates/member/Cargo.toml", + "spl-core = { workspace = true }", + 'spl-core = "0.1.0"', + ), + lambda _repo: guard.M001_MEMBER_OVERRIDE.format( + manifest="core/crates/member/Cargo.toml", + table="dependencies", + dependency="spl-core", + package="spl-core", + keys="version", + ), + ), + ( + lambda repo: _append( + repo, + "core/crates/member/Cargo.toml", + ( + f'\nmyspl = {{ package = "spl-core", ' + f'git = "{guard.APPROVED_SOURCE_URL}", tag = "{BASE_TAG}" }}\n' + ), + ), + lambda _repo: guard.M001_MEMBER_OVERRIDE.format( + manifest="core/crates/member/Cargo.toml", + table="dependencies", + dependency="myspl", + package="spl-core", + keys="git, tag", + ), + ), + ( + lambda repo: _append( + repo, + "core/crates/member/Cargo.toml", + "\n[target.'cfg(unix)'.build-dependencies]\n" + 'spl-transport = { path = "../spl-transport" }\n', + ), + lambda _repo: guard.M001_MEMBER_OVERRIDE.format( + manifest="core/crates/member/Cargo.toml", + table="target.cfg(unix).build-dependencies", + dependency="spl-transport", + package="spl-transport", + keys="path", + ), + ), + ], +) +def test_member_properties_fail( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + mutator: Mutator, + expected: ExpectedMessage, +) -> None: + _expect_failure(tmp_path, capsys, mutator, expected(tmp_path)) + + +@pytest.mark.parametrize( + ("mutator", "expected"), + [ + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + _lock_spl_record("spl-core"), + "", + ), + lambda _repo: guard.L001_LOCK_PACKAGE_MISSING.format(package="spl-core"), + ), + ( + lambda repo: _append(repo, "core/Cargo.lock", _lock_spl_record("spl-core")), + lambda _repo: guard.L002_LOCK_PACKAGE_DUPLICATED.format( + package="spl-core", + count=2, + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + f'source = "{_lock_source(BASE_TAG, BASE_COMMIT)}"\n', + "", + ), + lambda _repo: guard.L003_LOCK_SOURCE_MISSING.format(package="spl-core"), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + _lock_source(BASE_TAG, BASE_COMMIT), + "registry+https://github.com/rust-lang/crates.io-index", + ), + lambda _repo: guard.L004_LOCK_SOURCE_NOT_GIT.format( + package="spl-core", + source="registry+https://github.com/rust-lang/crates.io-index", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + guard.APPROVED_SOURCE_URL, + "https://example.invalid/spl-rust", + ), + lambda _repo: guard.L005_LOCK_GIT_URL.format( + package="spl-core", + url="https://example.invalid/spl-rust", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + f"?tag={BASE_TAG}", + f"?rev={BASE_COMMIT}", + ), + lambda _repo: guard.L006_LOCK_SELECTOR_TAG.format( + package="spl-core", + selector="rev", + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + f"?tag={BASE_TAG}", + f"?tag={SECOND_TAG}", + ), + lambda _repo: guard.L007_LOCK_TAG_WORKSPACE.format( + package="spl-core", + lock_tag=SECOND_TAG, + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + BASE_COMMIT, + BASE_COMMIT[:-1], + ), + lambda _repo: guard.L008_LOCK_COMMIT_INVALID.format( + package="spl-core", + commit=BASE_COMMIT[:-1], + ), + ), + ( + lambda repo: _replace( + repo, + "core/Cargo.lock", + _lock_spl_record("spl-transport"), + _lock_spl_record("spl-transport", BASE_TAG, OTHER_COMMIT), + ), + lambda _repo: guard.L009_LOCK_COMMITS_SPLIT, + ), + ], +) +def test_lockfile_properties_fail( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + mutator: Mutator, + expected: ExpectedMessage, +) -> None: + _expect_failure(tmp_path, capsys, mutator, expected(tmp_path)) + + +@pytest.mark.parametrize( + ("mutator", "expected"), + [ + ( + lambda repo: _append( + repo, + "core/Cargo.toml", + f'\n[patch."{guard.APPROVED_SOURCE_URL}"]\n', + ), + lambda _repo: guard.R001_WORKSPACE_PATCH_SOURCE.format( + source=guard.APPROVED_SOURCE_URL + ), + ), + ( + lambda repo: _append( + repo, + "core/Cargo.toml", + '\n[patch.crates-io]\nspl-core = { path = "../spl-core" }\n', + ), + lambda _repo: guard.R002_WORKSPACE_PATCH_PACKAGE.format( + source="crates-io", + dependency="spl-core", + package="spl-core", + ), + ), + ( + lambda repo: _append( + repo, + "core/Cargo.toml", + '\n[replace]\n"spl-core:0.1.0" = { path = "../spl-core" }\n', + ), + lambda _repo: guard.R003_REPLACE_PACKAGE.format( + replace_key="spl-core:0.1.0", + package="spl-core", + ), + ), + ( + lambda repo: _write( + repo, + ".cargo/config.toml", + ( + "[source.spl-rust]\n" + f'git = "{guard.APPROVED_SOURCE_URL}"\n' + 'replace-with = "local-spl"\n' + "\n" + "[source.local-spl]\n" + 'directory = "vendor"\n' + ), + ), + lambda _repo: guard.R004_CONFIG_SOURCE_REPLACEMENT.format( + config_path=".cargo/config.toml", + source_name="spl-rust", + keys="replace-with", + ), + ), + ( + lambda repo: _write( + repo, + "core/.cargo/config", + f'[patch."{guard.APPROVED_SOURCE_URL}"]\n', + ), + lambda _repo: guard.R005_CONFIG_PATCH_SOURCE.format( + config_path="core/.cargo/config", + source=guard.APPROVED_SOURCE_URL, + ), + ), + ( + lambda repo: _write( + repo, + "core/.cargo/config.toml", + ( + "[patch.crates-io]\n" + 'myspl = { package = "spl-transport", ' + 'path = "vendor/spl-transport" }\n' + ), + ), + lambda _repo: guard.R006_CONFIG_PATCH_PACKAGE.format( + config_path="core/.cargo/config.toml", + source="crates-io", + dependency="myspl", + package="spl-transport", + ), + ), + ( + lambda repo: ( + _write( + repo, + "vendor/spl-core/Cargo.toml", + '[package]\nname = "spl-core"\nversion = "0.1.0"\n', + ), + _git(repo, "add", "."), + ), + lambda _repo: guard.R007_IN_TREE_PACKAGE_COPY.format( + manifest="vendor/spl-core/Cargo.toml", + package="spl-core", + ), + ), + ], +) +def test_local_route_properties_fail( + tmp_path: Path, + capsys: pytest.CaptureFixture[str], + mutator: Mutator, + expected: ExpectedMessage, +) -> None: + _expect_failure(tmp_path, capsys, mutator, expected(tmp_path)) + + +def test_core_unlisted_member_manifest_is_still_checked( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path) + _write( + repo, + "core/crates/unlisted/Cargo.toml", + ( + "[package]\n" + 'name = "unlisted"\n' + 'version = "0.1.0"\n' + "\n" + "[dependencies]\n" + 'spl-core = "0.1.0"\n' + ), + ) + _git(repo, "add", ".") + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 1 + assert ( + guard.M001_MEMBER_OVERRIDE.format( + manifest="core/crates/unlisted/Cargo.toml", + table="dependencies", + dependency="spl-core", + package="spl-core", + keys="version", + ) + in captured.err + ) + + +def test_config_source_name_url_replacement_is_rejected( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path) + _write( + repo, + ".cargo/config", + ( + f'[source."{guard.APPROVED_SOURCE_URL}"]\n' + 'registry = "sparse+https://example.invalid/index"\n' + ), + ) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 1 + assert ( + guard.R004_CONFIG_SOURCE_REPLACEMENT.format( + config_path=".cargo/config", + source_name=guard.APPROVED_SOURCE_URL, + keys="registry", + ) + in captured.err + ) -- 2.51.2 From d2f546b5cd5dc9c17a538ed127519002d49c1ad8 Mon Sep 17 00:00:00 2001 From: Jer Miller Date: Fri, 31 Jul 2026 17:29:35 -0600 Subject: [PATCH 6/6] test(rust): pin the SPL guard's member-inheritance boundary Adds a passing case for a member entry that renames an inherited SPL dependency (package set, workspace = true, no source or selector keys). It is accepted because the entry draws its source and tag from the workspace pin and its lockfile record is keyed by the real package name, so both remain governed by the existing workspace and lockfile checks. Records that boundary as an invariant so the guard is not later tightened into a false positive. Removes an unused test helper. --- tests/test_spl_dependency_pin_guard.py | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/tests/test_spl_dependency_pin_guard.py b/tests/test_spl_dependency_pin_guard.py index a1315b417..9d62ec366 100644 --- a/tests/test_spl_dependency_pin_guard.py +++ b/tests/test_spl_dependency_pin_guard.py @@ -36,10 +36,6 @@ def _write(repo: Path, rel_path: str, text: str) -> None: path.write_text(text, encoding="utf-8") -def _read(repo: Path, rel_path: str) -> str: - return (repo / rel_path).read_text(encoding="utf-8") - - def _replace(repo: Path, rel_path: str, old: str, new: str) -> None: path = repo / rel_path source = path.read_text(encoding="utf-8") @@ -173,6 +169,24 @@ def test_valid_repo_passes_with_derived_second_tag( assert captured.err == "" +def test_member_workspace_inheritance_with_rename_is_accepted( + tmp_path: Path, capsys: pytest.CaptureFixture[str] +) -> None: + repo = _write_valid_repo(tmp_path) + # The local rename inherits the workspace pin; W and L still govern source, tag, and lock binding. + _append( + repo, + "core/crates/member/Cargo.toml", + 'myspl = { package = "spl-core", workspace = true }\n', + ) + + exit_code = _run_guard(repo) + + captured = capsys.readouterr() + assert exit_code == 0 + assert captured.err == "" + + def test_python_spl_surfaces_do_not_trip_in_tree_package_copy( tmp_path: Path, capsys: pytest.CaptureFixture[str] ) -> None: