Something went wrong. Try again.
Hardware-in-the-loop development platform for the Commodore 64.
Something went wrong. Try again.
43 kB · 848 lines
Python
at main
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849#!/usr/bin/env python3"""Build both host executables from a prepared frozen source archive, before packing.
``tools/package_release.py --prepare-source`` writes a frozen source directory(``source-manifest.json`` plus the ``-source.tar.gz`` it describes). This toolbuilds ``ultimate-tool`` and ``ultimate-tool-mcp`` from exactly those bytes, sothe packager can be handed binaries that were produced from the source it isabout to ship.
Why the prepared sources and not a checkout: the prepared archive carries avendored Cargo configuration written for the archive's own ``vendor/`` directory.Building from a developer checkout substitutes registry cache paths into thebinaries, so the paths recorded in those binaries no longer match the vendors thearchive lays down. Building from the prepared sources puts the vendored tree atthe recorded relative paths, which is what makes a later independent rebuildcomparable at all. This tool does not itself prove that comparison: anindependent ``tools/verify_release_local.py --mode source-rebuild`` rebuild of thesame archive is what produces the pack gate's evidence, and that gate is the onlything that may report the hashes as matching.
What runs, in order:
1. Read ``source-manifest.json`` through an anchored reader; recompute the file aggregate; require a canonical archive name and a mandatory, well-formed archive SHA-256; require vendored crate sources; and re-verify every listed member against the archive with the packager's own ``verify_prepared_source``.2. Build a *closed* inventory for this archive — the shared ``verify_release_local.prepared_source_inventory`` (first-party files, the vendored crate files and the distribution-document aliases) plus, locally here, the runtime bundle's ``source_members`` recorded in the manifest and the packager's generated members. Extract with the shared bounded extractor, then require the extracted tree to match that inventory exactly: an unlisted ``build.rs`` or any other unknown member is refused before a container starts, so nothing unlisted can ever be compiled.3. Create the output directory exclusively (it must be new — an existing path or symlink is refused) and run with ``--network none`` and only the extracted tree mounted read-write at ``/work``: copy the vendored ``.cargo/config.toml``, write the environment descriptor with the pinned image's own exclusive writer, then run the exact documented command ``cargo build --locked --release -p ultimate-tool -p ultimate-tool-mcp``, offline, with a fixed job count.4. Write one schema 1 ``release_source_build`` record beside the extracted tree.
Result record (``<work-dir>/build-result.json``) — the caller contract:
``record_kind`` ``release_source_build``. Deliberately *not* ``release_rebuild_evidence``: this is the first build, made before the package exists, so it must never be accepted by the packager's ``--rebuild-status verified`` gate. That gate consumes evidence from an independent second rebuild of the same archive.``source_archive`` / ``source_aggregate_sha256`` the prepared archive name, SHA-256 and prefix, and the manifest file aggregate the build is bound to.``build_argv`` / ``environment`` the exact documented command and the observed pinned environment.``binary_dir`` absolute host path of the rebuilt ``target/release``, to pass as ``--binary-dir``.``build_env`` absolute host path of the observed descriptor inside the extracted tree; ``build_env_beside_binaries`` is the copy the packager finds by default.``commands`` every container command actually launched, with its real exit code.``outputs`` / ``verified_outputs`` the rebuilt binaries with SHA-256 and size, or ``"none"``.``extraction`` / ``closure`` the closed inventory and the proof the extracted tree matched it.
The record is written to a fixed path inside the tool's own new work directoryand never overwrites an existing file. A refusal that happens before anycontainer starts (a bad archive, an unsafe work directory, a source withoutvendored crates, an unknown archive member, the reserved descriptor or``.cargo/config.toml`` already present) aborts with no record. Every failureafter the first command has launched — a non-zero exit, a missing or oversizedbinary, an unreadable or malformed environment descriptor, a failed copy — iscaptured in the record with a non-zero exit, so it can never be mistaken forsuccess. Only those expected operational errors are caught; nothing else isswallowed.
This tool never tags, pushes, publishes, prunes, contacts a device, or touchesthe network, and it never mounts this checkout, the Docker socket or any parentdirectory into the container. It never deletes the source directory or thearchive it was given, and it always keeps its own output directory."""
from __future__ import annotations
import argparseimport jsonimport mathimport osimport reimport sysfrom pathlib import Path
_HERE = Path(__file__).resolve().parentif str(_HERE) not in sys.path: sys.path.insert(0, str(_HERE))
import package_release as packaging # noqa: E402 - the prepared-archive verifierimport verify_release_local as verify # noqa: E402 - bounded helpers, reused not copied
SCHEMA = 1RECORD_KIND = "release_source_build"CONTAINER_LABEL = "ultimate-tool-build-release-source"SOURCE_MANIFEST_NAME = "source-manifest.json"BUILD_ENV_NAME = "build-env.json"RECORD_NAME = "build-result.json"# The descriptor path is reserved at the extracted tree's root: the pinned# image's writer must be the only thing that can create it.DESCRIPTOR_HOST_NAME = ".release-build-env.json"MOUNT_PURPOSE = "extracted prepared source tree"MAX_RECORD_BYTES = 1 << 20MAX_TIMEOUT_SECONDS = 1800.0MAX_BINARY_BYTES = 1 << 30SHA256_PATTERN = re.compile(r"^[0-9a-f]{64}$")
EXIT_OK, EXIT_FAILED, EXIT_USAGE, EXIT_ENVIRONMENT = 0, 1, 2, 3
BUILD_ENV_DESCRIPTOR = "/usr/local/bin/ultimate-tool-build-env"# The writer is invoked WITHOUT --replace, so it refuses to overwrite anything.DESCRIPTOR_ARGV = [BUILD_ENV_DESCRIPTOR, f"/work/{DESCRIPTOR_HOST_NAME}"]# A fixed command with no interpolation: caller input never reaches a shell. It# copies a real file already inside the mount; no configuration content is# spelled out here.VENDOR_CONFIG_COMMAND = [ "sh", "-euc", "if [ -f vendor-sources.toml ]; then install -D -m644 vendor-sources.toml .cargo/config.toml; " "echo copied vendor config; else echo no vendor config; fi",]
def parse_arguments(argv: list[str] | None = None) -> argparse.Namespace: parser = argparse.ArgumentParser( description="Build both host executables from a prepared frozen source archive.", formatter_class=argparse.RawDescriptionHelpFormatter, ) parser.add_argument("--source-dir", type=Path, required=True, help="prepared dist directory holding source-manifest.json") parser.add_argument("--work-dir", type=Path, required=True, help="a new directory this tool creates and owns") parser.add_argument("--image", default=verify.RELEASE_ENV_IMAGE, help="pinned release environment image") parser.add_argument("--build-jobs", type=int, default=2, help="CARGO_BUILD_JOBS for the build (default 2)") parser.add_argument( "--timeout", type=float, default=MAX_TIMEOUT_SECONDS, help=f"per-container-command timeout in seconds, at most {MAX_TIMEOUT_SECONDS:g}", ) parser.add_argument("--checkout-root", type=Path, default=_HERE.parent, help="this checkout; never mounted") parser.add_argument("--forbid", type=Path, action="append", default=None, help="a path the container must not reach") parser.add_argument("--json", action="store_true", help="print the record to stdout") return parser.parse_args(argv)
def normalize_path(value: Path) -> Path: """Make a path absolute and lexically normal, without resolving symlinks.""" return Path(os.path.abspath(os.fspath(value)))
def reject_symlink_chain(path: Path, description: str) -> None: """Refuse a path whose own leaf or any existing ancestor is a symlink.""" current = Path(path.anchor) for part in path.parts[1:]: current = current / part if os.path.islink(current): raise verify.HarnessError(f"{description} {path} passes through the symlink {current}")
def forbidden_paths(args: argparse.Namespace) -> list[Path]: return [Path(path).resolve() for path in (args.forbid or verify.default_forbidden(args.checkout_root))]
def prepare_work_directory(work_dir: Path, forbidden: list[Path]) -> Path: """Create this tool's own output directory, exclusively.
The directory must not exist: reusing an existing tree could merge these outputs with someone else's, and clearing one would delete data this tool was never given. Unsafe parents are refused too, so neither the extracted sources nor the rebuilt binaries can land inside the distribution they came from. """ work_dir = normalize_path(work_dir) reject_symlink_chain(work_dir, "--work-dir") if os.path.lexists(work_dir): raise verify.HarnessError( f"--work-dir {work_dir} already exists or is a symlink; this tool creates a new directory and " "refuses to reuse or clear one" ) if not work_dir.parent.is_dir(): raise verify.HarnessError(f"the parent of --work-dir {work_dir} is not an existing directory") resolved = work_dir.resolve() if resolved.parent == resolved or resolved == Path("/root") or Path("/root") in resolved.parents: raise verify.HarnessError(f"--work-dir {work_dir} is a system directory; refusing to build there") for forbidden_path in forbidden: anchor = forbidden_path.resolve() if resolved == anchor or anchor in resolved.parents: raise verify.HarnessError(f"--work-dir {work_dir} is inside the forbidden path {anchor}; refusing to build there") work_dir.mkdir(mode=0o700) return work_dir
def canonical_archive_name(archive: dict, source_dir: Path) -> str: """The one canonical archive file name the manifest may name in this directory.""" name = archive.get("name") if not isinstance(name, str) or not name: raise verify.HarnessError("the prepared source manifest does not name its archive") if name != Path(name).name or name in (".", "..") or os.sep in name or (os.altsep and os.altsep in name): raise verify.HarnessError(f"the prepared source manifest names a non-canonical archive {name!r}") archive_path = source_dir / name if archive_path.is_symlink() or not archive_path.is_file(): raise verify.HarnessError(f"the prepared source archive {name} is missing or is not a regular file in {source_dir}") return name
def open_prepared_source(source_dir: Path) -> dict: """Read and close the prepared inventory before anything is extracted.
The directory is validated as given — no symlink in its chain — and the manifest is read through the anchored reader, which never follows a symlink leaf. The aggregate is recomputed from the manifest's own file list, the archive name must be canonical, the archive SHA-256 is mandatory and must match the bytes on disk, and the packager's ``verify_prepared_source`` re-checks every listed member. """ source_dir = normalize_path(source_dir) reject_symlink_chain(source_dir, "--source-dir") if not source_dir.is_dir(): raise verify.HarnessError(f"--source-dir {source_dir} is not a directory") manifest = verify.load_json_file(source_dir / SOURCE_MANIFEST_NAME, "source manifest") files = manifest.get("files") or [] if not files: raise verify.HarnessError("the prepared source manifest lists no files") if verify.aggregate_digest(files) != manifest.get("aggregate_sha256"): raise verify.HarnessError("the prepared source aggregate does not match its file list") archive = manifest.get("archive") or {} archive_name = canonical_archive_name(archive, source_dir) recorded = archive.get("sha256") if not isinstance(recorded, str) or not SHA256_PATTERN.match(recorded): raise verify.HarnessError("the prepared source manifest records no valid archive SHA-256") digest = verify.sha256_file(source_dir / archive_name) if digest != recorded: raise verify.HarnessError("the prepared source archive does not match the hash in its manifest") try: packager_check = verify.verify_prepared_source_with_packager(source_dir / archive_name, manifest) except packaging.CheckFailure as error: raise verify.HarnessError(f"the prepared source archive does not match its manifest: {error}") from error if not packager_check.get("checked"): raise verify.EnvironmentUnavailable( "tools/package_release.py could not re-verify the prepared archive, so its inventory is not " "closed; refusing to extract without it" ) return { "source_dir": source_dir, "manifest": manifest, "files": files, "vendor": manifest.get("vendor") or {}, "archive_name": archive_name, "archive_sha256": digest, "packager_check": packager_check, }
def closed_inventory(manifest: dict) -> dict: """The finite member set this archive may contain, and nothing else.
The shared ``prepared_source_inventory`` supplies the first-party files, the vendored crate files and the distribution-document aliases, and the packager's generated member names. The runtime bundle's ``source_members`` are added here because the shared helper does not carry them: they ship inside this archive at ``<prefix>/<path>`` and the manifest records their size and SHA-256, so they are hashed members rather than unexplained extras.
Members that must be present are required by name; everything else in ``expected`` is required by size and hash. The returned ``files`` set is the closed allowlist the extracted tree is compared against. """ base = verify.prepared_source_inventory(manifest) prefix = base["prefix"] expected = dict(base["expected"]) required = list(base["required"]) runtime = manifest.get("runtime_bundle") or {} if runtime.get("present"): for entry in runtime.get("source_members") or []: path = entry.get("path") if not isinstance(path, str) or not path: raise verify.HarnessError("the runtime bundle records a source member with no path") expected[f"{prefix}/{path}"] = {"sha256": entry.get("sha256"), "size": entry.get("size")} return { "prefix": prefix, "expected": expected, "required": tuple(required), "files": frozenset(set(expected) | set(required)), "runtime_members": sorted( f"{prefix}/{entry['path']}" for entry in (runtime.get("source_members") or []) if runtime.get("present") ), }
def tree_closure(extraction: Path, allowed_files: set[str]) -> dict: """Compare the extracted tree with the closed inventory, exactly.
Returns the members that are *not* allowed — files, extra directories and any symlink at all. Files are the members that can change a build, so an unlisted ``build.rs`` shows up here; directories are only allowed when they are an ancestor of an allowed file, so an unexplained empty directory is caught too. """ allowed_dirs = set() for name in allowed_files: parts = name.split("/") for index in range(1, len(parts)): allowed_dirs.add("/".join(parts[:index])) files, directories, links = set(), set(), [] for current, dirnames, filenames in os.walk(extraction): for name in list(dirnames) + list(filenames): path = Path(current) / name relative = path.relative_to(extraction).as_posix() if path.is_symlink(): links.append(relative) elif name in dirnames: directories.add(relative) else: files.add(relative) return { "members": len(files) + len(directories), "unknown_files": sorted(files - allowed_files), "unknown_directories": sorted(directories - allowed_dirs), "symlinks": sorted(links), "listed_files": len(files & allowed_files), }
def descriptor_read_problems(path: Path) -> tuple[dict | None, bytes | None, list[str]]: """Read the observed descriptor through the anchored reader, never raising.
``verify.read_regular_file`` opens with ``O_NOFOLLOW`` and a byte bound, so a symlinked or oversized descriptor is reported rather than followed. The raw bytes are returned too, so the copy beside the binaries writes exactly the bytes that were read and the record can carry their digest. """ try: data = verify.read_regular_file(path, description="observed build-environment descriptor") except verify.HarnessError as error: return None, None, [f"the build-environment descriptor could not be read: {error}"] try: document = json.loads(data) except ValueError as error: return None, data, [f"the build-environment descriptor is not valid JSON: {error}"] if not isinstance(document, dict): return None, data, ["the build-environment descriptor is not a JSON object"] return document, data, []
def post_build_directory_problems(binary_dir: Path, source_root: Path) -> list[str]: """Constrain every host read and write to the extracted tree after the build.
The build runs in a container, so the directory tree it leaves behind is not trusted: an ancestor that is a symlink — ``target`` or ``target/release`` — would redirect the output scan, the hashing and the descriptor copy to a host path outside this tool's own tree. Both the ancestor chain and the resolved containment are checked before anything is read or created. """ problems: list[str] = [] try: reject_symlink_chain(binary_dir, "the build output directory") except verify.HarnessError as error: problems.append(str(error)) resolved_root, resolved_dir = source_root.resolve(), binary_dir.resolve() if resolved_dir != resolved_root and resolved_root not in resolved_dir.parents: problems.append(f"the build output directory {resolved_dir} is outside the extracted tree {resolved_root}") return problems
def write_new_exclusive(directory: Path, name: str, data: bytes) -> Path: """Create one new file inside ``directory``, anchored and never following a link.
``verify.write_new_file`` looks at ``path.exists``, which is false for a dangling symlink, and only checks the immediate parent; this tool needs the leaf and every ancestor to be real, because the container just wrote them. The parent directory is therefore opened as a directory descriptor and the file is created relative to it with ``O_CREAT|O_EXCL|O_NOFOLLOW``, so an existing entry of any kind — including a dangling symlink pointing outside the tree — fails the create instead of being written through or replaced. """ reject_symlink_chain(directory, "the build output directory") target = directory / name if os.path.lexists(target): raise verify.HarnessError(f"refusing to overwrite the existing {target}") directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0) try: dir_fd = os.open(directory, directory_flags) except OSError as error: raise verify.HarnessError(f"the build output directory {directory} could not be opened: {error}") from error try: flags = os.O_WRONLY | os.O_CREAT | os.O_EXCL | getattr(os, "O_NOFOLLOW", 0) try: handle = os.open(name, flags, 0o644, dir_fd=dir_fd) except FileExistsError as error: raise verify.HarnessError(f"refusing to overwrite the existing {target}") from error except OSError as error: raise verify.HarnessError(f"{target} could not be created: {error}") from error with os.fdopen(handle, "wb") as stream: stream.write(data) finally: os.close(dir_fd) return target
def descriptor_shape_problems(descriptor: dict) -> list[str]: """Shape and type checks, so a malformed descriptor is never inventoried.""" problems: list[str] = [] if descriptor.get("schema") != SCHEMA or descriptor.get("record_kind") != "build_environment": problems.append( f"the descriptor must be schema {SCHEMA} build_environment; got schema " f"{descriptor.get('schema')!r} record_kind {descriptor.get('record_kind')!r}" ) base = descriptor.get("base_image") if not isinstance(base, dict): problems.append(f"the descriptor's base_image is not an object: {base!r}") elif not isinstance(base.get("digest"), str) or not base["digest"]: problems.append(f"the descriptor's base_image records no digest: {base.get('digest')!r}") toolchain = descriptor.get("toolchain") if not isinstance(toolchain, dict): problems.append(f"the descriptor's toolchain is not an object: {toolchain!r}") else: for tool in ("rustc", "cargo"): if not isinstance(toolchain.get(tool), str) or not toolchain[tool]: problems.append(f"the descriptor's toolchain records no {tool}: {toolchain.get(tool)!r}") if not isinstance(descriptor.get("target_triple"), str) or not descriptor.get("target_triple"): problems.append(f"the descriptor records no target_triple: {descriptor.get('target_triple')!r}") if not descriptor.get("observed_at"): problems.append("the descriptor records no observed_at timestamp") return problems
def observed_environment(descriptor: dict, image: str, build_jobs: int) -> dict: """The reviewer-facing environment summary, taken from the observed descriptor.""" base, toolchain = descriptor.get("base_image") or {}, descriptor.get("toolchain") or {} return { "base_image_digest": base.get("digest"), "base_image_reference": base.get("reference"), "base_image_pinned": base.get("pinned"), "rustc": toolchain.get("rustc"), "cargo": toolchain.get("cargo"), "target_triple": descriptor.get("target_triple"), "os": descriptor.get("os"), "glibc": descriptor.get("glibc"), "observed_in": f"docker image {image}", "observed_at": descriptor.get("observed_at"), "build_jobs": build_jobs, "build_jobs_source": "CARGO_BUILD_JOBS in the build container; the recorded command keeps no -j flag", "descriptor": descriptor, }
def environment_problems(environment: dict) -> list[str]: """The pinned-policy checks the observed environment must pass.""" problems: list[str] = [] if environment.get("base_image_digest") != verify.PINNED_BASE_DIGEST: problems.append( "the observed build environment records a different base image digest than the pinned policy: " f"{environment.get('base_image_digest')!r}" ) if environment.get("target_triple") != verify.PINNED_TARGET_TRIPLE: problems.append(f"the observed target triple is {environment.get('target_triple')!r}") token = re.search(r"rustc (\d+\.\d+\.\d+)", environment.get("rustc") or "") if not token or token.group(1) != verify.PINNED_RUST_VERSION: problems.append(f"the observed rustc is {environment.get('rustc')!r}, expected {verify.PINNED_RUST_VERSION}") return problems
def measure_binary(path: Path) -> tuple[dict | None, str | None]: """Measure one present rebuilt binary, reporting an oversize or unreadable one.""" if path.is_symlink() or not path.is_file(): return None, f"the rebuilt {path.name} is not a regular file" try: size = path.stat().st_size if size > MAX_BINARY_BYTES: return None, f"the rebuilt {path.name} is {size} bytes, above the {MAX_BINARY_BYTES} byte bound" digest = verify.sha256_file(path) except (verify.HarnessError, OSError) as error: return None, f"the rebuilt {path.name} could not be measured: {error}" return {"name": path.name, "sha256": digest, "size": size, "path": f"target/release/{path.name}"}, None
def bound_record(document: dict) -> None: """Keep the record inspectable by dropping command output tails last.""" def size() -> int: return len(json.dumps(document, sort_keys=True).encode("utf-8"))
if size() <= MAX_RECORD_BYTES: return for command in (document.get("harness") or {}).get("commands") or []: command["stdout_tail"] = command["stderr_tail"] = "" document.setdefault("harness", {}).setdefault("notes", []).append( "command output tails were dropped so the record stays inspectable" ) if size() > MAX_RECORD_BYTES: raise verify.HarnessError("the build record cannot be reduced to its byte bound")
def container(args, journal, mount, user, argv, *, note, environment=None): """One container command, always offline, always inside the extracted tree.""" return verify.docker_run( image=args.image, argv=argv, journal=journal, workdir="/work", mount=mount, mount_purpose=MOUNT_PURPOSE, user=user, environment=environment, network="none", timeout=args.timeout, note=note, )
def run_build(args, journal: verify.Journal, cleanup: verify.Cleanup) -> tuple[dict, int, list[str]]: # --- Preflight: nothing has run yet, so a refusal here writes no record. --- available, detail = verify.docker_available(journal) if not available: raise verify.EnvironmentUnavailable(f"docker is unavailable: {detail}") if not verify.docker_image_present(args.image): raise verify.EnvironmentUnavailable( f"release environment image {args.image} is not present locally; build it before running this tool" ) forbidden = forbidden_paths(args) journal.note(f"harness checkout root (never mounted): {args.checkout_root}") journal.note(f"forbidden paths: {[str(path) for path in forbidden]}")
prepared = open_prepared_source(args.source_dir) manifest, vendor = prepared["manifest"], prepared["vendor"] if not vendor.get("files"): raise verify.HarnessError( "the prepared source carries no vendored crate sources, so the archive is not complete for an " "offline build of the corresponding source; refusing to compile" ) inventory = closed_inventory(manifest) prefix = inventory["prefix"] reserved = f"{prefix}/{DESCRIPTOR_HOST_NAME}" if reserved in inventory["files"]: raise verify.HarnessError( f"the prepared source manifest lists the reserved descriptor path {reserved}; it is written by the " "pinned image, so it must not ship inside the archive" )
work_dir = prepare_work_directory(args.work_dir, forbidden) extraction = work_dir / "extracted" observed_archive = verify.extract_verified_archive( prepared["source_dir"] / prepared["archive_name"], extraction, expected=inventory["expected"], required=inventory["required"], limits_note="prepared source archive", ) if observed_archive["archive_sha256"] != prepared["archive_sha256"]: raise verify.HarnessError("the source archive changed between the manifest check and extraction") if observed_archive["prefix"] and observed_archive["prefix"] != prefix: raise verify.HarnessError( f"the archive's top-level directory {observed_archive['prefix']!r} does not match the prefix the " f"prepared manifest recorded, {prefix!r}" ) source_root = extraction / prefix # Reserved and reserved-effect paths are checked before the archive's member # closure, so a listed or unlisted copy is refused either way, with its bytes # left exactly as the archive delivered them. for name, guard in ((DESCRIPTOR_HOST_NAME, "the pinned image writes it"), (".cargo/config.toml", "this tool writes it")): if os.path.lexists(source_root / name): raise verify.HarnessError( f"the extracted source already carries {name} ({guard}); refusing to overwrite or continue" ) if not (source_root / "Cargo.toml").is_file(): raise verify.HarnessError(f"the extracted source has no workspace Cargo.toml at {source_root}")
closure = tree_closure(extraction, set(inventory["files"])) unknown = closure["unknown_files"] + closure["unknown_directories"] if unknown or closure["symlinks"]: raise verify.HarnessError( "the extracted tree does not match the closed inventory for this archive; refusing to build " "anything from it:\n" + "\n".join(f" unlisted member: {name}" for name in unknown[:20]) + "\n".join(f" symlink: {name}" for name in closure["symlinks"][:20]) + f"\n ({len(unknown)} unlisted member(s), {len(closure['symlinks'])} symlink(s))" )
problems = verify.isolation_problems( checkout_root=args.checkout_root, mounts=[{"source": str(source_root), "target": "/work"}], forbidden_paths=forbidden, expected_source=source_root, ) if problems: raise verify.HarnessError("isolation guard refused the build:\n" + "\n".join(f" {item}" for item in problems))
# --- From here on every launched command is recorded, pass or fail. --- failures: list[str] = [] steps: list[dict] = [] user, mount = f"{os.getuid()}:{os.getgid()}", (source_root, "/work", "rw") journal.note("only the extracted source tree is mounted, read-write at /work; this checkout was not visible inside")
def run_step(name: str, argv: list[str], environment: dict | None, note: str): """Run one container command and record it, pass or fail.
A command that cannot run at all — a timeout, or a Docker invocation that failed — raises out of the shared runner. That is still a launched command's failure, so it becomes a recorded failed step. Returns ``(record, error_message_or_None)``. """ error = None try: record = container(args, journal, mount, user, argv, note=note, environment=environment) except verify.HarnessError as raised: error = str(raised) record = verify.CommandRecord(argv=list(argv), exit_code=-1, duration_ms=0, stderr=str(raised)) steps.append( { "step": name, "argv": list(argv), "exit_code": record.exit_code, "duration_ms": record.duration_ms, "environment": dict(environment or {}), "stderr_tail": record.stderr[-2000:], "error": error, } ) return record, error
copied, copied_error = run_step( "vendor-config", VENDOR_CONFIG_COMMAND, None, "copy the vendored crate configuration to .cargo/config.toml" ) if copied_error: failures.append(f"the vendor config step could not run: {copied_error}") elif copied.exit_code != 0: failures.append(f"the vendor config step exited {copied.exit_code}: {copied.stderr.strip()[:400]}")
descriptor_host = source_root / DESCRIPTOR_HOST_NAME environment = None descriptor_bytes: bytes | None = None if not failures: descriptor, descriptor_error = run_step( "build-environment", DESCRIPTOR_ARGV, {"RELEASE_BASE_REFERENCE": verify.PINNED_BASE_REFERENCE}, "observed environment descriptor, written exclusively by the pinned image", ) if descriptor_error: failures.append(f"the build-environment descriptor step could not run: {descriptor_error}") elif descriptor.exit_code != 0: failures.append( f"the build-environment descriptor step exited {descriptor.exit_code}: " f"{descriptor.stderr.strip()[:400]}" ) elif not os.path.lexists(descriptor_host): failures.append( f"the build-environment descriptor step reported success but wrote no descriptor at {descriptor_host}" ) else: observed, descriptor_bytes, problems = descriptor_read_problems(descriptor_host) if observed is not None: problems = list(problems) + descriptor_shape_problems(observed) if not problems and observed is not None: environment = observed_environment(observed, args.image, args.build_jobs) problems = environment_problems(environment) failures.extend(problems)
offline_env = {"CARGO_NET_OFFLINE": "true", "CARGO_BUILD_JOBS": str(args.build_jobs)} binary_dir = source_root / "target" / "release" outputs: list[dict] = [] missing_outputs: list[str] = [] rebuild = None directory_problems: list[str] = [] if not failures: rebuild, rebuild_error = run_step( "build", verify.BUILD_COMMAND, offline_env, "documented release build command, offline, inside the pinned image" ) # Before a single byte is read back from the build output, prove the tree # the container left behind is still inside this tool's own extraction. directory_problems = post_build_directory_problems(binary_dir, source_root) if directory_problems: failures.extend(directory_problems) journal.note("the build output directory was not scanned because it is not a plain directory inside the extracted tree") else: for name in verify.BINARY_NAMES: binary = binary_dir / name if not binary.is_file(): missing_outputs.append(name) continue measured, measure_problem = measure_binary(binary) if measure_problem: failures.append(measure_problem) continue outputs.append(measured) if rebuild_error: failures.append(f"the documented build command could not run: {rebuild_error}") elif rebuild.exit_code != 0: failures.append(f"the documented build command exited {rebuild.exit_code}") for name in missing_outputs: failures.append(f"the build produced no {name} at target/release/{name}") else: journal.note("the build step was not run because an earlier step failed")
# The observed descriptor stays where the pinned image wrote it, inside the # extracted tree, and is copied next to the binaries because that is where the # packager looks for it by default. The copy is created exclusively through a # directory descriptor, so an existing entry — including a dangling symlink # pointing outside the tree — is refused instead of being written through, and # no host path outside the extracted tree is ever written. beside_binaries = None if descriptor_bytes is not None and not directory_problems and binary_dir.is_dir(): try: beside_binaries = write_new_exclusive(binary_dir, BUILD_ENV_NAME, descriptor_bytes) except verify.HarnessError as error: failures.append(f"the descriptor could not be copied beside the binaries: {error}")
document = { "schema": SCHEMA, "record_kind": RECORD_KIND, "produced_by": "tools/build_release_source.py", "verdict": "built" if not failures else "failed", "verdict_reason": ( "the first build completed and both binaries are bound to the prepared source archive; this is " "not pack gate evidence" if not failures else "the build did not complete, so nothing from this run is verified" ), "pack_gate_evidence": False, "pack_gate_note": ( "the packager's --rebuild-status verified gate consumes release_rebuild_evidence from an " "independent second rebuild of the same source archive; this record is not that evidence" ), "source_archive": { "name": prepared["archive_name"], "sha256": prepared["archive_sha256"], "prefix": prefix, "aggregate_sha256": manifest.get("aggregate_sha256"), }, "source_aggregate_sha256": manifest.get("aggregate_sha256"), "build_argv": list(verify.BUILD_COMMAND), "environment": environment, "image": args.image, "image_id": verify.docker_image_identity(args.image), "commands": steps, "outputs": outputs if not failures else [], "verified_outputs": [output["name"] for output in outputs] if not failures else "none", "binary_dir": str(binary_dir), "build_env": str(descriptor_host) if descriptor_bytes is not None else None, "build_env_sha256": verify.sha256_bytes(descriptor_bytes) if descriptor_bytes is not None else None, "build_env_beside_binaries": str(beside_binaries) if beside_binaries else None, "source_manifest": { "directory": str(prepared["source_dir"]), "aggregate_sha256": manifest.get("aggregate_sha256"), "files": len(prepared["files"]), "vendor_files": len(vendor.get("files") or []), "runtime_members": inventory["runtime_members"], }, "extraction": { "archive": observed_archive["archive"], "archive_sha256": observed_archive["archive_sha256"], "prefix": observed_archive["prefix"], "members": observed_archive["members"], "actual_bytes": observed_archive["actual_bytes"], "verified_against": SOURCE_MANIFEST_NAME, "generated_members": list(inventory["required"]), "extractor_extras": observed_archive["extras"], "allowed_files": len(inventory["files"]), "audit": observed_archive["audit"], "packager_check": prepared["packager_check"], }, "closure": closure, "isolation": { "mounts": journal.as_json()["mounts"], "network": "none", "checkout_mounted": False, "note": "only the extracted source tree was mounted; this checkout was not visible inside", }, "vendor": {"vendored": True, "config_step": copied.stdout.strip() if copied.exit_code == 0 else None}, "harness": journal.as_json(), "failures": failures, } if failures: # A failed build writes a complete record rather than nothing: the steps # that ran, their real exit codes, bounded output tails and the source # archive it is bound to. It claims no verified outputs. document["failure"] = { "steps_run": [step["step"] for step in steps], "step_exit_codes": {step["step"]: step["exit_code"] for step in steps}, "missing_outputs": missing_outputs, "outputs_produced": [output["name"] for output in outputs], "build_exit_code": rebuild.exit_code if rebuild else None, "build_environment": offline_env, "source_archive": {"name": prepared["archive_name"], "sha256": prepared["archive_sha256"]}, "preserved_work_dir": str(work_dir), "investigate": "the extracted source and any partial build outputs are kept under the recorded work directory; nothing here is verified", } bound_record(document) return document, EXIT_OK if not failures else EXIT_FAILED, failures
def main(argv: list[str] | None = None) -> int: args = parse_arguments(argv) if not 1 <= args.build_jobs <= 64: print("build_release_source: --build-jobs must be between 1 and 64", file=sys.stderr) return EXIT_USAGE if not math.isfinite(args.timeout) or not 0 < args.timeout <= MAX_TIMEOUT_SECONDS: print(f"build_release_source: --timeout must be a finite number up to {MAX_TIMEOUT_SECONDS:g}", file=sys.stderr) return EXIT_USAGE verify.INVOCATION = verify.new_invocation_id() verify.CONTAINER_LABEL = CONTAINER_LABEL args.checkout_root = args.checkout_root.resolve() args.work_dir = normalize_path(args.work_dir) record_path = args.work_dir / RECORD_NAME
journal, cleanup = verify.Journal(), verify.Cleanup() cleanup.install_signal_handlers() # Only the containers this invocation labelled are removed: never "all" # containers, and never one a concurrent invocation started. cleanup.add( f"remove containers labelled {verify.container_label()}", lambda: verify.remove_containers_with_label(verify.container_label()), ) try: document, code, failures = run_build(args, journal, cleanup) except verify.EnvironmentUnavailable as error: print(f"build_release_source: cannot run here: {error}", file=sys.stderr) return EXIT_ENVIRONMENT except (verify.HarnessError, packaging.CheckFailure) as error: print(f"build_release_source: {error}", file=sys.stderr) return EXIT_FAILED finally: cleanup.run()
payload = json.dumps(document, indent=2, sort_keys=True) + "\n" try: # Never overwrite: the work directory is this tool's own new directory, # so an existing record means something else is already there. written = verify.write_new_file(record_path, payload.encode("utf-8"), replace=False) except verify.HarnessError as error: print(f"build_release_source: {error}", file=sys.stderr) if args.json: print(payload, end="") return EXIT_FAILED print(f"wrote {written['path']} (sha256 {written['sha256']}, {written['size']} bytes)") if args.json: print(payload, end="") for failure in failures: print(f"build_release_source: {failure}", file=sys.stderr) print(f"verdict: {document['verdict']} (binary_dir {document['binary_dir']})") return code
if __name__ == "__main__": sys.exit(main())