david.boston / ultimate-tool
Hardware-in-the-loop development platform for the Commodore 64.
Star
Fork 0
Watch
ultimate-tool docs PUBLISHING.md
12 kB
Markdown
at main

Publishing release artifacts to Tangled #

tools/publish_tangled_artifacts.py is the local, explicit publisher for an Ultimate Tool release. It takes a release directory already produced by tools/package_release.py and either verifies it (the default dry run) or uploads one binary archive and creates one sh.tangled.repo.artifact record that points at the release's annotated tag.

It is deliberately narrow. It does not build, package, tag, push, or sign anything, and it never changes repository or Knot/Spindle settings. It publishes files that already exist and records exactly what it did.

The independent preflight is mandatory #

Before the publisher resolves any identity, opens a session, or obtains a credential, it runs the shared, independent release checker:

tools/check_release.py:  validate_release(release_dir, source_root=…, require_ready=True)

The publisher does not re-implement any release check. It loads the reviewed tools/check_release.py that ships beside it and calls validate_release directly. That function re-reads every archive member, re-measures both host binaries from the bytes inside the binary archive, re-derives the privacy findings, re-validates the recorded build environment, and re-reads the Git state — and returns a verdict document instead of raising.

The gate passes only when the checker reports ready (a passed verdict and the manifest's recorded release_ready is true). If it does not pass, the publisher refuses with release_preflight_failed and never touches the network or a secret.

There is no flag and no environment variable that skips this gate. The module is loaded from an explicit file location rather than a bare module name, so a PYTHONPATH entry cannot shadow it with a no-op stand-in. A missing checker is a hard failure (release_checker_missing), not a skipped step.

--release-source-root selects the source tree the checker re-derives Git state from; it defaults to --repo-dir (the tree that holds the annotated tag).

Prerequisites #

  • A release directory produced by tools/package_release.py, containing the binary archive, the source archive, release-manifest.json, source-manifest.json and SHA256SUMS.
  • tools/check_release.py present next to the publisher (it is part of the repository).
  • The Git working tree that holds the release's annotated tag (--repo-dir).
  • For a real publication: a reviewed sh.tangled.repo.artifact lexicon document, a confirmed PDS blob upload bound, and an app password.

Dry run (default) #

A dry run performs every local check — including the independent preflight — prints the exact record it would create, and makes no network call and no credential prompt.

python3 tools/publish_tangled_artifacts.py \
  --release-manifest dist/release-manifest.json \
  --repo https://tangled.org/<handle>/ultimate-tool \
  --owner-did did:plc:… \
  --identifier <handle> \
  --tag v0.2.0-preview.1 \
  --repo-dir .

Add --json for a single machine-readable envelope on stdout. Add --reviewed-schema <path> --max-blob-bytes <n> to also check the reviewed lexicon and the declared PDS upload bound without publishing.

Publishing #

Publishing is never implicit. It additionally requires --publish, a reviewed schema, a declared blob bound, and a separately approved credential:

TANGLED_APP_PASSWORD='…' python3 tools/publish_tangled_artifacts.py \
  --release-manifest dist/release-manifest.json \
  --repo https://tangled.org/<handle>/ultimate-tool \
  --owner-did did:plc:… \
  --identifier <handle> \
  --tag v0.2.0-preview.1 \
  --repo-dir . \
  --publish \
  --reviewed-schema <reviewed-sh.tangled.repo.artifact.json> \
  --max-blob-bytes <confirmed PDS bound> \
  --receipt dist/publish-receipt.json

The credential comes from TANGLED_APP_PASSWORD (configurable with --password-env) or, only from a terminal, an interactive prompt. It is never echoed, never stored, and removed from the process environment once read.

Owner binding #

--owner-did is the DID that must own both the repository and the record. The publisher resolves --identifier (a handle or DID) and, when --repo names the owner with a handle, the repository's owner, and refuses to continue unless both resolve to exactly that DID. createSession must also authenticate as the same DID before any blob is uploaded.

Transport rules #

  • --pds-origin must be an https base origin with no path, query, fragment or URL credentials. Redirects are rejected outright.
  • TLS uses the system trust store with default verification.
  • Every request is bounded by --timeout, and the whole publication by a single finite, monotonic --total-timeout deadline. The deadline shuts the request socket down when it is reached, so a server that trickles a byte at a time cannot extend the run past its budget, and DNS resolution is bounded by the same deadline. Non-finite or non-positive values are rejected.
  • Failures are reported redacted and stable: a rejected request reports only the operation and the HTTP status, never any part of the remote body (which could echo a credential or an identity field). The app password, the session access token, the refresh token and raw remote bodies never appear in output, receipts or logs.
  • The git helper runs with the credential environment variables (TANGLED_APP_PASSWORD, C64U_PASSWORD) removed, so a read command can never inherit a secret.

What is uploaded #

A publication distributes the complete required set — nothing less:

role file content type
binary-archive the release's binary archive --content-type (default application/octet-stream)
source-archive the corresponding source archive archive content type
checksums SHA256SUMS text/plain; charset=utf-8
release-manifest release-manifest.json application/json

Each file is uploaded with repo.uploadBlob and bound to the same annotated tag object by its own sh.tangled.repo.artifact record (created with repo.createRecord, name = the file's own name). The binary record's name can be overridden with --artifact-name; the other three always keep their real names. Publishing the binary archive alone is no longer possible.

Canonical inputs #

The shared checker validates exactly one directory, and the publisher uploads exactly the files inside that directory. --release-manifest must be <release-dir>/release-manifest.json and --checksums (when given) must be <release-dir>/SHA256SUMS; a manifest or checksum file with any other name or location is refused (noncanonical_manifest / noncanonical_checksums) before any credential or network call, so the checker and the uploader can never be pointed at different bytes.

Private staging and content-derived blobs #

Immediately after the local checks pass, the publisher copies exactly the reviewed files the checker reads — the four distribution objects, the source manifest and the registered rebuild evidence — into a fresh private (0700, tool-owned) staging directory. It never recursively copies the release directory, so an unreviewed or private file cannot travel with the release. The shared checker then runs against the staged copy, so the bytes that are validated are the bytes that are uploaded.

Each object is streamed from staging and hashed as it goes; the byte count and digest must match the reviewed snapshot or the upload is abandoned (upload_tampered) with no record created. Every object's size bound is re-applied to the staged bytes. The blob reference the PDS returns must be a CIDv1 (raw codec, sha2-256) whose digest equals the SHA-256 of the uploaded bytes, and its reported size and MIME type must match (a missing MIME type is refused) — otherwise the record is never created (blob_content_mismatch / blob_size_mismatch / blob_mime_mismatch). The createRecord response is accepted only when the returned URI names this owner and collection with a well-formed record key and the returned cid is a valid CIDv1 dag-cbor (0x71) sha2-256 reference; a missing or malformed identifier, or one that echoes a known credential, is rejected as an ambiguous attempt (never scrubbed into a claimed success), and the whole per-object state is scrubbed of any credential before it reaches stdout, a receipt or a partial envelope.

Lexicon constraints #

The reviewed sh.tangled.repo.artifact lexicon is enforced client-side before any upload. Every one of the four planned record bodies is checked against every declared constraint — field types, string and byte lengths, the createdAt datetime format (a real calendar date, not just a regex), and the blob's maxSize and accept MIME list — using a locally derived placeholder blob (the PDS returns a content-derived CID, so the reference is known without a network call). The constraint vocabulary is a closed set: an unsupported type, an unknown constraint key, or an unsupported format is refused (reviewed_schema_unsupported), as is a lexicon that requires fields this tool does not publish. The recipe posts validate: false; this client-side enforcement is what stands in for server-side validation.

Ambiguous outcomes #

A failed or ambiguous outcome never triggers a blind retry, because a failed upload is not a failed record and replaying it could publish twice. The publisher tracks per-object state, records exactly which objects were uploaded and which records were created, and reports a structured partial result — on stdout with --json even when no --receipt was given, and in the receipt when one was. Nothing is uploaded or recorded twice.

Receipts #

--receipt <path> writes a receipt (schema 1, record_kind: ultimate-tool-tangled-publish-receipt). The path is reserved before any remote call, created exclusively (O_CREAT|O_EXCL, never truncating an existing file, a FIFO, a hard link or a symlink) with mode 0600. It records the release facts, per-object hashes and sizes, the session DID, each object's blob reference and created record URI/CID/rkey, and any ambiguity, and is verified to be free of the credential and the session token before it is written. An existing receipt path is refused (receipt_unsafe); an ambiguous outcome still writes a partial receipt with everything known, and the structured state is attached to the error before the file is written, so a write or fsync failure can only mark receipt_written: false — it never loses the remote facts or escapes as a traceback. With --json, the same partial state is emitted on stdout even if no --receipt was given.

Exit codes #

Code Meaning
0 success (dry run completed, or the record was created)
2 usage error (bad arguments)
3 local preflight failure, including the independent release check
4 remote failure (transport, session, upload, create-record)
5 refused (mock/loopback misuse)

What the tests prove — and do not #

The publisher's tests build their release directory by running the real packager (tools/package_release.py, two-stage --prepare-source then --pack) over a synthetic project, so they exercise genuine archive bytes and a genuine release manifest, not a hand-shaped stand-in. The independent checker runs against those bytes. Publication is driven by a disposable HTTP server bound to 127.0.0.1, and the tags are created in a throwaway local repository.

A passing suite therefore proves the local gates (including the shared independent release check and its ordering before any identity or secret), the exact XRPC request shapes, the credential handling and the failure paths.

It does not prove that any artifact was published, that a real Tangled PDS accepts these records, or that the reviewed lexicon matches what that server would accept. Those remain operator gates.