diff --git a/CHANGELOG.md b/CHANGELOG.md --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,25 @@ ## Unreleased +## v5.10.6 (2026-08-07) + +Closes F5 (write-back semantics) from the summarize/integrity plan, plus the housekeeping that recorded which acceptance criteria are still open. + +### Features + +- **summarize**: `--write-back / --no-write-back`. `--no-write-back` computes statistics and README content and reports them without touching the repository — a dry inspection of a mirror you do not own. +- **summarize**: `--summary --write-back` places the aggregate beside each collection as `_collection_summary.json`, instead of only on local disk where nobody reading the mirror can find it. One aggregate per collection, not per dataset; a write that fails is reported, never swallowed. + +### Bug Fixes + +- **repository**: owilix's own collection-level artifacts are no longer read as datasets. `list()` treats **any** `*.json` in a collection directory as a dataset sidecar — `check_path_for_uuid_filename` accepts every name that does not start with `.` — so writing an aggregate there would have created a **phantom dataset in every listing**, the same shape as the unrelated `latest.json` that once emptied a mirror listing. `COLLECTION_ARTIFACTS` now excludes them from listings, from the index, and from the staleness fingerprint (they are derived, so changing them does not make the index stale). Found before shipping the feature that would have caused it. + +### Documentation + +- **review**: Part III added to the summarize/integrity plan, recording what shipped against each acceptance criterion and what did not. Three gaps remain and are named: `remote ls --observed` (M1's stated acceptance — `ls` still reports declared numbers only), `verify --since` (F7), and a per-collection verification endpoint (M5 shipped in a different shape than specified). +- **backlog**: M3 marked delivered with a pointer to the open gaps. + + ## v5.10.5 (2026-08-07) M5 (HTTP exposure) plus incremental checksums, which M4's index work made worth building. diff --git a/Readme.md b/Readme.md --- a/Readme.md +++ b/Readme.md @@ -1,6 +1,6 @@ # OWILIX - Open Web Index CLI -[![Version](https://img.shields.io/badge/version-5.10.5-blue.svg)](https://openwebsearcheu-public.pages.it4i.eu/owi-cli/) +[![Version](https://img.shields.io/badge/version-5.10.6-blue.svg)](https://openwebsearcheu-public.pages.it4i.eu/owi-cli/) [![Python](https://img.shields.io/badge/python-3.11+-green.svg)](https://www.python.org/) [![License](https://img.shields.io/badge/license-Apache_2.0-orange.svg)](http://www.apache.org/licenses/LICENSE-2.0) diff --git a/pyproject.toml b/pyproject.toml --- a/pyproject.toml +++ b/pyproject.toml @@ -5,7 +5,7 @@ [project] name = "owilix" -version = "5.10.5" +version = "5.10.6" description = "OWILIX - the Command Line Interface for slicing and consuming the Open Web Index. " readme = "Readme.md" license = { text = "MIT" } diff --git a/uv.lock b/uv.lock --- a/uv.lock +++ b/uv.lock @@ -1102,7 +1102,7 @@ [[package]] name = "owilix" -version = "5.10.5" +version = "5.10.6" source = { editable = "." } dependencies = [ { name = "ciff-toolkit" }, diff --git a/docs/backlog.md b/docs/backlog.md --- a/docs/backlog.md +++ b/docs/backlog.md @@ -159,8 +159,9 @@ - **Why it matters to the consumer:** their catalog marks anything absent from the latest listing as `missing_since` and hides it. A credential lapse in one zone therefore retires that zone's collections labelled *"no longer listed upstream"* — a statement about the data when the truth was about the credential. They cannot defend against it because nothing in the response distinguishes the two. - **Cost:** changes the `/remote/ls` response shape and the `list()` signature, so it should not ride in a bugfix release. Additive fields make it a minor bump; the `remote pull` default change is the genuinely behaviour-altering part and could ship separately or behind a flag first. -### M3 — Verify a dataset, not just list it (`observed` vs `declared`) +### ~~M3 — Verify a dataset, not just list it (`observed` vs `declared`)~~ (delivered v5.10.1–v5.10.5) +- **Status:** delivered across v5.10.1–v5.10.5, superseded by the summarize/integrity plan. All three tiers exist (`owi remote verify --tier presence|readable|checksum`), results are stored beside the data as `checksums.json`, iRODS checksums are read where available, and the collection index makes incremental re-verification possible. **Four acceptance gaps remain** — `remote ls --observed`, `summarize --write-back`, `verify --since`, and a per-collection verification endpoint — recorded in Part III of [`review/summarize_and_integrity_feature_plan_2026-08-07.md`](review/summarize_and_integrity_feature_plan_2026-08-07.md). - **Proposal:** [`docs/review/dataset_integrity_verification_2026-08-07.md`](review/dataset_integrity_verification_2026-08-07.md) - **What:** the mirror writes a ~4 KB sidecar per dataset *and* the parquet. `remote ls` counts a dataset as present when the sidecar exists — defensible, but it means **a dataset whose transfer failed after the sidecar looks identical to one that completed**. `fileCount` does not disambiguate: it comes from the sidecar and describes the *upstream* dataset, so one reported `fileCount 567, 14.7 GB` against a single 4 KB object. Their catalog told visitors 586 collections were mirrored when 4 held data. - **Three tiers, in cost order:** diff --git a/docs/changes.md b/docs/changes.md --- a/docs/changes.md +++ b/docs/changes.md @@ -24,6 +24,24 @@ ## Unreleased +### main @ v5.10.6 - 2026-08-07 + +#### Write-back, and the phantom dataset it would have created + +**Summary**: F5 delivered, plus documentation housekeeping recording which acceptance criteria are still unmet. + +**Changes**: +- `summarize --write-back/--no-write-back`, and `--summary --write-back` placing `_collection_summary.json` beside each collection. +- **The guard that had to come first**: `list()` treats any `*.json` in a collection directory as a dataset sidecar, so the aggregate would have appeared in every listing as a phantom — the `latest.json` failure shape, self-inflicted. `COLLECTION_ARTIFACTS` excludes owilix's own collection-level artifacts from listings, the index, and the fingerprint. Verified on the fixture: 3 datasets before writing the aggregate, 3 after. +- Part III of the plan now records shipped-vs-open per acceptance criterion, after "essentially complete" was claimed here once and was wrong. + +**Tested against the local file:// fixture only.** `--write-back` writes into the repository, and the LEXIS credential in this environment is currently valid, so running it against a real specifier would have written to production storage. + +**Breaking Changes**: None. `--write-back` defaults to true, which is the previous behaviour. + +**Still open**: `remote ls --observed` (the number a catalogue reads is still declared-only), `verify --since`, and a per-collection verification endpoint. + + ### main @ v5.10.5 - 2026-08-07 #### Derived artifacts over HTTP, and checksums that can actually be re-run diff --git a/owilix/_version.py b/owilix/_version.py --- a/owilix/_version.py +++ b/owilix/_version.py @@ -1,3 +1,3 @@ # Version is set here and imported elsewhere -__version__ = "5.10.5" -__version_tuple__ = (5, 10, 5) +__version__ = "5.10.6" +__version_tuple__ = (5, 10, 6) diff --git a/docs/review/summarize_and_integrity_feature_plan_2026-08-07.md b/docs/review/summarize_and_integrity_feature_plan_2026-08-07.md --- a/docs/review/summarize_and_integrity_feature_plan_2026-08-07.md +++ b/docs/review/summarize_and_integrity_feature_plan_2026-08-07.md @@ -451,3 +451,88 @@ a human decides. Derived descriptive artifacts may be written back; verification verdicts go to a separate namespaced artifact with provenance. 5. **No silent caps.** Anything that bounds coverage logs what it dropped. + +--- + +# Part III — What shipped, and what did not + +Added 2026-08-07 after M0–M6 were released across v5.10.1–v5.10.5. Recorded +against the acceptance criteria in Part II rather than against a memory of the +work, because "essentially complete" was claimed once here and was wrong. + +## Delivered + +| milestone | release | notes | +| --- | --- | --- | +| M0 provenance + honest empties | v5.10.1 | `computed_at`, `owilix_version`, `parquet_files_found` | +| M1 observed vs declared | v5.10.1 | in `stats.json`; **see gap 1** | +| M2 content-addressed writes | v5.10.1 | `--force` no longer rewrites identical bytes | +| M3 README front matter | v5.10.2 | YAML head; nearly re-introduced M2's churn, see below | +| M6 tiers 2–3 | v5.10.3 | `owi remote verify`, format validation, `checksums.json` | +| M4 index + `reindex` | v5.10.4 | fallback on absent / unreadable / stale / unknown-format | +| M5 HTTP exposure | v5.10.5 | **shape differs from spec, see gap 4** | +| M6 incremental | v5.10.5 | digest reuse on unchanged size + mtime | + +## Not delivered + +Four gaps when this was written; **gap 2 closed in v5.10.6**. Gap 1 is the one that still matters operationally. + +**1. `remote ls --observed` (M1 acceptance).** M1's acceptance criterion was +*"the summary line stops asserting 570 files when 3 exist"*. `observed` reached +`stats.json` and the HTTP endpoints but never `ls`, so the command a catalogue +is most likely to read **still reports declared numbers only**. On the +three-dataset fixture it still prints `570 files, 250,200 objects` where 3 files +and 200 objects exist. This is the headline of the 586-vs-4 problem, unfixed at +the CLI. + +**2. ~~`remote summarize --write-back / --no-write-back`, and `--summary +--write-back` (Part I F5)~~ — delivered in v5.10.6.** `--no-write-back` computes +and reports without touching the repository; `--summary --write-back` places the +aggregate beside each collection as `_collection_summary.json`. + +**F5 was never given a milestone in Part II** — it went missing through a gap in +the plan rather than a decision to defer it, which is worth noting because a +plan that silently drops a requirement is harder to catch than one that defers +it loudly. + +Building it surfaced a hazard that had to be fixed first: `list()` treats *any* +`*.json` in a collection directory as a dataset sidecar, because +`check_path_for_uuid_filename` accepts every name not starting with ".". Writing +an aggregate there would therefore have produced a **phantom dataset in every +listing** — the same shape as the unrelated `latest.json` that emptied a mirror +listing. `COLLECTION_ARTIFACTS` now excludes owilix's own collection-level +artifacts from listings, from the index, and from the staleness fingerprint. + +**3. `verify --since` (F7).** `--max-files`, `--max-datasets`, `--only-missing` +and `--recheck-all` all exist. Time-bounding does not. + +**4. M5 shipped in a different shape than specified.** Part II named +`GET /datasets/{id}/summary`, `GET /collections/{c}/index` and +`GET /collections/{c}/verify`. What exists is +`GET /remote/datasets/{id}/summary` and `GET /remote/catalog/summary`. The +summary half is covered; **there is no endpoint returning last-verification per +dataset across a collection**, which is what a dashboard health view would poll. +The substitution should have been flagged when it was made. + +## Three things this work taught that are worth keeping + +**Every derived artifact tried to break the thing it describes.** M2 found +`--force` rewriting identical objects; M3 nearly reinstated that by putting a +timestamp in the README; M6 had `checksums.json` verifying itself, which cannot +converge by construction. The pattern is consistent enough to be a design rule: +*when adding an artifact that describes the data, check what its own existence +does to the read path.* + +**Two features were correct code that could not see its inputs.** Incremental +digest reuse and "prefer the stored iRODS checksum" were both written, both +tested, and both incapable of firing against a real repository, because +`fill_file_details` fetched only `size`. Tests that inject the input they need +cannot catch this. It surfaced only from running the command against a real +collection. + +**Tests passed for the wrong reason twice.** The verify fixture wrote its +parquet twice, so the recorded mtime never matched and every case took the +"changed" branch. A new HTTP test read a `NameError` as "no stats", because +`_read_stats_json` catches broadly. Both passed while asserting nothing about +the intended path. A helper that swallows exceptions turns a test defect into a +plausible result. diff --git a/owilix/cli/remote.py b/owilix/cli/remote.py --- a/owilix/cli/remote.py +++ b/owilix/cli/remote.py @@ -671,6 +671,11 @@ "--languages-topk", help="Top-K languages to include in summary/Markdown report (0 to disable)", ), + write_back: bool = typer.Option( + True, "--write-back/--no-write-back", + help="Upload README/stats back to the repository. With --summary, also place " + "the aggregate beside each collection instead of only on local disk.", + ), markdown_report: bool = typer.Option( False, "--markdown-report", @@ -729,6 +734,7 @@ topk_collection="all", command=command, markdown_report=markdown_report, + write_back=write_back, console=cli_ctx.console, ) @@ -748,6 +754,7 @@ details=details, hosts_topk=hosts_topk, hosts_only=hosts_only, + write_back=write_back, console=cli_ctx.console, auto_yes=yes or cli_ctx.auto_yes ) diff --git a/owilix/core/index.py b/owilix/core/index.py --- a/owilix/core/index.py +++ b/owilix/core/index.py @@ -41,6 +41,7 @@ __all__ = [ "INDEX_FILENAME", + "COLLECTION_ARTIFACTS", "INDEX_FORMAT_VERSION", "collection_fingerprint", "build_index_table", @@ -50,6 +51,17 @@ ] INDEX_FILENAME = "_index.parquet" + +#: Collection-level artifacts owilix writes itself. +#: +#: These live in the collection directory beside the dataset sidecars, and must +#: never be mistaken for one. `check_path_for_uuid_filename` accepts any name +#: that does not start with ".", so a `*.json` artifact here would be read as a +#: dataset and appear in every listing as a phantom -- the same shape as the +#: unrelated `latest.json` that emptied a mirror listing. Excluded from +#: listings, from the index, and from the staleness fingerprint (they are +#: derived, so their changing does not make the index stale). +COLLECTION_ARTIFACTS = (INDEX_FILENAME, "_collection_summary.json") #: Bumped when the schema changes in a way an older reader cannot handle. A #: reader that sees a version it does not know falls back to the walk rather @@ -79,7 +91,7 @@ for entry in fs.ls(collection_dir, detail=True): name = entry["name"] if isinstance(entry, dict) else entry base = os.path.basename(name.rstrip("/")) - if base == INDEX_FILENAME or not name.endswith(".json"): + if base in COLLECTION_ARTIFACTS or not name.endswith(".json"): continue info = entry if isinstance(entry, dict) else {} mtime = info.get("mtime") or info.get("LastModified") or info.get("last_modified") @@ -220,6 +232,8 @@ for entry in fs.ls(collection_dir, detail=True): path = entry["name"] if isinstance(entry, dict) else entry if not path.endswith(".json"): + continue + if os.path.basename(path) in COLLECTION_ARTIFACTS: continue dataset_dir = path[: -len(".json")] if not check_path_for_uuid_filename(os.path.basename(dataset_dir)): diff --git a/owilix/core/repository/file.py b/owilix/core/repository/file.py --- a/owilix/core/repository/file.py +++ b/owilix/core/repository/file.py @@ -312,6 +312,11 @@ path = entry["name"] if isinstance(entry, dict) else entry if not path.endswith(".json"): continue + # owilix's own collection-level artifacts are not datasets. + from owilix.core.index import COLLECTION_ARTIFACTS + + if os.path.basename(path) in COLLECTION_ARTIFACTS: + continue dataset_dir = path[: -len(".json")] if not check_path_for_uuid_filename(os.path.basename(dataset_dir)): continue diff --git a/owilix/core/tasks/remote.py b/owilix/core/tasks/remote.py --- a/owilix/core/tasks/remote.py +++ b/owilix/core/tasks/remote.py @@ -3020,6 +3020,7 @@ details: bool = False, hosts_topk: int = 1000, hosts_only: bool = False, + write_back: bool = True, console: Optional[Console] = None, auto_yes: bool = False ) -> CommandResult: @@ -3355,10 +3356,16 @@ if create_readme: readme_content = _generate_readme_markdown(d, stats) - # Write files, but only if they would actually change. - _write_if_changed(d, "stats.json", json.dumps(stats, indent=2), console=console) - if create_readme and readme_content is not None: - _write_if_changed(d, "README.md", readme_content, console=console) + # Write files, but only if they would actually change -- and + # only if writing back was asked for. --no-write-back computes + # and reports without touching the repository, which is what a + # dry inspection of a mirror someone else owns needs. + if write_back: + _write_if_changed(d, "stats.json", json.dumps(stats, indent=2), console=console) + if create_readme and readme_content is not None: + _write_if_changed(d, "README.md", readme_content, console=console) + else: + console.print(" [dim]--no-write-back: computed, not uploaded[/dim]") console.print(f"[green] ✓ Completed {title}[/green]") processed += 1 @@ -3393,6 +3400,7 @@ topk_collection: Optional[str] = "main", command: Optional[str] = None, markdown_report: bool = False, + write_back: bool = False, console: Optional[Console] = None, ) -> CommandResult: if console is None: @@ -3661,6 +3669,17 @@ with open(output_file, "w") as f: json.dump(summary, f, indent=2) console.print(f"\n[green]Summary written to {output_file}[/green]") + + if write_back: + written, failed = _write_collection_summary(datasets, summary, console=console) + if written: + console.print( + f"[green]Aggregate written beside {len(written)} collection(s).[/green]" + ) + for problem in failed: + # Never silent: an aggregate that failed to land is not an + # aggregate that was not asked for. + console.print(f"[red] could not write aggregate: {problem}[/red]") console.print("\n[bold]Summary Statistics:[/bold]") console.print(f" Datasets processed: {processed}") @@ -4694,3 +4713,46 @@ console.print(", ".join(_ronly)) return CommandResult(success=True, object=None, msg="Diff complete") + + +def _write_collection_summary(datasets, summary: Dict, console: Optional[Console] = None): + """Place the aggregate beside each collection it describes. + + `--summary` alone writes only to local disk, which is useless to anyone + reading the mirror. This puts the same document at the collection root so a + consumer with read access to the store can find it without owilix. + + The filename is one of `COLLECTION_ARTIFACTS`, and that is load-bearing: + `list()` treats any `*.json` in a collection directory as a dataset sidecar + unless it is excluded, so an aggregate written here without that guard would + appear in every listing as a phantom dataset -- the same shape as the + unrelated `latest.json` that once emptied a mirror listing. + """ + from owilix.core.index import COLLECTION_ARTIFACTS + + name = "_collection_summary.json" + assert name in COLLECTION_ARTIFACTS, "the aggregate name must be excluded from listings" + + # One aggregate per collection directory, deduplicated by path. + targets = {} + for dataset in datasets: + repository = getattr(dataset, "repository", None) + path = getattr(dataset, "path", None) + if repository is None or not path: + continue + collection_dir = os.path.dirname(str(path).rstrip("/")) + if collection_dir: + targets.setdefault(collection_dir, repository) + + payload = json.dumps(summary, indent=2, default=str) + written, failed = [], [] + for collection_dir, repository in sorted(targets.items()): + target = os.path.join(collection_dir, name) + try: + with repository.fs.open(target, "w") as handle: + handle.write(payload) + written.append(target) + except Exception as e: + logger.error(f"Could not write collection summary {target}: {e}") + failed.append(f"{target}: {e}") + return written, failed diff --git a/tests/owilix/core/test_index.py b/tests/owilix/core/test_index.py --- a/tests/owilix/core/test_index.py +++ b/tests/owilix/core/test_index.py @@ -230,3 +230,55 @@ _build(tmp_path, repo) assert len(read_index(repo, _collection_dir(tmp_path), "public")) == 2 + + +class TestCollectionArtifactsAreNotDatasets: + """A `*.json` at collection level would otherwise be read as a sidecar. + + `check_path_for_uuid_filename` accepts any name not starting with ".", so + writing `_collection_summary.json` beside the sidecars without an explicit + exclusion produces a phantom dataset in every listing -- the same shape as + the unrelated `latest.json` that emptied a mirror listing. This is the guard + that lets `summarize --summary --write-back` exist at all. + """ + + def _write_artifact(self, tmp_path, name, collection="main", access="public"): + base = tmp_path / access / collection + base.mkdir(parents=True, exist_ok=True) + (base / name).write_text(json.dumps({"generated": True}), encoding="utf-8") + + def test_the_aggregate_is_not_listed_as_a_dataset(self, tmp_path): + _write_dataset(tmp_path) + self._write_artifact(tmp_path, "_collection_summary.json") + repo = _repo(tmp_path) + + assert len(repo.list("public")) == 1 + + def test_it_is_not_indexed_as_a_dataset(self, tmp_path): + _write_dataset(tmp_path) + self._write_artifact(tmp_path, "_collection_summary.json") + repo = _repo(tmp_path) + + report, skipped = _build(tmp_path, repo) + + assert report["datasets"] == 1 + assert skipped == [], "it is excluded, not treated as an unreadable sidecar" + + def test_it_does_not_make_the_index_stale(self, tmp_path): + """It is derived; writing it must not invalidate the index.""" + _write_dataset(tmp_path) + repo = _repo(tmp_path) + _build(tmp_path, repo) + + self._write_artifact(tmp_path, "_collection_summary.json") + + # Still served from the index rather than falling back. + assert len(read_index(repo, _collection_dir(tmp_path), "public")) == 1 + + def test_a_normal_json_is_still_treated_as_a_dataset(self, tmp_path): + """The exclusion must be by exact name, not a blanket underscore rule.""" + _write_dataset(tmp_path) + self._write_artifact(tmp_path, "_something_else.json") + repo = _repo(tmp_path) + + assert len(repo.list("public")) == 2 diff --git a/tests/owilix/core/tasks/test_summarize_integrity.py b/tests/owilix/core/tasks/test_summarize_integrity.py --- a/tests/owilix/core/tasks/test_summarize_integrity.py +++ b/tests/owilix/core/tasks/test_summarize_integrity.py @@ -318,3 +318,89 @@ dataset = self._dataset_holding("# Old README\n") assert _write_if_changed(dataset, "README.md", "# New README\n") is True + + +class TestWriteBack: + """F5 -- summarize must be able to not write, and to place the aggregate.""" + + def test_the_aggregate_name_is_excluded_from_listings(self): + """Load-bearing: without this the aggregate becomes a phantom dataset.""" + from owilix.core.index import COLLECTION_ARTIFACTS + + assert "_collection_summary.json" in COLLECTION_ARTIFACTS + + def _repo_capturing_writes(self): + written = {} + + class _FS: + def open(self, path, mode="rb"): + from io import StringIO + + class _H(StringIO): + def __exit__(_self, *a): + written[path] = _self.getvalue() + return False + + return _H() + + return SimpleNamespace(fs=_FS()), written + + def _dataset_at(self, repo, path): + return SimpleNamespace(repository=repo, path=path) + + def test_one_aggregate_per_collection(self): + from owilix.core.tasks.remote import _write_collection_summary + + repo, written = self._repo_capturing_writes() + datasets = [ + self._dataset_at(repo, "/store/public/main/ds-1"), + self._dataset_at(repo, "/store/public/main/ds-2"), + self._dataset_at(repo, "/store/public/other/ds-3"), + ] + + paths, failed = _write_collection_summary(datasets, {"totals": {"files": 3}}) + + assert failed == [] + assert sorted(paths) == [ + "/store/public/main/_collection_summary.json", + "/store/public/other/_collection_summary.json", + ], "one per collection, not one per dataset" + + def test_the_aggregate_content_is_the_summary(self): + from owilix.core.tasks.remote import _write_collection_summary + + repo, written = self._repo_capturing_writes() + summary = {"totals": {"files": 7}, "datasets": 2} + + _write_collection_summary([self._dataset_at(repo, "/store/public/main/ds-1")], summary) + + assert json.loads(written["/store/public/main/_collection_summary.json"]) == summary + + def test_a_failed_write_is_reported_not_swallowed(self): + """An aggregate that failed to land is not one that was not asked for.""" + from owilix.core.tasks.remote import _write_collection_summary + + class _FS: + def open(self, path, mode="rb"): + raise OSError("permission denied") + + repo = SimpleNamespace(fs=_FS()) + + paths, failed = _write_collection_summary( + [self._dataset_at(repo, "/store/public/main/ds-1")], {} + ) + + assert paths == [] + assert len(failed) == 1 and "permission denied" in failed[0] + + def test_datasets_without_a_path_are_skipped_not_fatal(self): + from owilix.core.tasks.remote import _write_collection_summary + + repo, _ = self._repo_capturing_writes() + datasets = [SimpleNamespace(repository=repo, path=None), + self._dataset_at(repo, "/store/public/main/ds-1")] + + paths, failed = _write_collection_summary(datasets, {}) + + assert paths == ["/store/public/main/_collection_summary.json"] + assert failed == []