diff --git a/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md new file mode 100644 index 0000000..bee9c99 --- /dev/null +++ b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md @@ -0,0 +1,153 @@ +# `/remote/ls` drops the DataCite fields that are already on `Dataset.metadata` + +Status: **request** — small, self-contained, no new upstream calls, no new credential. +Reported by: the openwebindex.eu dashboard rebuild (ours-and-yoars-main, M5h), 2026-08-08. +Measured against: owilix **5.10.8** as deployed on `ourrs-dev` (`ours-owi-mirror-http`). + +## The ask, in one line + +Emit the DataCite fields on `/remote/ls` records — opt-in, e.g. +`?include=metadata` — using their **raw** values, not the display renderings. + +## The finding + +`_dataset_record()` in `owilix/http_api/server.py` builds each listing row by +hand, from a fixed set of eleven keys: + +```python +def _dataset_record(ds: Any) -> dict[str, Any]: + metadata = ds.metadata + record = { + "id": ..., "title": ..., "collectionName": ..., "dataCenter": ..., + "zone": ..., "access": ..., "startDate": ..., "endDate": ..., + "size": ..., "fileCount": ..., "objectCount": ..., + } +``` + +`ds.metadata` for the same dataset has **38 keys**. Listed live in-pod against +`lexis/collectionName=corpora`: + +``` +access, additionalMetadata, collectionName, contributor, creation_date, creator, +creators, dataCite.language, dataset_id, dataset_type, descriptions, endDate, +fileCount, fundingReferences, id, identifiers, internalID, lastChanged, +last_modified_date, location_name, objectCount, project_shortname, +publicationYear, publisher, relatedIdentifiers, resourceType, +resourceTypeGeneral, resource_hash, resource_name, rightsIdentifier, rightsList, +schema, startDate, title, titles, totalSize, types, user, zone +``` + +So the descriptive record is retrieved, carried all the way onto the `Dataset`, +and then dropped one function before the response. There is no response model +enforcing this — `/remote/ls` is declared as an untyped object with +`additionalProperties: true` — so widening it is a change to one helper. + +### Two details that surprised us, worth stating + +**1. There is no nested `datacite` key.** `OwiDDIRepository.get_all_datasets` +does `all_records[-1]["datacite"] = record["datacite"]`, so we expected one. +By the time the record reaches `Dataset.metadata` the DataCite fields are +**flattened at the top level** (`creators`, `descriptions`, `publisher`, +`publicationYear`, `rightsList`, `relatedIdentifiers`, `dataCite.language`). +Our first parser looked for the nested block and would have matched nothing, +silently, for ever. Either shape is fine to emit — please just say which. + +**2. `metadata.get(...)` returns a display rendering, not the value.** This is +the important one. Reading the fields off `DatasetMetadata` gives: + +``` +creators 'Lukas Gienapp+7' +relatedIdentifiers 'https://huggingface.co/datasets/coral-nlp/german-commons+1' +descriptions 'Large language model development relies on large-s...' +publisher {'name': 'CORAL Project'} +``` + +`+7` means "and seven more"; the abstract is truncated with an ellipsis. Those +are fine for a terminal listing and useless to a consumer — a page cannot show +eight authors from `+7`, and a truncated abstract cannot be un-truncated. So +adding `md.get("creators")` to the record would look like a fix and ship lossy +data. **The request is for the underlying values.** + +`DatasetMetadata.to_dict()` is not the escape hatch either: on the same record +it returns 13 keys, and its `datacite` entry contains only `fundingReferences`. + +## Why it matters + +Not for the crawl datasets — their titles are machine specifiers and nobody +reads their abstract. It matters for the **corpora**: datasets filed under +`collectionName` of `corpora` or `special`. Those are curated publications and +the descriptive record *is* the product. Four on this deployment, and all four +have real provenance sitting behind the projection: + +| title | creators | publisher | year | licence | related | +| --- | --- | --- | --- | --- | --- | +| German Commons | Lukas Gienapp +7 | CORAL Project | 2025 | — | HuggingFace | +| OWSI — Aggregated Web Crawl | Granitzer, Michael +2 | University of Passau | 2026 | — | Zenodo | +| German Imprints Dataset | Michael Dinzinger +6 | University of Passau | 2026 | — | — | +| Open Web Search Curlie 2025 | OpenWebSearch.eu Consortium | OpenWebSearch.eu Consortium | 2026 | OWIL V1.0 | DOI +94 | + +Today the public page for these can show a title, a size, a file count and a +link. It cannot say who made them or under what licence — while owilix is +holding all of it in memory one call earlier. + +The alternative we deliberately did not take: giving the dashboard's API its own +LEXIS credential. `LEXIS_ADMIN_REFRESH_TOKEN` is not to be adopted by other +services, and owilix already holds a working LEXIS session — a second one would +be a second thing to rotate for no gain. + +## Shape we would consume + +Preferred: **`GET /remote/ls?include=metadata`** adds the DataCite fields to +each record, raw. Opt-in matters — the unfiltered listing is 1,864 records here +and most callers want the eleven flat keys. + +Acceptable alternative: return them from +`GET /remote/datasets/{dataset_id}/summary`, which today carries mirror and +verification state only. Four extra calls a night is fine for four corpora; it +would not be fine as the only route if a caller ever needs this for the whole +catalogue. + +Fields we read, all optional — anything absent stays absent rather than being +blanked: `descriptions` (preferring `descriptionType: Abstract`), `creators`, +`publisher`, `publicationYear`, `rightsList[].rights` / `.rightsURI`, +`relatedIdentifiers[].relatedIdentifier`, `dataCite.language`. We store the +whole block alongside the derived columns, so an unusual or partial record +costs us nothing. + +## While you are in that code: `DatasetMetadata` is not a usable mapping + +Separate, small, and it will bite whoever implements the above — because the +obvious implementation is `dict(ds.metadata)`, and that raises. + +``` + in keys() md[k] md.get(k) +endDate True KeyError: 'endDate' None +startDate True KeyError: 'startDate' None +creators True 'Lukas Gienapp+7' 'Lukas Gienapp+7' +title True 'German Commons' 'German Commons' + +dict(md) -> KeyError: 'startDate' +``` + +`keys()` advertises keys that `__getitem__` refuses. For a corpus there is +legitimately no `startDate` — corpora are not dated crawl partitions — so the +value being absent is correct; what is wrong is that `keys()` lists it anyway +while `__getitem__` raises rather than returning the same `None` that `.get()` +does. + +Consequences: + +- `dict(md)`, `{**md}`, `json.dumps(dict(md))` and any `for k, v in md.items()` + loop all fail on exactly the records that matter here. +- It is presumably why `_dataset_record()` hand-picks keys with `.get()` in the + first place. + +Either make `__getitem__` agree with `.get()` (return the value, `None` when +unset), or make `keys()` report only keys that resolve. Either is fine; the two +disagreeing is the problem. + +## Not part of this request + +Nothing about the mirror, transfers, or the listing's own correctness. One +projection on a read path, plus one mapping-contract fix in the object it reads +from. diff --git a/docs/review/open_asks_against_5.10.8_2026-08-08.md b/docs/review/open_asks_against_5.10.8_2026-08-08.md new file mode 100644 index 0000000..d7f6cc5 --- /dev/null +++ b/docs/review/open_asks_against_5.10.8_2026-08-08.md @@ -0,0 +1,81 @@ +# What is still open for owilix, measured against 5.10.8 + +Status: **index** — supersedes the running list in +[`owilix_improvement_brief_2026-08-07.md`](owilix_improvement_brief_2026-08-07.md). +Verified 2026-08-08 against owilix **5.10.8** as deployed on `ourrs-dev` +(`ours-owi-mirror-http`), by probing the running server rather than by reading +release notes. + +Most of the brief has shipped. This exists so nobody re-implements the closed +half, and so the two genuinely open items are not buried in a 200-line document. + +## Closed — verified live, no action + +| Item | Evidence | +| --- | --- | +| **Part 1** — the silent-empty-listing / stale-ETag class, all five asks | released v5.9.0; [`silent_empty_listing_stale_etag_2026-08-07.md`](silent_empty_listing_stale_etag_2026-08-07.md) | +| **Part 2** — per-collection metadata index, Parquet not SQLite, rebuildable | `_index.parquet` + `_collection_summary.json` are written (5.10.4/5.10.6), and `owilix remote reindex` exists in 5.10.8 | +| **Part 4a** — observed vs declared integrity | `owilix remote verify`; `GET /remote/collections/{collection}/verify`, `GET /remote/datasets/{id}/summary`, `GET /remote/catalog/summary` | +| **Part 4b** — partial-listing contract | `/remote/ls` returns `partial: false, failed: []` beside `total`; consumed by us and load-bearing | +| `remote pull --push-to` fallback | [`remote_pull_and_mirror_operability_2026-08-04.md`](remote_pull_and_mirror_operability_2026-08-04.md), closed | + +`summarize`, `summarize-hosts` and `catalog` are all present too. Thank you — +the partial contract in particular is the thing that lets us tell "empty" from +"broken" on our side, and we rely on it in the catalog snapshot job. + +## Open — new, from the corpora work (2026-08-08) + +1. **`/remote/ls` drops the DataCite fields it already holds.** One helper + projects `ds.metadata` (38 keys) down to eleven. The descriptive record — + creators, abstract, publisher, licence, related identifiers — is retrieved, + carried onto the `Dataset`, and discarded one function before the response. + Detail, including two traps that make the naive fix ship lossy data: + [`datacite_dropped_by_remote_ls_2026-08-08.md`](datacite_dropped_by_remote_ls_2026-08-08.md). + **This is the one that blocks a real corpus page.** + +2. **`DatasetMetadata` is not a usable mapping.** `keys()` advertises keys that + `__getitem__` raises `KeyError` on, so `dict(md)` fails on exactly the + records we care about. Same document, last section. Small, and whoever fixes + (1) will hit it. + +## Open — carried over from the brief, still true in 5.10.8 + +3. **`/health` is a literal and cannot fail** (§3.1). Confirmed: it answers + `{"ok": true, "service": "owilix-http", "version": "5.10.8"}` with no + dependency check, and there is no `/ready` among the 39 routes. As a + *liveness* probe that is correct and should stay. What is missing is a + readiness signal that goes unhealthy when the configured remotes cannot be + listed — which is precisely the state the v5.9.0 bug class produced, and the + state an orchestrator could have self-healed. + +4. **Pagination re-walks the backend for every page** (§3.4). Measured on the + full LEXIS listing (1,864 datasets): + + | request | wall clock | + | --- | --- | + | `limit=1&offset=0` | 7.90 s | + | `limit=1&offset=0` (immediate repeat) | 7.38 s | + | `limit=1&offset=1800` | 7.27 s | + + Two readings. First, a deep offset costs the same as a shallow one, so + paging is O(pages × full listing) — our nightly catalog snapshot pages at + `limit=200`, so it pays ten full LEXIS walks for one logical listing. + Second, the identical repeat is not faster, so whatever HTTP-side listing + cache exists is not serving this path. + + Not urgent for us — the job is nightly and 74 s is affordable — but it is + load on a partner endpoint that a single walk would avoid, and it will matter + to any interactive consumer. + +5. **Job durability for the delegation case** (§3.5). Not re-verified today. + Flagging only because we were bitten by the adjacent shape: a delegated + `remote.pull` outlived the worker that started it while its record lived + only in the in-memory job store, so a restart lost the record but not the + work. + +## Not asks + +§3.2 (the three-state cache warning) and §3.3 (filesystem lifecycle) were +written as design advice rather than defects, and Part 1 landing may well have +addressed the concrete cases behind them. We have nothing new to add unless the +symptom returns.