From 89bea6b44222553b9ef038654d3a8931c663e4cb Mon Sep 17 00:00:00 2001 From: mgrani Date: Sat, 8 Aug 2026 11:14:43 +0200 Subject: [PATCH] docs(review): sidecar compatibility evidence for the DataCite ask LOCAL COMMIT ONLY -- consortium-owned remote, never pushed from here. Two measurements that change the shape of the request, both taken against the running 5.10.8: 1. The mirrored (file-based) path already carries the DataCite fields. `owi-up/collectionName=main` yields 43 metadata keys per dataset, more than the live LEXIS path's 42, provenance included. So no sidecar migration is required -- the fields are in the sidecars written at pull time, and only the HTTP projection loses them. I had assumed the opposite. 2. Sidecar format is not uniform, so the fix must not assume a fixed shape. Across 585 mirrored `main` datasets on this one deployment: six distinct key sets, 33 to 43 keys, differing by `provenance`, `workflow_ex_id`/`workflow_id`, `fundingReferences` and `relatedIdentifiers`. One outlier at 33 keys is missing nearly all provenance. Per-field presence: creators 585/585, descriptions/publisher/publicationYear/rightsList 584/585, relatedIdentifiers only 381/585. `schema` is http://datacite.org/schema/kernel-4 on 584 and absent on one. Recorded as constraints: every field optional, no keying on a fixed key set or count, `schema` as the version marker where present with absence tolerated, and one malformed sidecar must not fail the listing. Migration of the sidecars is acceptable to us if owilix would rather normalise -- `remote reindex` is the natural place -- but must not be a precondition for reading the fields, since other installations have mirrors nobody will migrate. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_019nPNpEG2QRnmjxxn5LeEb7 --- ...atacite_dropped_by_remote_ls_2026-08-08.md | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md index bee9c99..6cf57de 100644 --- a/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md +++ b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md @@ -114,6 +114,64 @@ blanked: `descriptions` (preferring `descriptionType: Abstract`), `creators`, whole block alongside the derived columns, so an unusual or partial record costs us nothing. +## Backwards compatibility: sidecars vary, and already carry the fields + +Two things we checked before asking, because both change the shape of the work. + +**1. The mirrored (file-based) path already has the data too.** We assumed this +ask only helped the live LEXIS path and that mirrored datasets would need a +re-pull. Not so — listing `owi-up/collectionName=main` gives 43 metadata keys +per dataset, *more* than the LEXIS path's 42, including `creators`, +`descriptions`, `publisher`, `publicationYear`, `rightsList` and +`relatedIdentifiers`. **So no sidecar migration is required for this request.** +The fields are in the sidecars that were written at pull time; only the HTTP +projection loses them. + +**2. Sidecar format is not uniform, and a fix must not assume it is.** Across +the 585 mirrored `main` datasets on this one deployment there are **six +distinct key sets**, from 33 to 43 keys: + +| datasets | keys | difference from the most common shape | +| --- | --- | --- | +| 192 | 41 | — (baseline) | +| 189 | 43 | `+workflow_ex_id`, `+workflow_id` | +| 168 | 40 | `+provenance`, `−fundingReferences`, `−relatedIdentifiers` | +| 24 | 42 | `+provenance`, `+workflow_ex_id`, `+workflow_id`, `−fundingReferences`, `−relatedIdentifiers` | +| 11 | 41 | `+workflow_ex_id`, `+workflow_id`, `−fundingReferences`, `−relatedIdentifiers` | +| **1** | **33** | also missing `descriptions`, `publisher`, `publicationYear`, `rightsList`, `schema`, `types`, `contributors`, `dates` | + +Per-field presence over the same 585: + +``` +creators 585/585 schema 584/585 +descriptions 584/585 publisher 584/585 +publicationYear 584/585 rightsList 584/585 +relatedIdentifiers 381/585 <- only 65% +``` + +`schema` is `http://datacite.org/schema/kernel-4` on 584 and absent on the +remaining one. + +What that implies for the implementation: + +- **Every field must be optional.** `relatedIdentifiers` is missing from a third + of them and one sidecar has almost no provenance at all. A widened record must + omit what is absent rather than emit `null`-filled or partially-constructed + objects — a consumer has to be able to tell "not published" from "not + carried". +- **Do not key on a fixed key set or a key count.** Six shapes exist in *one* + mirror; other installations will have others. +- **`schema` is the right version marker** where present, and its absence must + be tolerated rather than treated as an error. +- **Do not require a re-pull or a sidecar rewrite.** Older mirrors are not ours + to migrate, and a change that only works for freshly pulled data leaves every + existing mirror behind. + +Sidecar migration *is* available to us if you would rather normalise the format +— `owilix remote reindex` already rebuilds the index from sidecars, so there is +a natural place for it — but please do not make it a **precondition** for +reading the fields, for the reason above. + ## While you are in that code: `DatasetMetadata` is not a usable mapping Separate, small, and it will bite whoever implements the above — because the -- 2.51.2