diff --git a/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md index bee9c99..6cf57de 100644 --- a/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md +++ b/docs/review/datacite_dropped_by_remote_ls_2026-08-08.md @@ -114,6 +114,64 @@ blanked: `descriptions` (preferring `descriptionType: Abstract`), `creators`, whole block alongside the derived columns, so an unusual or partial record costs us nothing. +## Backwards compatibility: sidecars vary, and already carry the fields + +Two things we checked before asking, because both change the shape of the work. + +**1. The mirrored (file-based) path already has the data too.** We assumed this +ask only helped the live LEXIS path and that mirrored datasets would need a +re-pull. Not so — listing `owi-up/collectionName=main` gives 43 metadata keys +per dataset, *more* than the LEXIS path's 42, including `creators`, +`descriptions`, `publisher`, `publicationYear`, `rightsList` and +`relatedIdentifiers`. **So no sidecar migration is required for this request.** +The fields are in the sidecars that were written at pull time; only the HTTP +projection loses them. + +**2. Sidecar format is not uniform, and a fix must not assume it is.** Across +the 585 mirrored `main` datasets on this one deployment there are **six +distinct key sets**, from 33 to 43 keys: + +| datasets | keys | difference from the most common shape | +| --- | --- | --- | +| 192 | 41 | — (baseline) | +| 189 | 43 | `+workflow_ex_id`, `+workflow_id` | +| 168 | 40 | `+provenance`, `−fundingReferences`, `−relatedIdentifiers` | +| 24 | 42 | `+provenance`, `+workflow_ex_id`, `+workflow_id`, `−fundingReferences`, `−relatedIdentifiers` | +| 11 | 41 | `+workflow_ex_id`, `+workflow_id`, `−fundingReferences`, `−relatedIdentifiers` | +| **1** | **33** | also missing `descriptions`, `publisher`, `publicationYear`, `rightsList`, `schema`, `types`, `contributors`, `dates` | + +Per-field presence over the same 585: + +``` +creators 585/585 schema 584/585 +descriptions 584/585 publisher 584/585 +publicationYear 584/585 rightsList 584/585 +relatedIdentifiers 381/585 <- only 65% +``` + +`schema` is `http://datacite.org/schema/kernel-4` on 584 and absent on the +remaining one. + +What that implies for the implementation: + +- **Every field must be optional.** `relatedIdentifiers` is missing from a third + of them and one sidecar has almost no provenance at all. A widened record must + omit what is absent rather than emit `null`-filled or partially-constructed + objects — a consumer has to be able to tell "not published" from "not + carried". +- **Do not key on a fixed key set or a key count.** Six shapes exist in *one* + mirror; other installations will have others. +- **`schema` is the right version marker** where present, and its absence must + be tolerated rather than treated as an error. +- **Do not require a re-pull or a sidecar rewrite.** Older mirrors are not ours + to migrate, and a change that only works for freshly pulled data leaves every + existing mirror behind. + +Sidecar migration *is* available to us if you would rather normalise the format +— `owilix remote reindex` already rebuilds the index from sidecars, so there is +a natural place for it — but please do not make it a **precondition** for +reading the fields, for the reason above. + ## While you are in that code: `DatasetMetadata` is not a usable mapping Separate, small, and it will bite whoever implements the above — because the