diff --git a/docs/design/oura-import.md b/docs/design/oura-import.md index ccb21dddd..449e2b9a5 100644 --- a/docs/design/oura-import.md +++ b/docs/design/oura-import.md @@ -295,6 +295,95 @@ endpoints backfill from the horizon automatically). --- +## 9a. Amendments — 2026-07-07 (granted-scope endpoints + blood_glucose partner-gating) + +The owner reauthorized with the full printed scope set and directed that +every newly granted endpoint "make it into solstone". All shapes below +were verified two ways on 2026-07-07: against openapi-1.35 (fetched from +Oura's docs) and by read-only live GET probes with the freshly granted +token (no `--save` run, no journal writes). + +### Four endpoints join `SYNC_ENDPOINTS` + +| Endpoint | Live rows (probe) | Shape (openapi-1.35, live-confirmed) | Normalization | +|---|---|---|---| +| `workout` | 93 since Jun 1; 390 since 2024-01-01 | required `id, activity, day, start_datetime, end_datetime, intensity (easy/moderate/hard), source (manual/autodetected/confirmed/workout_heart_rate)`; nullable `calories` (kcal), `distance` (m), `label` | `oura.workout`, kind=`workout` (mirrors the AH workout event shape); no scalar value; activity/intensity/source/label/calories/distance in metadata | +| `session` | 1 row | required `id, day, start_datetime, end_datetime, type (breathing/meditation/nap/relaxation/rest/body_status)`; nullable `mood (bad/worse/same/good/great)` and `heart_rate`/`heart_rate_variability`/`motion_count` sample blocks `{interval, items[], timestamp}` | `oura.session`, kind=`session`; no scalar value; metadata carries `type`/`mood` only — sample blocks stay in the raw page (raw_ref), never in normalized rows | +| `enhanced_tag` | 2 rows | required `id, start_time, start_day` — **the one document endpoint with no `day` field**; nullable `tag_type_code, end_time, end_day, comment, custom_name` | `oura.enhanced_tag`, kind=`tag`; journal day = `start_day` verbatim (`_DOCUMENT_DAY_FIELDS`); tag text fields are metadata, never a value | +| `vO2_max` | 0 rows (endpoint valid: 200 empty page; lowercase `vo2_max` 404s — the route casing is exact) | required `id, day, timestamp, vo2_max (integer)` (PublicVO2Max) | `oura.vo2_max`, kind=`daily_summary`, value=`vo2_max`, unit `mL/kg/min` (by definition; the spec carries no unit field); friendly name "VO2 max" | + +**Timestamp finding (load-bearing).** Workout/session datetimes and tag +times are wearer-local offset instants (`LocalizedDateTime` / +`LocalDateTime`; live workout rows carry `-04:00`…`-07:00` across +travel/DST), **not** UTC-Z — so unlike heartrate there is no +UTC→owner-local conversion: datetimes pass through verbatim and the +journal day is Oura's day field verbatim (a 23:12 local workout whose +UTC instant crosses midnight stays on its local day; timezone-pinned in +tests). + +**Window limits.** All four are day-paged (`start_date`/`end_date`). +Live probes accepted a 2.5-year window on each (200), so the default +364-day chunking is comfortably safe; no per-endpoint cap entries were +added. Horizon backfill from 2015-01-01 is ~12 chunks per endpoint. + +**Cursor upgrade.** Exactly the §9 semantics: the four endpoints are +missing from existing cursors, so the first post-upgrade save walks each +from `BACKFILL_HORIZON_DAY`; endpoints that come back empty (vO2_max, +likely) are marked `backfill_complete` and poll a 30-day trailing window +thereafter. + +### blood_glucose is partner-gated — demoted from the poll set + +Portal finding (owner, 2026-07): the developer portal shows **all** +grantable scopes already enabled and **no `metabolic` option** — +blood_glucose is available only to Oura partner integrations +(Tidepool-class). Every poll would 401 forever, and the hourly lane was +reporting that error each cycle. Resolution: + +- `_PARTNER_GATED_ENDPOINTS = ("blood_glucose",)` in `oura.py`; + `SYNC_ENDPOINTS` no longer contains it, so syncs make zero + blood_glucose requests and report zero errors (test-pinned). +- The parse/normalize/dedupe machinery, fixture, and §9 pinned + assumptions all stay wired for a future partner grant or file import. +- The cursor never carries the endpoint (stale entries from the 2026-07 + cursor generation are dropped on the next save rewrite), so it is + never marked `backfill_complete` — re-enabling is one line (move the + name back into `SYNC_ENDPOINTS`) and still backfills from the horizon. + +### Display-pass notes (body app — NOT changed by this lane) + +The same ring's workouts arrive through both pipes: `oura.workout` rows +(this lane) and AH-mirror `HKWorkoutActivityType*` rows sourced "Oura". +Day-level aggregation must keep one canonical pipe (O-5C). The exact +one-line seam for `_MIRROR_SUPERSEDED_FRAGMENTS` in +`solstone/apps/body/routes.py`: + +```python +OURA_WORKOUT_TYPE: ("WorkoutActivityType",), # OURA_WORKOUT_TYPE = "oura.workout" +``` + +(The fragment must NOT be spelled with the `HK` prefix: `HK`-prefixed +fragments match exactly one identifier, while the bare fragment +substring-matches every `HKWorkoutActivityType*` activity. The +`_is_oura_named_mirror_row` guard already restricts the drop to +Oura-sourced mirror rows, so Watch/iPhone workouts are untouched.) + +Also for the display pass: `oura.workout` rows carry Oura's field names +in metadata (`calories` kcal, `distance` m, `activity`), not the AH +`totalEnergyBurned`/`totalDistance` keys `_workout_metrics` reads — the +day-card metrics line needs a small mapping (or the generic fallback) +if Oura workout calories/distance should render; duration already works +(computed from `start_date`/`end_date`). The card name currently +renders the friendly type ("Workout"); `metadata["activity"]` has the +specific activity if wanted. + +Operator step after this lands (owner-side backfill): +`journal importer --sync oura --save --confirm-health-save` — the four +new endpoints backfill from the horizon automatically; no +reauthorization needed (the scopes were granted 2026-07-07). + +--- + ## 10. What landed with this doc (phase O0 inventory) - `solstone/think/importers/oura.py` — parse layer (`parse_oura_bundle`, `parse_endpoint_document`, `parse_oura_day`), normalizer (`normalize_bundle` → rows + `HealthDedupeRecord`s via `health_schema`), §13 copy reference (`render_day_summary`), `OuraImporter` (detect/preview/dry-run live; save gated then seamed), `OuraSyncBackend` + OAuth seams (all raise, pointing here). Zero network imports, test-enforced. diff --git a/solstone/think/importers/health_schema.py b/solstone/think/importers/health_schema.py index 56065a64c..a63c8912f 100644 --- a/solstone/think/importers/health_schema.py +++ b/solstone/think/importers/health_schema.py @@ -96,6 +96,14 @@ FRIENDLY_TYPE_NAMES: Final[Mapping[str, str]] = { "oura.heartrate": "Heart rate", "oura.daily_cardiovascular_age": "Cardiovascular age", "oura.blood_glucose": "Blood glucose", + # 2026-07-07 granted-scope expansion. Workouts/sessions/tags are + # event rows (no scalar value); the generic name labels the kind and + # detail (activity, session type, tag text) lives in row metadata for + # the display side to surface. + "oura.workout": "Workout", + "oura.session": "Session", + "oura.enhanced_tag": "Tag", + "oura.vo2_max": "VO2 max", } # Owner-facing names for Oura score-contributor keys — the anatomies of diff --git a/solstone/think/importers/oura.py b/solstone/think/importers/oura.py index 7b6b80d28..4949adb89 100644 --- a/solstone/think/importers/oura.py +++ b/solstone/think/importers/oura.py @@ -25,13 +25,21 @@ Scope of this module: Timezone rule (load-bearing): Oura documents carry their own ``day`` field, already attributed by Oura (a night belongs to the day it ended, -matching the journal's cross-midnight canon). The journal day IS Oura's -``day`` field verbatim — never recomputed against local time. The -instant-only series (``heartrate``, and ``blood_glucose`` by pinned -assumption — see ``_normalize_item``) are the exception: they carry no -``day`` field and Oura returns UTC instants, so samples are converted to -the owner's journal timezone for ``start_date`` and day/month assignment -while the raw timestamp stays in ``source_record_id`` for stable dedupe. +matching the journal's cross-midnight canon; ``enhanced_tag`` spells its +day ``start_day`` — see ``_DOCUMENT_DAY_FIELDS``). The journal day IS +Oura's day field verbatim — never recomputed against local time, and +document datetimes (sleep bedtimes, workout/session intervals, tag +times) are wearer-local offset instants kept verbatim. The instant-only +series (``heartrate``, and ``blood_glucose`` by pinned assumption — see +``_normalize_item``) are the exception: they carry no ``day`` field and +Oura returns UTC instants, so samples are converted to the owner's +journal timezone for ``start_date`` and day/month assignment while the +raw timestamp stays in ``source_record_id`` for stable dedupe. + +Endpoint roster: ``SYNC_ENDPOINTS`` is what the engine polls; +``_PARTNER_GATED_ENDPOINTS`` (blood_glucose) stay fully wired for parse/ +normalize/dedupe but are never fetched — Oura grants their scope only to +partner integrations (2026-07 portal finding). Token boundary: OAuth tokens and the confidential-client secret live in the journal — the one trusted store (owner ruling, 2026-07-07; no @@ -155,6 +163,16 @@ ENDPOINT_RECORD_TYPES: Final[Mapping[str, tuple[str, ...]]] = { "heartrate": ("oura.heartrate",), "daily_cardiovascular_age": ("oura.daily_cardiovascular_age",), "blood_glucose": ("oura.blood_glucose",), + # 2026-07-07 granted-scope expansion, shapes verified against + # openapi-1.35 AND live probes (workout/session/enhanced_tag returned + # real rows; vO2_max is documented but empty for this account). The + # ``vO2_max`` key doubles as the route segment — that casing is + # exact (lowercase ``vo2_max`` 404s live); the record type uses the + # clean lowercase spelling. + "workout": ("oura.workout",), + "session": ("oura.session",), + "enhanced_tag": ("oura.enhanced_tag",), + "vO2_max": ("oura.vo2_max",), } # Endpoints the sync engine polls, in a fixed fetch order. @@ -168,14 +186,37 @@ SYNC_ENDPOINTS: Final[tuple[str, ...]] = ( "daily_activity", "heartrate", "daily_cardiovascular_age", - "blood_glucose", + "workout", + "session", + "enhanced_tag", + "vO2_max", ) +# Endpoints whose scope Oura grants only to partner integrations. The +# owner's developer portal (checked 2026-07) shows every grantable scope +# already enabled and NO ``metabolic`` option — blood_glucose is +# partner-gated (Tidepool-class integrations only), so polling it 401s +# on every run forever. It stays out of SYNC_ENDPOINTS so hourly runs +# stop reporting the unauthorized error each cycle, while the full +# normalization/fetch machinery (parse, dedupe, fixtures, tests) stays +# wired for a future partner grant or file import. Re-enabling is one +# line: move the name back into SYNC_ENDPOINTS. The cursor never carries +# a partner-gated endpoint (and so never marks it backfill_complete), so +# a future re-enable still backfills from BACKFILL_HORIZON_DAY. +_PARTNER_GATED_ENDPOINTS: Final[tuple[str, ...]] = ("blood_glucose",) + # Series endpoints that paginate by datetime (start_datetime/end_datetime), # not by day; their rows carry no document id or day field. blood_glucose # membership is a pinned assumption (see _SERIES_REQUIRED_FIELDS). _DATETIME_PAGED_ENDPOINTS: Final = frozenset({"heartrate", "blood_glucose"}) +# Document endpoints whose day attribution field is not literally +# ``day``. enhanced_tag is the one exception (openapi-1.35 +# EnhancedTagModel, live-confirmed 2026-07-07): rows carry required +# ``start_day`` (plus nullable ``end_day``) instead — the journal day is +# Oura's ``start_day`` verbatim, matching the day-verbatim rule. +_DOCUMENT_DAY_FIELDS: Final[Mapping[str, str]] = {"enhanced_tag": "start_day"} + # Required row fields per instant-series endpoint: (timestamp field, # value field). heartrate is documented (openapi-1.35 PublicHeartRateRow: # timestamp + bpm). blood_glucose is ABSENT from the published spec @@ -184,9 +225,10 @@ _DATETIME_PAGED_ENDPOINTS: Final = frozenset({"heartrate", "blood_glucose"}) # routes); its shape here is pinned to Oura's series-row convention of a # UTC ``timestamp`` plus a domain-named value field (heartrate -> bpm, # ring_battery_level -> level, hence blood_glucose -> glucose, mg/dL per -# Oura's Stelo integration). The first post-reauthorization fetch -# confirms or falsifies this pin — a mismatch fails loudly in -# parse_endpoint_document, naming the missing fields. +# Oura's Stelo integration). blood_glucose is partner-gated (see +# _PARTNER_GATED_ENDPOINTS) so no fetch can currently falsify the pin — +# it holds for a future partner grant or file import, and a mismatch +# fails loudly in parse_endpoint_document, naming the missing fields. _SERIES_REQUIRED_FIELDS: Final[Mapping[str, tuple[str, str]]] = { "heartrate": ("timestamp", "bpm"), "blood_glucose": ("timestamp", "glucose"), @@ -255,10 +297,12 @@ def parse_endpoint_document( """Validate one API-page-shaped document and return its data items. The Oura API v2 returns ``{"data": [...], "next_token": ...}`` pages. - Document-shaped endpoints require ``id`` and ``day`` on every item; - instant-series endpoints (``_SERIES_REQUIRED_FIELDS``) instead require - a timestamp plus their value field (their rows carry no document id — - see ``_normalize_item``). + Document-shaped endpoints require ``id`` and their day field + (``day``, except ``_DOCUMENT_DAY_FIELDS`` overrides — enhanced_tag + carries ``start_day``) on every item; instant-series endpoints + (``_SERIES_REQUIRED_FIELDS``) instead require a timestamp plus their + value field (their rows carry no document id — see + ``_normalize_item``). """ if endpoint not in ENDPOINT_RECORD_TYPES: @@ -273,6 +317,7 @@ def parse_endpoint_document( f"got {type(data).__name__}" ) series_fields = _SERIES_REQUIRED_FIELDS.get(endpoint) + day_field = _DOCUMENT_DAY_FIELDS.get(endpoint, "day") items: list[dict[str, Any]] = [] for index, item in enumerate(data): if not isinstance(item, dict): @@ -287,9 +332,9 @@ def parse_endpoint_document( f"Oura {endpoint} data[{index}] is missing " f"{timestamp_field!r} or {value_field!r}" ) - elif not item.get("id") or not item.get("day"): + elif not item.get("id") or not item.get(day_field): raise OuraDocumentError( - f"Oura {endpoint} data[{index}] is missing 'id' or 'day'" + f"Oura {endpoint} data[{index}] is missing 'id' or {day_field!r}" ) items.append(item) return items @@ -444,7 +489,7 @@ def _normalize_item( raw_ref: str, owner_timezone: dt.tzinfo, ) -> list[OuraNormalizedItem]: - day = parse_oura_day(item.get("day")) or "" + day = parse_oura_day(item.get(_DOCUMENT_DAY_FIELDS.get(endpoint, "day"))) or "" rows: list[OuraNormalizedItem] = [] if endpoint == "daily_sleep": rows.append( @@ -691,6 +736,119 @@ def _normalize_item( raw_ref=raw_ref, ) ) + elif endpoint == "workout": + # PublicWorkout (openapi-1.35; live-confirmed 2026-07-07): + # required id/activity/day/start_datetime/end_datetime/intensity/ + # source; calories (kcal), distance (m), and label nullable. + # Datetimes are LocalizedDateTime — wearer-local offsets (live + # rows carry -04:00..-07:00, never UTC-Z) — so they pass through + # verbatim like sleep periods, and the journal day is Oura's + # ``day`` verbatim (a 23:12 workout stays on its local day even + # though its UTC instant crosses midnight). An event row, like + # Apple Health workouts: no scalar value — calories/distance are + # metadata facts and duration derives from the interval at render + # time. Same-ring-two-pipes precedence against the AH mirror's + # HKWorkoutActivityType* rows is presentation-side (O-5C). + rows.append( + _build_item( + record_type="oura.workout", + kind="workout", + source_record_id=str(item["id"]), + day=day, + start_time=str(item.get("start_datetime") or item["day"]), + end_time=( + str(item["end_datetime"]) if item.get("end_datetime") else None + ), + value=None, + unit=None, + metadata=_pick( + item, + ( + "activity", + "intensity", + "source", + "label", + "calories", + "distance", + ), + ), + import_id=import_id, + raw_ref=raw_ref, + ) + ) + elif endpoint == "session": + # PublicSession (openapi-1.35; live-confirmed): required id/day/ + # start_datetime/end_datetime/type (breathing|meditation|nap| + # relaxation|rest|body_status); mood and the heart_rate/ + # heart_rate_variability/motion_count sample blocks nullable. + # Datetimes are wearer-local offsets, verbatim. The sample blocks + # ({interval, items[], timestamp}) stay in the raw page — reached + # through raw_ref — never in normalized metadata: this is an + # event row, not a series carrier. + rows.append( + _build_item( + record_type="oura.session", + kind="session", + source_record_id=str(item["id"]), + day=day, + start_time=str(item.get("start_datetime") or item["day"]), + end_time=( + str(item["end_datetime"]) if item.get("end_datetime") else None + ), + value=None, + unit=None, + metadata=_pick(item, ("type", "mood")), + import_id=import_id, + raw_ref=raw_ref, + ) + ) + elif endpoint == "enhanced_tag": + # EnhancedTagModel (openapi-1.35; live-confirmed): required id/ + # start_time/start_day — the one document endpoint with no + # ``day`` field (see _DOCUMENT_DAY_FIELDS). The journal day is + # Oura's ``start_day`` verbatim; start/end times are wearer-local + # offsets, verbatim. tag_type_code/comment/custom_name are the + # owner's own note content — metadata facts, never a value. + rows.append( + _build_item( + record_type="oura.enhanced_tag", + kind="tag", + source_record_id=str(item["id"]), + day=day, + start_time=str(item.get("start_time") or item["start_day"]), + end_time=str(item["end_time"]) if item.get("end_time") else None, + value=None, + unit=None, + metadata=_pick( + item, + ("tag_type_code", "comment", "custom_name", "end_day"), + ), + import_id=import_id, + raw_ref=raw_ref, + ) + ) + elif endpoint == "vO2_max": + # PublicVO2Max (openapi-1.35): required id/day/timestamp/vo2_max + # (integer). Zero rows on this account today, so the shape is + # documented-only until data appears; the route casing is exactly + # ``vO2_max`` (lowercase 404s live — verified 2026-07-07). Day is + # Oura's verbatim; ``timestamp`` is a wearer-local offset + # instant, verbatim. VO2 max is mL/kg/min by definition (the spec + # carries no unit field). + rows.append( + _build_item( + record_type="oura.vo2_max", + kind="daily_summary", + source_record_id=str(item["id"]), + day=day, + start_time=str(item.get("timestamp") or item["day"]), + value=item.get("vo2_max"), + unit="mL/kg/min", + metadata={}, + import_id=import_id, + raw_ref=raw_ref, + ) + ) else: # pragma: no cover - parse_endpoint_document rejects these first raise OuraDocumentError(f"Unsupported Oura endpoint {endpoint!r}") return rows @@ -814,6 +972,8 @@ def _fact_line(row: Mapping[str, Any]) -> str | None: return f"Activity score {value} · Oura's score" if record_type == "oura.daily_cardiovascular_age": return f"Cardiovascular age {value} · Oura's estimate" + if record_type == "oura.vo2_max": + return f"VO2 max {value} · Oura's estimate" if record_type == "oura.temperature_deviation": return f"Temperature deviation {value:+.2f} °C · Oura's measurement" if record_type == "oura.sleep": @@ -825,7 +985,9 @@ def _fact_line(row: Mapping[str, Any]) -> str | None: return line # Heartrate and blood-glucose samples are series, not day facts — # never summarized into prose here (no derived aggregates presented - # as ours). + # as ours). Workouts, sessions, and tags are value-less event rows — + # they exit at the value-is-None check above; day surfaces render + # them from their kind, never from prose here. return None @@ -1712,7 +1874,7 @@ def _item_day_iso(endpoint: str, item: Mapping[str, Any]) -> str | None: if endpoint in _DATETIME_PAGED_ENDPOINTS: raw = str(item.get("timestamp") or "")[:10] else: - raw = str(item.get("day") or "") + raw = str(item.get(_DOCUMENT_DAY_FIELDS.get(endpoint, "day")) or "") return raw if parse_oura_day(raw) else None diff --git a/tests/fixtures/importers/health/oura_synthetic/README.md b/tests/fixtures/importers/health/oura_synthetic/README.md index 2976077b1..31f08e246 100644 --- a/tests/fixtures/importers/health/oura_synthetic/README.md +++ b/tests/fixtures/importers/health/oura_synthetic/README.md @@ -16,9 +16,33 @@ page whose rows carry `timestamp`/`bpm`/`source` and no document `id` or plus the 2026-07-07 additions: `daily_cardiovascular_age` (document-shaped, per openapi-1.35) and `blood_glucose` (a time-series page whose PINNED-ASSUMPTION rows carry `timestamp`/`glucose` in UTC — -the endpoint is absent from the published spec; if the first -post-reauthorization fetch shows a different shape, fix this fixture and -its tests together). +the endpoint is absent from the published spec **and is partner-gated**: +Oura's developer portal exposes no `metabolic` scope to standard apps, +so `blood_glucose` sits in `_PARTNER_GATED_ENDPOINTS` and the sync +engine never polls it; the fixture keeps the normalization machinery +tested for a future partner grant or file import). + +The 2026-07-07 granted-scope expansion adds four more endpoint files, +all shapes verified against openapi-1.35 **and** live probes (workout / +session / enhanced_tag returned real rows; vO2_max returned an empty +page, so its rows here follow the documented `PublicVO2Max` shape): + +- `workout.json` — document-shaped `{id, activity, day, start_datetime, + end_datetime, intensity, source}` + nullable `calories` (kcal), + `distance` (m), `label`. Datetimes are wearer-local offsets (never + UTC-Z); the second row starts at 23:12 local so its UTC instant + crosses midnight — the journal day must stay Oura's `day` verbatim. +- `session.json` — `{id, day, start_datetime, end_datetime, type}` + + nullable `mood` and `heart_rate`/`heart_rate_variability`/ + `motion_count` sample blocks (`{interval, items, timestamp}`). The + sample blocks stay in raw pages only — normalized metadata carries + just `type`/`mood`. +- `enhanced_tag.json` — the ONLY document endpoint with no `day` field: + `{id, start_time, start_day}` required + nullable `tag_type_code`, + `end_time`, `end_day`, `comment`, `custom_name`. Journal day is + `start_day` verbatim; the second row spans into the next day. +- `vO2_max.json` — `{id, day, timestamp, vo2_max}` (route casing is + exactly `vO2_max`; lowercase 404s live). ## revisions/ diff --git a/tests/fixtures/importers/health/oura_synthetic/enhanced_tag.json b/tests/fixtures/importers/health/oura_synthetic/enhanced_tag.json new file mode 100644 index 000000000..19ea5ae9b --- /dev/null +++ b/tests/fixtures/importers/health/oura_synthetic/enhanced_tag.json @@ -0,0 +1,25 @@ +{ + "data": [ + { + "id": "synthetic-tag-2026-01-02", + "tag_type_code": "tag_generic_nocaffeine", + "start_time": "2026-01-02T14:45:12-07:00", + "end_time": null, + "start_day": "2026-01-02", + "end_day": null, + "comment": null, + "custom_name": null + }, + { + "id": "synthetic-tag-2026-01-03", + "tag_type_code": null, + "start_time": "2026-01-03T22:20:00-07:00", + "end_time": "2026-01-04T07:10:00-07:00", + "start_day": "2026-01-03", + "end_day": "2026-01-04", + "comment": "synthetic note text", + "custom_name": "synthetic custom tag" + } + ], + "next_token": null +} diff --git a/tests/fixtures/importers/health/oura_synthetic/session.json b/tests/fixtures/importers/health/oura_synthetic/session.json new file mode 100644 index 000000000..b92f91a59 --- /dev/null +++ b/tests/fixtures/importers/health/oura_synthetic/session.json @@ -0,0 +1,35 @@ +{ + "data": [ + { + "id": "synthetic-session-2026-01-02", + "day": "2026-01-02", + "start_datetime": "2026-01-02T17:10:00.000-07:00", + "end_datetime": "2026-01-02T17:30:00.000-07:00", + "type": "meditation", + "mood": "good", + "heart_rate": { + "interval": 300.0, + "items": [62.0, 60.5, null, 58.0], + "timestamp": "2026-01-02T17:10:00.000-07:00" + }, + "heart_rate_variability": { + "interval": 300.0, + "items": [48.0, 52.0, 55.0, null], + "timestamp": "2026-01-02T17:10:00.000-07:00" + }, + "motion_count": null + }, + { + "id": "synthetic-session-2026-01-03", + "day": "2026-01-03", + "start_datetime": "2026-01-03T21:02:10.500-07:00", + "end_datetime": "2026-01-03T21:09:41.200-07:00", + "type": "rest", + "mood": null, + "heart_rate": null, + "heart_rate_variability": null, + "motion_count": null + } + ], + "next_token": null +} diff --git a/tests/fixtures/importers/health/oura_synthetic/vO2_max.json b/tests/fixtures/importers/health/oura_synthetic/vO2_max.json new file mode 100644 index 000000000..5b107440a --- /dev/null +++ b/tests/fixtures/importers/health/oura_synthetic/vO2_max.json @@ -0,0 +1,17 @@ +{ + "data": [ + { + "id": "synthetic-vo2max-2026-01-02", + "day": "2026-01-02", + "timestamp": "2026-01-02T00:00:00-07:00", + "vo2_max": 41 + }, + { + "id": "synthetic-vo2max-2026-01-03", + "day": "2026-01-03", + "timestamp": "2026-01-03T00:00:00-07:00", + "vo2_max": 42 + } + ], + "next_token": null +} diff --git a/tests/fixtures/importers/health/oura_synthetic/workout.json b/tests/fixtures/importers/health/oura_synthetic/workout.json new file mode 100644 index 000000000..48c237e5b --- /dev/null +++ b/tests/fixtures/importers/health/oura_synthetic/workout.json @@ -0,0 +1,29 @@ +{ + "data": [ + { + "id": "synthetic-workout-2026-01-02", + "activity": "walking", + "calories": 148.5, + "day": "2026-01-02", + "distance": 2412.9, + "end_datetime": "2026-01-02T08:41:00.000-07:00", + "intensity": "moderate", + "label": null, + "source": "confirmed", + "start_datetime": "2026-01-02T08:05:00.000-07:00" + }, + { + "id": "synthetic-workout-2026-01-03", + "activity": "cycling", + "calories": 321.25, + "day": "2026-01-03", + "distance": 9120.4, + "end_datetime": "2026-01-03T23:58:00.000-07:00", + "intensity": "hard", + "label": "synthetic night ride", + "source": "manual", + "start_datetime": "2026-01-03T23:12:00.000-07:00" + } + ], + "next_token": null +} diff --git a/tests/test_oura_importer.py b/tests/test_oura_importer.py index c2b2b2bcc..2fd3e2def 100644 --- a/tests/test_oura_importer.py +++ b/tests/test_oura_importer.py @@ -59,9 +59,12 @@ APPLE_FIXTURE_ROOT = ( / "apple_health_synthetic" ) -# Fixture bundle shape: 21 documents across 10 endpoints; each readiness -# document also splits out a temperature-deviation row -> 23 rows. -_FIXTURE_ROW_COUNT = 23 +# Fixture bundle shape: 29 documents across 14 endpoints; each readiness +# document also splits out a temperature-deviation row -> 31 rows. +_FIXTURE_ROW_COUNT = 31 +# Sync runs fetch SYNC_ENDPOINTS only: the partner-gated blood_glucose +# fixture (4 rows) is parse/normalize-only, never polled. +_SYNC_ROW_COUNT = _FIXTURE_ROW_COUNT - 4 def _use_journal(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: @@ -288,6 +291,10 @@ def test_parse_bundle_reads_all_supported_endpoint_files(): "heartrate", "daily_cardiovascular_age", "blood_glucose", + "workout", + "session", + "enhanced_tag", + "vO2_max", } assert len(bundle["daily_sleep"]) == 2 assert len(bundle["sleep"]) == 2 @@ -296,6 +303,10 @@ def test_parse_bundle_reads_all_supported_endpoint_files(): assert len(bundle["heartrate"]) == 4 assert len(bundle["daily_cardiovascular_age"]) == 2 assert len(bundle["blood_glucose"]) == 4 + assert len(bundle["workout"]) == 2 + assert len(bundle["session"]) == 2 + assert len(bundle["enhanced_tag"]) == 2 + assert len(bundle["vO2_max"]) == 2 def test_parse_single_endpoint_file(): @@ -347,6 +358,28 @@ def test_parse_blood_glucose_requires_timestamp_and_glucose(): ) +def test_parse_enhanced_tag_requires_id_and_start_day(): + # enhanced_tag is the one document endpoint with no `day` field + # (openapi-1.35 EnhancedTagModel): its day attribution field is + # `start_day`, enforced through _DOCUMENT_DAY_FIELDS. + with pytest.raises(oura.OuraDocumentError, match="missing 'id' or 'start_day'"): + oura.parse_endpoint_document( + "enhanced_tag", + {"data": [{"id": "tag-1", "start_time": "2026-01-02T14:45:12-07:00"}]}, + ) + with pytest.raises(oura.OuraDocumentError, match="missing 'id' or 'start_day'"): + oura.parse_endpoint_document( + "enhanced_tag", {"data": [{"start_day": "2026-01-02"}]} + ) + # The other granted-scope endpoints validate on the plain id+day rule. + with pytest.raises(oura.OuraDocumentError, match="missing 'id' or 'day'"): + oura.parse_endpoint_document("workout", {"data": [{"id": "workout-1"}]}) + with pytest.raises(oura.OuraDocumentError, match="missing 'id' or 'day'"): + oura.parse_endpoint_document( + "vO2_max", {"data": [{"id": "vo2-1", "vo2_max": 41}]} + ) + + def test_parse_oura_day_normalizes_and_rejects(): assert oura.parse_oura_day("2026-01-02") == "20260102" assert oura.parse_oura_day("2026-13-02") is None @@ -399,6 +432,10 @@ def test_normalize_bundle_emits_expected_record_types(): "oura.heartrate", "oura.daily_cardiovascular_age", "oura.blood_glucose", + "oura.workout", + "oura.session", + "oura.enhanced_tag", + "oura.vo2_max", } # Temperature deviation splits out of each readiness document. assert len(by_type["oura.temperature_deviation"]) == 2 @@ -555,6 +592,129 @@ def test_normalize_blood_glucose_converts_utc_and_synthesizes_identity(): assert cross_midnight["start_date"] == "2026-01-02T21:20:00-07:00" +def test_normalize_workout_is_event_row_with_verbatim_local_times(): + # PublicWorkout (openapi-1.35; live-confirmed 2026-07-07). Workouts + # are event rows like Apple Health workouts: kind="workout", no + # scalar value, activity/intensity/calories/distance as metadata. + items = _normalized_items() + rows = [item.row for item in items if item.row["record_type"] == "oura.workout"] + + assert len(rows) == 2 + first = next(row for row in rows if row["day"] == "20260102") + assert first["kind"] == "workout" + assert "value" not in first + assert "unit" not in first + assert first["source_record_id"] == "synthetic-workout-2026-01-02" + # Datetimes are wearer-local offsets and pass through VERBATIM — + # never UTC-converted, never re-derived. + assert first["start_date"] == "2026-01-02T08:05:00.000-07:00" + assert first["end_date"] == "2026-01-02T08:41:00.000-07:00" + # Null label drops from metadata like every other null field. + assert first["metadata"] == { + "activity": "walking", + "intensity": "moderate", + "source": "confirmed", + "calories": 148.5, + "distance": 2412.9, + } + # Timezone pin: the second workout starts at 23:12 local, so its UTC + # instant is the NEXT calendar day — the journal day must stay Oura's + # `day` verbatim, with no local-time recomputation. + second = next(row for row in rows if row["start_date"].startswith("2026-01-03T23")) + assert second["day"] == "20260103" + assert second["metadata"]["label"] == "synthetic night ride" + + +def test_normalize_session_keeps_type_and_mood_never_sample_series(): + # PublicSession (openapi-1.35; live-confirmed). The heart_rate/ + # heart_rate_variability/motion_count sample blocks stay in the raw + # page only — normalized metadata carries just type and mood. + items = _normalized_items() + rows = [item.row for item in items if item.row["record_type"] == "oura.session"] + + assert len(rows) == 2 + first = next(row for row in rows if row["day"] == "20260102") + assert first["kind"] == "session" + assert "value" not in first + assert first["source_record_id"] == "synthetic-session-2026-01-02" + assert first["start_date"] == "2026-01-02T17:10:00.000-07:00" + assert first["end_date"] == "2026-01-02T17:30:00.000-07:00" + assert first["metadata"] == {"type": "meditation", "mood": "good"} + # Null mood drops; sample blocks never enter metadata. + second = next(row for row in rows if row["day"] == "20260103") + assert second["metadata"] == {"type": "rest"} + + +def test_normalize_enhanced_tag_uses_start_day_verbatim(): + # EnhancedTagModel (openapi-1.35; live-confirmed): no `day` field — + # the journal day is Oura's `start_day` verbatim, even for a tag + # whose span ends on a later day. + items = _normalized_items() + rows = [ + item.row for item in items if item.row["record_type"] == "oura.enhanced_tag" + ] + + assert len(rows) == 2 + first = next(row for row in rows if row["day"] == "20260102") + assert first["kind"] == "tag" + assert "value" not in first + assert first["source_record_id"] == "synthetic-tag-2026-01-02" + assert first["start_date"] == "2026-01-02T14:45:12-07:00" + assert "end_date" not in first + assert first["metadata"] == {"tag_type_code": "tag_generic_nocaffeine"} + # A spanning tag: attributed to its start_day, end fields kept. + second = next(row for row in rows if row["day"] == "20260103") + assert second["end_date"] == "2026-01-04T07:10:00-07:00" + assert second["metadata"] == { + "comment": "synthetic note text", + "custom_name": "synthetic custom tag", + "end_day": "2026-01-04", + } + + +def test_normalize_vo2_max_uses_documented_shape(): + # PublicVO2Max (openapi-1.35): {id, day, timestamp, vo2_max}. Zero + # rows on this account today, so the fixture follows the documented + # shape; VO2 max is mL/kg/min by definition (the spec has no unit). + items = _normalized_items() + rows = [item.row for item in items if item.row["record_type"] == "oura.vo2_max"] + + assert len(rows) == 2 + first = next(row for row in rows if row["day"] == "20260102") + assert first["kind"] == "daily_summary" + assert first["value"] == 41 + assert first["unit"] == "mL/kg/min" + assert first["source_record_id"] == "synthetic-vo2max-2026-01-02" + assert first["metadata"] == {} + + +def test_workout_dedupe_key_is_payload_independent(): + # Oura documents revise in place (same id, corrected payload) — a + # re-fetched workout with corrected calories must upsert, not + # duplicate, exactly like every other document-id-keyed endpoint. + workout = { + "id": "synthetic-workout-2026-01-02", + "activity": "walking", + "calories": 148.5, + "day": "2026-01-02", + "distance": 2412.9, + "end_datetime": "2026-01-02T08:41:00.000-07:00", + "intensity": "moderate", + "label": None, + "source": "confirmed", + "start_datetime": "2026-01-02T08:05:00.000-07:00", + } + revised = dict(workout, calories=152.0, intensity="hard") + + def key(item: dict) -> str: + normalized = oura.normalize_bundle( + {"workout": [item]}, import_id="x", raw_ref_root="x" + ) + return normalized[0].row["dedupe_key"] + + assert key(workout) == key(revised) + + def test_blood_glucose_dedupe_key_is_value_independent(): # A re-fetched sample at the same timestamp with a corrected reading # must update in place, exactly like heartrate revisions. @@ -982,7 +1142,7 @@ def test_preview_counts_documents_and_days(): assert preview.date_range == ("20260102", "20260103") assert preview.entity_count == 0 - # 15 documents; readiness docs each add a temperature-deviation row. + # 29 documents; readiness docs each add a temperature-deviation row. assert preview.item_count == _FIXTURE_ROW_COUNT assert "daily_readiness=2" in preview.summary assert "sleep=2" in preview.summary @@ -990,6 +1150,10 @@ def test_preview_counts_documents_and_days(): assert "heartrate=4" in preview.summary assert "daily_cardiovascular_age=2" in preview.summary assert "blood_glucose=4" in preview.summary + assert "workout=2" in preview.summary + assert "session=2" in preview.summary + assert "enhanced_tag=2" in preview.summary + assert "vO2_max=2" in preview.summary assert f"source_family={SOURCE_OURA_API}" in preview.summary @@ -1124,12 +1288,18 @@ def test_render_day_summary_attributes_every_score_to_oura(): assert "Temperature deviation -0.21 °C · Oura's measurement" in summary assert "Activity score 85 · Oura's score" in summary assert "Cardiovascular age 34 · Oura's estimate" in summary + assert "VO2 max 41 · Oura's estimate" in summary assert "Sleep 7h 19m · Oura's staging" in summary assert "deep 1h 31m" in summary # Heartrate and blood-glucose samples are series, never summarized # into day prose. assert "bpm" not in summary assert "glucose" not in summary.lower() + # Workouts, sessions, and tags are event rows — day surfaces render + # them from their kind, never as day-summary prose. + assert "walking" not in summary.lower() + assert "meditation" not in summary.lower() + assert "nocaffeine" not in summary.lower() assert "brought in via the Oura API · import 20260105_000000" in summary @@ -1356,6 +1526,24 @@ def test_blood_glucose_fetch_uses_datetime_window_params(): assert "start_date=" not in url +def test_granted_endpoints_fetch_day_paged_with_exact_route_casing(): + # The four granted-scope endpoints are day-paged documents. The + # vO2_max route casing is exact — lowercase vo2_max 404s live + # (verified 2026-07-07) — so the URL must carry it verbatim. + for endpoint in ("workout", "session", "enhanced_tag", "vO2_max"): + transport = ScriptedTransport() + transport.script(endpoint, _ok_page(_fixture_items(endpoint))) + client = _canned_client(transport) + + client.fetch_endpoint(endpoint, start_day="2026-01-01", end_day="2026-01-10") + + url = transport.calls[0][0] + assert f"/usercollection/{endpoint}?" in url + assert "start_date=2026-01-01" in url + assert "end_date=2026-01-10" in url + assert "start_datetime=" not in url + + # --------------------------------------------------------------------------- # Sync engine — end-to-end into a temp journal with canned transport # --------------------------------------------------------------------------- @@ -1381,7 +1569,7 @@ def test_first_sync_save_writes_bundle_dedupe_and_cursor(tmp_path: Path, monkeyp # Normalized monthly shard with every row stamped for this bundle. shard = import_dir / "normalized" / "2026-01.jsonl" rows = [json.loads(line) for line in shard.read_text().splitlines()] - assert len(rows) == _FIXTURE_ROW_COUNT + assert len(rows) == _SYNC_ROW_COUNT assert {row["source_family"] for row in rows} == {SOURCE_OURA_API} assert {row["import_id"] for row in rows} == {import_id} assert rows[0]["normalized_ref"].startswith(f"imports/{import_id}/normalized/") @@ -1395,26 +1583,34 @@ def test_first_sync_save_writes_bundle_dedupe_and_cursor(tmp_path: Path, monkeyp # Manifests match the apple_health bundle shape. manifest = json.loads((import_dir / "manifest.json").read_text()) assert manifest["source_type"] == SOURCE_OURA_API - assert manifest["entry_count"] == _FIXTURE_ROW_COUNT + assert manifest["entry_count"] == _SYNC_ROW_COUNT assert manifest["days_affected"] == ["20260102", "20260103"] content_lines = (import_dir / "content_manifest.jsonl").read_text().splitlines() assert json.loads(content_lines[0])["type"] == "health_normalized_month" # Dedupe upserts: every row inserted exactly once. - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT # Cursor advanced only after the writes, with per-endpoint watermarks. state = json.loads((journal / "imports" / "oura.json").read_text()) assert state["schema"] == oura.SYNC_STATE_SCHEMA assert state["trailing_refetch_days"] == oura.TRAILING_REFETCH_DAYS assert state["endpoints"]["daily_readiness"]["high_water_day"] == "2026-01-03" - assert state["endpoints"]["heartrate"]["high_water_day"] == "2026-01-03" # Datetime-paged watermarks come from the raw (UTC) day of the newest # sample; the >=7-day trailing refetch absorbs the local-day skew. - assert state["endpoints"]["blood_glucose"]["high_water_day"] == "2026-01-03" + assert state["endpoints"]["heartrate"]["high_water_day"] == "2026-01-03" assert ( state["endpoints"]["daily_cardiovascular_age"]["high_water_day"] == "2026-01-03" ) + # The granted-scope endpoints watermark like every other document + # endpoint (enhanced_tag from its start_day field). + assert state["endpoints"]["workout"]["high_water_day"] == "2026-01-03" + assert state["endpoints"]["session"]["high_water_day"] == "2026-01-03" + assert state["endpoints"]["enhanced_tag"]["high_water_day"] == "2026-01-03" + assert state["endpoints"]["vO2_max"]["high_water_day"] == "2026-01-03" + # Partner-gated endpoints never enter the cursor (so a future + # re-enable backfills from the horizon). + assert "blood_glucose" not in state["endpoints"] # Every fetched endpoint is marked covered so empty endpoints never # re-walk the backfill horizon on later runs. assert all( @@ -1422,15 +1618,15 @@ def test_first_sync_save_writes_bundle_dedupe_and_cursor(tmp_path: Path, monkeyp for endpoint in oura.SYNC_ENDPOINTS ) assert state["last_result"]["import_id"] == import_id - assert state["last_result"]["inserted"] == _FIXTURE_ROW_COUNT + assert state["last_result"]["inserted"] == _SYNC_ROW_COUNT # First sync fetches a trailing 30-day window — not deep history. readiness_url = next(u for u, _ in transport.calls if "daily_readiness" in u) assert "start_date=2025-12-11" in readiness_url assert "end_date=2026-01-10" in readiness_url - assert result["total"] == _FIXTURE_ROW_COUNT - assert result["downloaded"] == _FIXTURE_ROW_COUNT # inserted + assert result["total"] == _SYNC_ROW_COUNT + assert result["downloaded"] == _SYNC_ROW_COUNT # inserted assert result["imported"] == 0 # updated assert result["source_label"] == "Oura (API)" assert "cron_hint" not in result @@ -1440,15 +1636,11 @@ def test_window_days_overrides_first_sync_window(tmp_path: Path, monkeypatch): journal = _use_journal(tmp_path, monkeypatch) transport = _fixture_transport() # A 91-inclusive-day window chunks the datetime-paged series into - # three <=31-day requests each (Oura 400s over-wide heartrate ranges — - # found live during the 2026-07-06 full-history backfill; - # blood_glucose is capped identically by pinned assumption). + # three <=31-day requests (Oura 400s over-wide heartrate ranges — + # found live during the 2026-07-06 full-history backfill). transport.script( "heartrate", _fixture_page("heartrate"), _fixture_page("heartrate") ) - transport.script( - "blood_glucose", _fixture_page("blood_glucose"), _fixture_page("blood_glucose") - ) client = _canned_client(transport) oura.backend.sync( @@ -1467,8 +1659,8 @@ def test_window_days_overrides_first_sync_window(tmp_path: Path, monkeypatch): assert "start_datetime=2025-11-12" in heartrate_urls[1] assert "start_datetime=2025-12-13" in heartrate_urls[2] assert "end_datetime=2026-01-10" in heartrate_urls[2] - glucose_urls = [u for u, _ in transport.calls if "blood_glucose" in u] - assert len(glucose_urls) == 3 + # Partner-gated: even an explicit window never polls blood_glucose. + assert not [u for u, _ in transport.calls if "blood_glucose" in u] def test_window_chunks_split_per_endpoint_limits(): @@ -1525,7 +1717,7 @@ def test_second_sync_refetches_trailing_window_and_upserts_revisions( assert "start_date=2025-12-29" in readiness_url # Revisions update in place: no new dedupe rows, refreshed value_hash. - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT assert second["downloaded"] == 0 # nothing inserted assert second["imported"] == 4 # readiness + temperature rows updated revised = _items_by_row_identity( @@ -1562,8 +1754,8 @@ def test_catalog_sync_writes_nothing_and_needs_no_approval(tmp_path: Path, monke ) assert result["dry_run"] is True - assert result["total"] == _FIXTURE_ROW_COUNT - assert result["available"] == _FIXTURE_ROW_COUNT + assert result["total"] == _SYNC_ROW_COUNT + assert result["available"] == _SYNC_ROW_COUNT # Nothing written: no bundle, no dedupe DB, no cursor. assert not (journal / "imports").exists() @@ -1690,7 +1882,7 @@ def test_scheduled_sync_with_consent_passes_and_emits_cron_hint( today=dt.date(2026, 1, 10), ) - assert result["downloaded"] == _FIXTURE_ROW_COUNT + assert result["downloaded"] == _SYNC_ROW_COUNT assert result["cron_hint"].startswith("0 */6 * * * ") assert result["cron_hint"].endswith("importer --sync oura --save --scheduled") @@ -1809,13 +2001,13 @@ def test_quiet_second_sync_writes_nothing_and_advances_cursor( ) # CLI-facing keys: total says what was fetched, nothing importable. - assert second["total"] == _FIXTURE_ROW_COUNT + assert second["total"] == _SYNC_ROW_COUNT assert second["available"] == 0 assert second["imported"] == 0 assert second["downloaded"] == 0 assert second["months"] == [] assert second["summary"] == ( - f"Oura (API) sync quiet run: nothing new (rows={_FIXTURE_ROW_COUNT} all known)" + f"Oura (API) sync quiet run: nothing new (rows={_SYNC_ROW_COUNT} all known)" ) # The cursor advanced exactly as a full save would have. @@ -1824,10 +2016,10 @@ def test_quiet_second_sync_writes_nothing_and_advances_cursor( assert state["endpoints"]["daily_readiness"]["high_water_day"] == "2026-01-03" assert state["last_result"]["quiet_run"] is True assert state["last_result"]["import_id"] is None - assert state["last_result"]["rows"] == _FIXTURE_ROW_COUNT + assert state["last_result"]["rows"] == _SYNC_ROW_COUNT # The dedupe ledger is untouched (reads only). - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT def test_revision_within_refetch_window_triggers_full_bundle_not_quiet( @@ -1914,10 +2106,10 @@ def test_new_data_run_writes_full_bundle_with_all_rows(tmp_path: Path, monkeypat assert second["quiet_run"] is False shard = journal / "imports" / second["import_id"] / "normalized" / "2026-01.jsonl" rows = [json.loads(line) for line in shard.read_text().splitlines()] - assert len(rows) == _FIXTURE_ROW_COUNT + 1 + assert len(rows) == _SYNC_ROW_COUNT + 1 assert second["downloaded"] == 1 # the new document - assert second["imported"] == _FIXTURE_ROW_COUNT # known rows upserted - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT + 1 + assert second["imported"] == _SYNC_ROW_COUNT # known rows upserted + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT + 1 state = json.loads((journal / "imports" / "oura.json").read_text()) assert state["endpoints"]["daily_sleep"]["high_water_day"] == "2026-01-04" @@ -1996,15 +2188,12 @@ def test_quiet_backfill_with_window_days_writes_nothing(tmp_path: Path, monkeypa listing_before = _imports_contents(journal) # An explicit 90-day backfill window re-fetches only known rows (the - # datetime-paged series chunk into three requests each). Quiet + # datetime-paged heartrate series chunks into three requests). Quiet # backfills are rare but must also write nothing. transport = _fixture_transport() transport.script( "heartrate", _fixture_page("heartrate"), _fixture_page("heartrate") ) - transport.script( - "blood_glucose", _fixture_page("blood_glucose"), _fixture_page("blood_glucose") - ) second = oura.backend.sync( journal, dry_run=False, @@ -2018,12 +2207,12 @@ def test_quiet_backfill_with_window_days_writes_nothing(tmp_path: Path, monkeypa readiness_url = next(u for u, _ in transport.calls if "daily_readiness" in u) assert "start_date=2025-10-12" in readiness_url assert len([u for u, _ in transport.calls if "heartrate" in u]) == 3 - assert len([u for u, _ in transport.calls if "blood_glucose" in u]) == 3 + assert not [u for u, _ in transport.calls if "blood_glucose" in u] assert second["quiet_run"] is True - assert second["total"] > _FIXTURE_ROW_COUNT # chunk overlap re-reads + assert second["total"] > _SYNC_ROW_COUNT # chunk overlap re-reads assert _imports_contents(journal) == listing_before - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT # --------------------------------------------------------------------------- @@ -2033,8 +2222,8 @@ def test_quiet_backfill_with_window_days_writes_nothing(tmp_path: Path, monkeypa # --------------------------------------------------------------------------- # The endpoint set a pre-upgrade cursor knows about (live cursors written -# before daily_cardiovascular_age and blood_glucose joined SYNC_ENDPOINTS, -# and before backfill_complete existed). +# before daily_cardiovascular_age joined SYNC_ENDPOINTS, and before +# backfill_complete existed). _PRE_UPGRADE_ENDPOINTS = ( "daily_readiness", "daily_sleep", @@ -2046,6 +2235,15 @@ _PRE_UPGRADE_ENDPOINTS = ( "heartrate", ) +# The endpoints that joined SYNC_ENDPOINTS after those cursors were +# written (daily_cardiovascular_age, then the 2026-07-07 granted-scope +# four). Computed from the registry so the pin tracks future additions. +_POST_UPGRADE_ENDPOINTS = tuple( + endpoint + for endpoint in oura.SYNC_ENDPOINTS + if endpoint not in _PRE_UPGRADE_ENDPOINTS +) + def _write_pre_upgrade_cursor(journal: Path, high_water: str = "2026-01-03") -> None: state = { @@ -2098,8 +2296,10 @@ def test_cursor_upgrade_fetches_new_endpoints_from_backfill_horizon( transport = ScriptedTransport() for endpoint in _PRE_UPGRADE_ENDPOINTS: transport.script(endpoint, _fixture_page(endpoint)) - glucose_chunks = _script_horizon_pages(transport, "blood_glucose", today) - cardio_chunks = _script_horizon_pages(transport, "daily_cardiovascular_age", today) + chunk_counts = { + endpoint: _script_horizon_pages(transport, endpoint, today) + for endpoint in _POST_UPGRADE_ENDPOINTS + } result = oura.backend.sync( journal, dry_run=True, client=_canned_client(transport), today=today @@ -2111,14 +2311,14 @@ def test_cursor_upgrade_fetches_new_endpoints_from_backfill_horizon( # … the endpoints the cursor has never seen walk their FULL history # from the backfill horizon, chunked per endpoint limits — not just # the trailing window. - glucose_urls = [u for u, _ in transport.calls if "blood_glucose" in u] - assert len(glucose_urls) == glucose_chunks > 100 # 31-day chunks since 2015 - assert "start_datetime=2015-01-01" in glucose_urls[0] - cardio_urls = [u for u, _ in transport.calls if "daily_cardiovascular_age" in u] - assert len(cardio_urls) == cardio_chunks - assert "start_date=2015-01-01" in cardio_urls[0] - assert result["windows"]["blood_glucose"][0] == "2015-01-01" - assert result["windows"]["daily_cardiovascular_age"][0] == "2015-01-01" + for endpoint in _POST_UPGRADE_ENDPOINTS: + urls = [u for u, _ in transport.calls if f"/{endpoint}?" in u] + # 364-day chunks since 2015 — a dozen requests per endpoint. + assert len(urls) == chunk_counts[endpoint] > 10 + assert "start_date=2015-01-01" in urls[0] + assert result["windows"][endpoint][0] == "2015-01-01" + # The partner-gated endpoint is never fetched, upgrade or not. + assert not [u for u, _ in transport.calls if "blood_glucose" in u] # Catalog runs never advance the cursor, so the pre-upgrade cursor is # byte-identical afterwards. assert json.loads((journal / "imports" / "oura.json").read_text())[ @@ -2137,10 +2337,8 @@ def test_cursor_upgrade_save_adopts_new_endpoints_and_marks_backfill( transport = ScriptedTransport() for endpoint in _PRE_UPGRADE_ENDPOINTS: transport.script(endpoint, _fixture_page(endpoint)) - _script_horizon_pages(transport, "blood_glucose", today, data_on_last_chunk=True) - _script_horizon_pages( - transport, "daily_cardiovascular_age", today, data_on_last_chunk=True - ) + for endpoint in _POST_UPGRADE_ENDPOINTS: + _script_horizon_pages(transport, endpoint, today, data_on_last_chunk=True) result = oura.backend.sync( journal, @@ -2150,15 +2348,17 @@ def test_cursor_upgrade_save_adopts_new_endpoints_and_marks_backfill( today=today, ) - assert result["downloaded"] == _FIXTURE_ROW_COUNT # fresh journal: all new + assert result["downloaded"] == _SYNC_ROW_COUNT # fresh journal: all new state = json.loads((journal / "imports" / "oura.json").read_text()) - # The upgraded cursor now carries every endpoint: watermarks preserved - # or established, and the new endpoints marked backfilled. + # The upgraded cursor now carries every polled endpoint: watermarks + # preserved or established, the new endpoints marked backfilled, and + # the partner-gated endpoint still absent. assert set(state["endpoints"]) == set(oura.SYNC_ENDPOINTS) assert state["endpoints"]["daily_readiness"]["high_water_day"] == "2026-01-03" - assert state["endpoints"]["blood_glucose"]["high_water_day"] == "2026-01-03" - assert state["endpoints"]["blood_glucose"]["backfill_complete"] is True - assert state["endpoints"]["daily_cardiovascular_age"]["backfill_complete"] is True + for endpoint in _POST_UPGRADE_ENDPOINTS: + assert state["endpoints"][endpoint]["high_water_day"] == "2026-01-03" + assert state["endpoints"][endpoint]["backfill_complete"] is True + assert "blood_glucose" not in state["endpoints"] def test_empty_backfilled_endpoint_polls_trailing_window_not_horizon( @@ -2170,12 +2370,12 @@ def test_empty_backfilled_endpoint_polls_trailing_window_not_horizon( today = dt.date(2026, 1, 10) # First post-upgrade save: the new endpoints' full-horizon walks come - # back EMPTY (no CGM on the account, say). + # back EMPTY (no VO2 max estimates on the account, say). transport = ScriptedTransport() for endpoint in _PRE_UPGRADE_ENDPOINTS: transport.script(endpoint, _fixture_page(endpoint)) - _script_horizon_pages(transport, "blood_glucose", today) - _script_horizon_pages(transport, "daily_cardiovascular_age", today) + for endpoint in _POST_UPGRADE_ENDPOINTS: + _script_horizon_pages(transport, endpoint, today) oura.backend.sync( journal, dry_run=False, @@ -2185,11 +2385,11 @@ def test_empty_backfilled_endpoint_polls_trailing_window_not_horizon( ) state = json.loads((journal / "imports" / "oura.json").read_text()) - assert state["endpoints"]["blood_glucose"]["high_water_day"] is None - assert state["endpoints"]["blood_glucose"]["backfill_complete"] is True + assert state["endpoints"]["vO2_max"]["high_water_day"] is None + assert state["endpoints"]["vO2_max"]["backfill_complete"] is True # Next run: the empty-but-backfilled endpoint polls a modest trailing - # window — one request, never the ~130-chunk horizon walk again. + # window — one request, never the dozen-chunk horizon walk again. second_transport = ScriptedTransport() for endpoint in oura.SYNC_ENDPOINTS: second_transport.script(endpoint, _ok_page([])) @@ -2200,10 +2400,118 @@ def test_empty_backfilled_endpoint_polls_trailing_window_not_horizon( today=dt.date(2026, 1, 11), ) - glucose_urls = [u for u, _ in second_transport.calls if "blood_glucose" in u] - assert len(glucose_urls) == 1 - assert "start_datetime=2015-01-01" not in glucose_urls[0] - assert "start_datetime=2025-12-12" in glucose_urls[0] # today - 30d + vo2_urls = [u for u, _ in second_transport.calls if "vO2_max" in u] + assert len(vo2_urls) == 1 + assert "start_date=2015-01-01" not in vo2_urls[0] + assert "start_date=2025-12-12" in vo2_urls[0] # today - 30d + + +# --------------------------------------------------------------------------- +# Partner-gated endpoints — blood_glucose stays fully wired for parse/ +# normalize/dedupe but is never polled: Oura's developer portal (2026-07) +# exposes no `metabolic` scope to standard apps, so every fetch would 401 +# forever and the hourly lane would report the gap each cycle. +# --------------------------------------------------------------------------- + + +def test_partner_gate_registry_membership(): + assert set(oura._PARTNER_GATED_ENDPOINTS) == {"blood_glucose"} + # Gated endpoints are never polled but keep their machinery. + assert set(oura._PARTNER_GATED_ENDPOINTS).isdisjoint(oura.SYNC_ENDPOINTS) + assert set(oura._PARTNER_GATED_ENDPOINTS) <= set(oura.ENDPOINT_RECORD_TYPES) + # The 2026-07-07 granted-scope endpoints ARE polled. + for endpoint in ("workout", "session", "enhanced_tag", "vO2_max"): + assert endpoint in oura.SYNC_ENDPOINTS + + +def test_sync_never_polls_partner_gated_blood_glucose(tmp_path: Path, monkeypatch): + # The directive behind the demotion: a full-registry sync must run + # clean — zero blood_glucose requests, zero errors — instead of + # reporting the unauthorized endpoint every scheduled cycle. + journal = _use_journal(tmp_path, monkeypatch) + _write_sync_artifact(journal, _sync_artifact(journal)) + + transport = _fixture_transport() + result = oura.backend.sync( + journal, + dry_run=False, + confirm_health_save=True, + client=_canned_client(transport), + today=dt.date(2026, 1, 10), + ) + + assert not [u for u, _ in transport.calls if "blood_glucose" in u] + assert result["errors"] == [] + assert "blood_glucose" not in result["endpoints"] + assert "blood_glucose" not in result["windows"] + # Never backfill_complete: a future re-enable (one line — move the + # name back into SYNC_ENDPOINTS) still walks the full horizon. + state = json.loads((journal / "imports" / "oura.json").read_text()) + assert "blood_glucose" not in state["endpoints"] + + +def test_cursor_upgrade_drops_stale_partner_gated_entry(tmp_path: Path, monkeypatch): + # The live 2026-07 cursor generation carries a blood_glucose entry + # (never backfilled — every poll 401d on the missing scope). After + # the demotion, the next save rewrites the cursor without it, never + # fetches it, and backfills the granted-scope four from the horizon. + journal = _use_journal(tmp_path, monkeypatch) + _write_sync_artifact(journal, _sync_artifact(journal)) + today = dt.date(2026, 1, 10) + ten_era_endpoints = (*_PRE_UPGRADE_ENDPOINTS, "daily_cardiovascular_age") + state = { + "schema": oura.SYNC_STATE_SCHEMA, + "last_sync": "2026-01-04T00:00:00+00:00", + "trailing_refetch_days": oura.TRAILING_REFETCH_DAYS, + "endpoints": { + **{ + endpoint: { + "high_water_day": "2026-01-03", + "backfill_complete": True, + "next_token": None, + } + for endpoint in ten_era_endpoints + }, + "blood_glucose": { + "high_water_day": None, + "backfill_complete": False, + "next_token": None, + }, + }, + "last_result": { + "import_id": "20260104_000000", + "quiet_run": False, + "rows": 19, + "inserted": 19, + "updated": 0, + "pages": 10, + }, + } + cursor = journal / "imports" / "oura.json" + cursor.parent.mkdir(parents=True, exist_ok=True) + cursor.write_text(json.dumps(state), encoding="utf-8") + + transport = ScriptedTransport() + for endpoint in ten_era_endpoints: + transport.script(endpoint, _fixture_page(endpoint)) + for endpoint in ("workout", "session", "enhanced_tag", "vO2_max"): + _script_horizon_pages(transport, endpoint, today, data_on_last_chunk=True) + + result = oura.backend.sync( + journal, + dry_run=False, + confirm_health_save=True, + client=_canned_client(transport), + today=today, + ) + + assert not [u for u, _ in transport.calls if "blood_glucose" in u] + assert result["errors"] == [] + new_state = json.loads((journal / "imports" / "oura.json").read_text()) + assert "blood_glucose" not in new_state["endpoints"] + for endpoint in ("workout", "session", "enhanced_tag", "vO2_max"): + assert new_state["endpoints"][endpoint]["backfill_complete"] is True + assert new_state["endpoints"][endpoint]["high_water_day"] == "2026-01-03" # --------------------------------------------------------------------------- @@ -2226,10 +2534,10 @@ def test_sync_skips_endpoint_missing_scope_and_keeps_run_alive( transport = ScriptedTransport() for endpoint in oura.SYNC_ENDPOINTS: - if endpoint == "blood_glucose": - # Pre-reauthorization reality: the token lacks the metabolic - # scope, so the endpoint 401s, once before the refresh and - # once after it. + if endpoint == "workout": + # A scope gap (say, a token minted before the workout scope + # existed on this grant): the endpoint 401s, once before the + # refresh and once after it. transport.script(endpoint, _status(401), _status(401)) else: transport.script(endpoint, _fixture_page(endpoint)) @@ -2244,31 +2552,29 @@ def test_sync_skips_endpoint_missing_scope_and_keeps_run_alive( # One refresh attempt total, then the endpoint-scoped skip. assert refresh_calls == ["synthetic-client-id"] - # Every other endpoint landed: full bundle minus the 4 glucose rows. - assert result["downloaded"] == _FIXTURE_ROW_COUNT - 4 - assert _dedupe_row_count(journal) == _FIXTURE_ROW_COUNT - 4 - assert result["endpoints"]["blood_glucose"] == 0 + # Every other endpoint landed: full sync bundle minus 2 workout rows. + assert result["downloaded"] == _SYNC_ROW_COUNT - 2 + assert _dedupe_row_count(journal) == _SYNC_ROW_COUNT - 2 + assert result["endpoints"]["workout"] == 0 # The gap is reported factually, with the reauthorization command. assert len(result["errors"]) == 1 - assert "blood_glucose" in result["errors"][0] + assert "workout" in result["errors"][0] assert "journal importer --connect oura" in result["errors"][0] # The skipped endpoint is not marked backfilled, so the first sync # after reauthorization walks it from the horizon. state = json.loads((journal / "imports" / "oura.json").read_text()) - assert state["endpoints"]["blood_glucose"]["backfill_complete"] is False - assert state["endpoints"]["blood_glucose"]["high_water_day"] is None + assert state["endpoints"]["workout"]["backfill_complete"] is False + assert state["endpoints"]["workout"]["high_water_day"] is None assert state["endpoints"]["daily_readiness"]["backfill_complete"] is True - # Post-reauthorization: the next save backfills blood_glucose from the + # Post-reauthorization: the next save backfills workout from the # horizon and clears the gap. today = dt.date(2026, 1, 11) second_transport = ScriptedTransport() for endpoint in oura.SYNC_ENDPOINTS: - if endpoint != "blood_glucose": + if endpoint != "workout": second_transport.script(endpoint, _ok_page([])) - _script_horizon_pages( - second_transport, "blood_glucose", today, data_on_last_chunk=True - ) + _script_horizon_pages(second_transport, "workout", today, data_on_last_chunk=True) second = oura.backend.sync( journal, dry_run=False, @@ -2277,13 +2583,13 @@ def test_sync_skips_endpoint_missing_scope_and_keeps_run_alive( today=today, ) - glucose_urls = [u for u, _ in second_transport.calls if "blood_glucose" in u] - assert "start_datetime=2015-01-01" in glucose_urls[0] + workout_urls = [u for u, _ in second_transport.calls if "/workout?" in u] + assert "start_date=2015-01-01" in workout_urls[0] assert second["errors"] == [] - assert second["downloaded"] == 4 + assert second["downloaded"] == 2 state = json.loads((journal / "imports" / "oura.json").read_text()) - assert state["endpoints"]["blood_glucose"]["backfill_complete"] is True - assert state["endpoints"]["blood_glucose"]["high_water_day"] == "2026-01-03" + assert state["endpoints"]["workout"]["backfill_complete"] is True + assert state["endpoints"]["workout"]["high_water_day"] == "2026-01-03" def test_token_death_still_fails_the_whole_run(tmp_path: Path, monkeypatch):