diff --git a/docs/design/oura-import.md b/docs/design/oura-import.md index d908c8eea..3cf68a9bd 100644 --- a/docs/design/oura-import.md +++ b/docs/design/oura-import.md @@ -49,7 +49,7 @@ Endpoint names below are from model knowledge of the v2 API; **verify each again | `sleep` | per-period: `bedtime_start/end`, `type` (long_sleep/late_nap/…), stage durations (`deep_sleep_duration`, `rem_sleep_duration`, `light_sleep_duration`, `awake_time`), `efficiency`, `latency`, `average_heart_rate`, `lowest_heart_rate`, `average_hrv`, `average_breath`, `sleep_phase_5_min` hypnogram (1=deep, 2=light, 3=REM, 4=awake) | **Partly** — stages also mirror into AH as `HKCategoryValueSleepAnalysis*` intervals, but Oura-native durations, efficiency/latency, per-period HRV/HR aggregates, and the 5-minute hypnogram string are richer and carry Oura's own period identity (`id`, `day` attribution) | ✅ | | `heartrate`, `daily_activity`, `workout`, `session`, `enhanced_tag`, `sleep_time`, `ring_configuration`, `vO2max` / `daily_cardiovascular_age` | series + activity + tags + device | Mostly **duplicates the AH mirror** (HR, steps, workouts) or is metadata; excluded from the first import scope to avoid double-counting in presentation | ⚠ names | -Other API facts to verify live at O2: OAuth2 endpoints (`cloud.ouraring.com/oauth/authorize`, `api.ouraring.com/oauth/token` ⚠), scopes (`email personal daily heartrate workout tag session spo2` ⚠), rate limit (historically 5000 requests / 5 min ⚠), the no-auth sandbox (`/v2/sandbox/usercollection/*` ⚠), personal-access-token deprecation status ⚠, webhook subscription API ⚠. +Other API facts to verify live at O2: OAuth2 endpoints (`cloud.ouraring.com/oauth/authorize`, `api.ouraring.com/oauth/token` ⚠), scopes (`daily heartrate workout tag session spo2 stress heart_health metabolic`), rate limit (historically 5000 requests / 5 min ⚠), the no-auth sandbox (`/v2/sandbox/usercollection/*` ⚠), personal-access-token deprecation status ⚠, webhook subscription API ⚠. **Skeleton scope (implemented):** `daily_sleep`, `daily_readiness` (+ split-out `temperature_deviation` rows), `daily_resilience`, `daily_stress`, `daily_spo2`, `sleep`. That is exactly the "scores + stages" slice the day pages need and the AH mirror can't provide. @@ -198,7 +198,21 @@ Never tokens, never client credentials, never raw values in the cursor. Catalog - Redirect: loopback `http://localhost:/callback` on the journal host, opened in the owner's browser with the owner at the keyboard. No headless, no automated retry, no unattended re-auth ever; if tokens die, sync degrades to a factual "authorization needed" status until Jack runs the step again. - **Token boundary: journal configuration, never the repo.** Client id, (secret if applicable), access + refresh tokens live in the journal's config domain under the reserved key `oura` (`OAUTH_CONFIG_KEY` in the skeleton), written exclusively through the config owner `solstone/think/journal_config.py` (L2). Never in this repository, never in env vars, never in logs, never in `imports/oura.json`, never through Oracle/Claude prompts. Refresh-token rotation writes through the same owner. - **Dev-account cap noted:** Oura developer apps are limited to roughly 10 users before requiring Oura's partnership review — irrelevant for a single owner, but it means client credentials must never be shared or committed, and a future multi-owner story needs Oura's blessing first. -- Scopes: request the minimum for the import scope (daily + sleep + spo2-family; exact scope names ⚠ verify at O2). +- Scopes: future owner-present authorization requests ask for `daily`, `heartrate`, `workout`, `tag`, `session`, `spo2`, `stress`, `heart_health`, and `metabolic`. `email` and `personal` are no longer requested. + +| Scope | Endpoint family authorized | Polled? | Notes | +|---|---|---|---| +| `daily` | daily sleep/readiness/activity documents | Yes | Core daily scores and activity documents. | +| `heartrate` | `heartrate` series | Yes | Oura-native high-frequency series; normalized with owner-local day attribution. | +| `workout` | `workout` documents | Yes | Activity interval documents. | +| `tag` | `enhanced_tag` documents | Yes | Owner-entered tag metadata. | +| `session` | `session` documents | Yes | Meditation/breathing/rest sessions. | +| `spo2` | `daily_spo2` documents | Yes | Nightly oxygen summary documents. | +| `stress` | `daily_resilience`, `daily_stress` | Yes | Live Oura scope system maps this to resilience/stress families. | +| `heart_health` | `daily_cardiovascular_age`, `vO2_max` | Yes | Live Oura scope system maps this to cardiovascular age and VO2 max. | +| `metabolic` | `blood_glucose` only | No | Kept deliberately, but `blood_glucose` remains partner-gated in `_PARTNER_GATED_ENDPOINTS` and is never in `SYNC_ENDPOINTS`. | + +Removing `email` and `personal` changes only what future authorization requests ask for. It does not retroactively revoke scopes already granted on an already-issued token. Narrowing an existing token's granted scopes requires owner-present re-consent and/or revoking the old token; this implementation does not perform that operator action. --- @@ -213,6 +227,10 @@ Landed in the skeleton: Phase O3 extends the health gate to sync with a separate `oura_sync_preflight` artifact: any save-mode `sync()` calls `enforce_oura_sync_gate(...)` before its first journal write, with the confirm flag passed explicitly from the CLI/scheduler invocation. Scheduled runs require `scheduled_sync.approved: true`, a cadence, and an unexpired timezone-aware `scheduled_sync.valid_until`; a scheduled job never self-confirms implicitly. +The save-mode sync lock is `hold_lock(journal_root / "imports" / "oura.json", mode=0o600)`, which creates the private sidecar `imports/oura.json.lock`. The first gate runs before the lock so an initially invalid artifact creates nothing. The authoritative gate then runs again inside the lock before cursor read, client construction, token refresh, fetch, import-id allocation, bundle writes, or cursor advance. The lock is held through all of those steps, closing the import-id TOCTOU. If token refresh happens while the health lock is held, the ordering is health lock -> config lock; config writes do not acquire the health lock, so there is no inverse path. + +Oura sync applies the validated raw-retention decision from `PreSaveGateDecision`: `retain_parsed` writes raw API page JSONL under `imports//raw/oura/`; `discard` writes no raw pages and no new `raw_ref` values; `retain_complete` is rejected by the gate as source-incompatible. Scheduled consent is valid only while `now < scheduled_sync.valid_until`; missing, malformed, naive, or expired values block before any network or journal write. Consent expiring after the authoritative in-lock gate passes does not abort the running transaction. + --- ## 7. (f) Phased rollout @@ -258,8 +276,11 @@ commit (the hourly sync lane runs against repo HEAD): untouched as a safety copy; they are simply no longer read). - **Scopes.** The connect flow (`journal importer --connect oura`) now requests an explicit scope set and prints it for the owner: - `email personal daily heartrate workout tag session spo2 stress - heart_health metabolic`. The first eight are Oura's documented set; + `daily heartrate workout tag session spo2 stress heart_health metabolic`. + `email` and `personal` were removed from future authorization requests; + existing tokens keep whatever scopes they were already granted until the + owner re-consents or revokes them. The first six are Oura's documented + health-data set; `stress` / `heart_health` / `metabolic` are live but undocumented (evidence: tidepool-org/platform's Oura partner integration maps `extapi:stress` → daily_resilience, `extapi:heart_health` → diff --git a/docs/health_imports.md b/docs/health_imports.md index 322602b6c..5819224fe 100644 --- a/docs/health_imports.md +++ b/docs/health_imports.md @@ -54,9 +54,8 @@ Apple Health records are assigned to the local calendar day encoded in each reco Health-owned day-summary stream identities live in `HEALTH_IMPORTER_REGISTRY` in `health_schema.py`. The Apple Health summary stream name is `import.apple_health`. -The Oura stream identity `import.oura` is reserved there for exclusion/ownership bookkeeping only; no Oura day-summary writer exists. Later source-specific streams can be added only after privacy preflight and save-mode tests exist: +The Oura stream identity `import.oura` is reserved there for exclusion/ownership bookkeeping only; no Oura day-summary writer exists. The reserved identity makes the think pipeline exclude future Oura day cards only if a writer is explicitly added later. Later source-specific streams can be added only after privacy preflight and save-mode tests exist: -- `import.oura` - `import.dexcom_clarity` - `import.health_auto_export` @@ -82,7 +81,11 @@ Before any live-journal save-mode health import: - Require explicit user confirmation that the export contains sensitive health data. - Print or display the target journal path before writing. -- Confirm whether raw export files should be discarded, parsed-only retained, or completely retained. +- Choose a closed raw-retention decision: + - `discard`: Apple Health writes no `raw/`; Oura writes no raw API page JSONL; normalized rows and newly inserted dedupe rows carry no `raw_ref`. + - `retain_parsed`: Apple Health stores only `imports//raw/export.xml` for either zip or directory inputs; Oura stores parsed raw API page JSONL under `imports//raw/oura/`. + - `retain_complete`: Apple Health may copy the original zip or full export tree; Oura rejects this value as source-incompatible. +- Set `raw_retention.unparsed_sensitive_modalities_acknowledged: true` when, and only when, `retain_complete` is chosen. - Confirm whether replicated devices or backups are allowed to carry raw health data. - Never send raw health data, tokens, service-account JSON, or export files through Oracle, Claude, or other remote review prompts. - Never commit real health fixtures. @@ -90,6 +93,12 @@ Before any live-journal save-mode health import: The required approval artifact lives at `imports/_approvals/health_import_preflight.json` in the target journal. It must match `solstone.health_import_preflight.checklist.v3`, bind an absolute `journal_root`, contain a closed `raw_retention.decision` of `discard`, `retain_parsed`, or `retain_complete`, and contain a decision for each replication destination: `time_machine`, `icloud`, `solbase`, `hosted_backup`, and `other`. `retain_complete` also requires `raw_retention.unparsed_sensitive_modalities_acknowledged: true`. +The Oura sync approval artifact lives at `imports/_approvals/oura_sync_preflight.json`. It must match `solstone.oura_sync_preflight.checklist.v2`, use the same raw-retention enum (`discard` or `retain_parsed` only), bind an absolute `journal_root`, and include the replication decisions. Scheduled sync consent additionally requires `scheduled_sync.approved: true`, a cadence, and a timezone-aware ISO-8601 `scheduled_sync.valid_until`; expired or malformed standing consent fails closed before any network or journal write. + +Owner remediation after this hardening lands is manual by design: update both approval artifacts to checklist v3/v2, migrate old raw-retention strings to the enum (`retain_compressed_zip` -> `retain_complete` only if the owner accepts complete Apple Health raw retention; `retain_raw_pages` -> `retain_parsed` for Oura), add `unparsed_sensitive_modalities_acknowledged: true` only for `retain_complete`, and add/refresh scheduled `valid_until` for Oura if unattended sync should continue. + +The Oura connect flow now asks future authorization requests for nine scopes: `daily`, `heartrate`, `workout`, `tag`, `session`, `spo2`, `stress`, `heart_health`, and `metabolic`. Removing `email` and `personal` changes only what future authorization requests ask for. It does not retroactively revoke scopes already granted on an already-issued token. Narrowing an existing token's granted scopes requires owner-present re-consent and/or revoking the old token; this code change does not perform that operator action. + ## Current Deferred Work Shipped on this branch: @@ -97,6 +106,7 @@ Shipped on this branch: - Oura OAuth via `journal importer --connect oura`. - Oura token storage and refresh in journal config through `oura_auth.py`. - Oura API sync that writes health bundles and sync cursor state. +- Oura save-mode sync locking, scheduled-consent expiry, and egress/scope guardrails. Still deferred: diff --git a/solstone/think/importers/oura_auth.py b/solstone/think/importers/oura_auth.py index f84c0f018..e54a05440 100644 --- a/solstone/think/importers/oura_auth.py +++ b/solstone/think/importers/oura_auth.py @@ -48,7 +48,7 @@ CONFIG_SECTION = "oura" # OAuth scopes the connect flow requests, researched 2026-07-07: # -# - The first eight are Oura's documented scope set +# - The first six are Oura's documented health-data scope set # (cloud.ouraring.com/docs/authentication). # - ``stress`` (daily_resilience), ``heart_health`` # (daily_cardiovascular_age, vo2_max), and ``metabolic`` (blood_glucose) @@ -58,6 +58,10 @@ CONFIG_SECTION = "oura" # authorize front door verifiably rewrites a plain ``scope=X`` query to # ``extapi:X`` before handing off to its authorization server — so the # plain names below are the correct request form. +# - ``metabolic`` authorizes ``blood_glucose`` only. That endpoint stays +# partner-gated and is not polled by the sync engine; the scope remains +# requested so a future owner-approved polling change will not force an +# immediate second consent round-trip. # - Empirically (2026-07-06), a token granted with NO scope parameter # reads resilience and cardiovascular-age documents but 401s on # usercollection/blood_glucose: ``metabolic`` is not part of the default @@ -66,8 +70,6 @@ CONFIG_SECTION = "oura" # The consent screen (owner-present) is the final validator: scope names # Oura rejects or drops surface there, with the owner at the keyboard. OAUTH_SCOPES: tuple[str, ...] = ( - "email", - "personal", "daily", "heartrate", "workout", diff --git a/tests/test_oura_auth.py b/tests/test_oura_auth.py index 0ccbddec5..c8522b148 100644 --- a/tests/test_oura_auth.py +++ b/tests/test_oura_auth.py @@ -117,8 +117,12 @@ def test_owner_present_auth_uses_fixed_redirect_pkce_and_private_token_exchange( # The full researched scope set is requested by default — including # the blood-glucose (metabolic) scope the default grant lacks. assert params["scope"] == [" ".join(OAUTH_SCOPES)] - assert "metabolic" in params["scope"][0].split() - assert "heart_health" in params["scope"][0].split() + requested_scopes = params["scope"][0].split() + assert len(requested_scopes) == 9 + assert "metabolic" in requested_scopes + assert "heart_health" in requested_scopes + assert "email" not in requested_scopes + assert "personal" not in requested_scopes def test_wrong_state_and_wrong_path_do_not_exchange_tokens(): diff --git a/tests/test_oura_egress_guard.py b/tests/test_oura_egress_guard.py new file mode 100644 index 000000000..e936da6f2 --- /dev/null +++ b/tests/test_oura_egress_guard.py @@ -0,0 +1,272 @@ +# SPDX-License-Identifier: AGPL-3.0-only +# Copyright (c) 2026 sol pbc + +"""Offline egress guard for Oura importer modules.""" + +from __future__ import annotations + +import ast +from pathlib import Path + +IMPORTER_ROOT = Path(__file__).resolve().parents[1] / "solstone" / "think" / "importers" + +ALLOWED_EGRESS: frozenset[tuple[str, str, str]] = frozenset( + { + ("oura", "_default_transport", "http_client"), + ("oura_auth", "", "module_network_import"), + ("oura_auth", "_default_http_transport", "http_client"), + ("oura_auth", "_post_token_request", "http_client"), + ("oura_auth", "_authorization_url", "authorization_url"), + ("oura_auth", "_CallbackHTTPServer", "loopback_http_server"), + ("oura_auth", "_CallbackHandler", "loopback_http_server"), + ("oura_auth", "run_owner_present_auth", "browser_open"), + ("oura_auth", "run_owner_present_auth", "loopback_http_server"), + } +) + +SAFE_IMPORTS: frozenset[str] = frozenset({"urllib.parse"}) + +IMPORT_CAPABILITIES: dict[str, str] = { + "aiohttp": "http_client", + "http": "http_client", + "http.client": "http_client", + "http.server": "loopback_http_server", + "httpx": "http_client", + "importlib": "dynamic_import", + "requests": "http_client", + "smtplib": "mail_client", + "socket": "socket", + "ssl": "tls", + "subprocess": "process_escape", + "urllib": "http_client", + "urllib.error": "http_client", + "urllib.request": "http_client", + "webbrowser": "browser_open", +} + +CALL_CAPABILITIES: dict[str, str] = { + "__import__": "dynamic_import", + "_CallbackHTTPServer": "loopback_http_server", + "aiohttp": "http_client", + "http.client": "http_client", + "http.server.BaseHTTPRequestHandler": "loopback_http_server", + "http.server.HTTPServer": "loopback_http_server", + "httpx": "http_client", + "importlib": "dynamic_import", + "os.popen": "process_escape", + "os.system": "process_escape", + "requests": "http_client", + "smtplib": "mail_client", + "socket": "socket", + "ssl": "tls", + "subprocess": "process_escape", + "urllib.request": "http_client", + "webbrowser.open": "browser_open", +} + + +def _oura_module_paths() -> list[Path]: + return sorted(IMPORTER_ROOT.glob("oura*.py")) + + +def _parents(tree: ast.AST) -> dict[ast.AST, ast.AST]: + parents: dict[ast.AST, ast.AST] = {} + for parent in ast.walk(tree): + for child in ast.iter_child_nodes(parent): + parents[child] = parent + return parents + + +def _owner(node: ast.AST, parents: dict[ast.AST, ast.AST]) -> str: + function_owner: str | None = None + cursor: ast.AST | None = node + while cursor is not None: + if isinstance(cursor, ast.ClassDef): + if cursor.name in {"_CallbackHTTPServer", "_CallbackHandler"}: + return cursor.name + return function_owner or cursor.name + if isinstance(cursor, (ast.FunctionDef, ast.AsyncFunctionDef)): + function_owner = cursor.name + cursor = parents.get(cursor) + return function_owner or "" + + +def _rooted_capability(name: str, capabilities: dict[str, str]) -> str | None: + if any(name == safe or name.startswith(f"{safe}.") for safe in SAFE_IMPORTS): + return None + for prefix, capability in sorted( + capabilities.items(), key=lambda item: len(item[0]), reverse=True + ): + if name == prefix or name.startswith(f"{prefix}."): + return capability + return None + + +def _import_names(node: ast.Import | ast.ImportFrom) -> list[tuple[str, str]]: + if isinstance(node, ast.Import): + return [ + (alias.name, alias.asname or alias.name.split(".")[0]) + for alias in node.names + ] + module = node.module or "" + names: list[tuple[str, str]] = [] + for alias in node.names: + if alias.name == "*": + names.append((module, alias.asname or alias.name)) + else: + names.append( + ( + f"{module}.{alias.name}" if module else alias.name, + alias.asname or alias.name, + ) + ) + return names + + +def _bindings(tree: ast.AST) -> dict[str, str]: + bindings: dict[str, str] = {} + for node in ast.walk(tree): + if not isinstance(node, (ast.Import, ast.ImportFrom)): + continue + for full_name, bound_name in _import_names(node): + bindings[bound_name] = full_name + root = full_name.split(".", 1)[0] + if bound_name == root: + bindings[root] = root + return bindings + + +def _dotted_name(node: ast.AST) -> str | None: + if isinstance(node, ast.Name): + return node.id + if isinstance(node, ast.Attribute): + base = _dotted_name(node.value) + if base: + return f"{base}.{node.attr}" + return None + + +def _resolve_name(name: str, bindings: dict[str, str]) -> str: + head, dot, tail = name.partition(".") + bound = bindings.get(head) + if not bound: + return name + return f"{bound}{dot}{tail}" if dot else bound + + +def _allowed(module: str, owner: str, capability: str) -> bool: + if (module, owner, capability) in ALLOWED_EGRESS: + return True + return ( + owner == "" + and capability != "dynamic_import" + and (module, owner, "module_network_import") in ALLOWED_EGRESS + ) + + +def _violation( + module: str, + owner: str, + capability: str, + detail: str, +) -> str | None: + if _allowed(module, owner, capability): + return None + return f"{module}.{owner}: {capability}: {detail}" + + +def _egress_violations(module: str, source: str) -> list[str]: + tree = ast.parse(source) + parents = _parents(tree) + bindings = _bindings(tree) + violations: list[str] = [] + + for node in ast.walk(tree): + if isinstance(node, (ast.Import, ast.ImportFrom)): + owner = _owner(node, parents) + for full_name, _bound_name in _import_names(node): + capability = _rooted_capability(full_name, IMPORT_CAPABILITIES) + if capability is None: + continue + violation = _violation(module, owner, capability, f"import {full_name}") + if violation: + violations.append(violation) + continue + + if isinstance(node, ast.ClassDef): + for base in node.bases: + name = _dotted_name(base) + if name is None: + continue + resolved = _resolve_name(name, bindings) + capability = _rooted_capability(resolved, CALL_CAPABILITIES) + if capability is None: + continue + violation = _violation( + module, node.name, capability, f"class base {resolved}" + ) + if violation: + violations.append(violation) + continue + + if isinstance(node, ast.Call): + name = _dotted_name(node.func) + if name is None: + continue + resolved = _resolve_name(name, bindings) + capability = _rooted_capability(resolved, CALL_CAPABILITIES) + if capability is None: + continue + owner = _owner(node, parents) + violation = _violation(module, owner, capability, f"call {resolved}") + if violation: + violations.append(violation) + continue + + if isinstance(node, ast.Attribute): + name = _dotted_name(node) + if name is None: + continue + resolved = _resolve_name(name, bindings) + capability = CALL_CAPABILITIES.get(resolved) + if capability is None: + continue + owner = _owner(node, parents) + violation = _violation(module, owner, capability, f"reference {resolved}") + if violation: + violations.append(violation) + + return violations + + +def test_oura_egress_guard_covers_all_oura_modules() -> None: + paths = _oura_module_paths() + + assert paths + assert {path.name for path in paths} >= {"oura.py", "oura_auth.py"} + + violations: list[str] = [] + for path in paths: + violations.extend( + _egress_violations(path.stem, path.read_text(encoding="utf-8")) + ) + + assert violations == [] + + +def test_oura_egress_guard_reports_synthetic_violation() -> None: + source = """ +import socket + +def added_escape_hatch(): + import subprocess + socket.socket() + __import__("ssl") + subprocess.run(["true"]) +""" + + violations = _egress_violations("oura_future", source) + + assert any("socket" in violation for violation in violations) + assert any("dynamic_import" in violation for violation in violations) + assert any("process_escape" in violation for violation in violations) diff --git a/tests/test_oura_importer.py b/tests/test_oura_importer.py index 7200910db..2c22aa02c 100644 --- a/tests/test_oura_importer.py +++ b/tests/test_oura_importer.py @@ -5,7 +5,6 @@ from __future__ import annotations -import ast import datetime as dt import json import os @@ -3111,6 +3110,8 @@ def test_cli_connect_oura_prints_scopes_and_saves_tokens_to_config( # the blood-glucose (metabolic) scope this reauthorization adds. assert "Requesting scopes: " + " ".join(oura_auth.OAUTH_SCOPES) in out assert "metabolic" in out + assert "email" not in out + assert "personal" not in out assert "saved to journal config" in out # No token material in owner-facing output. assert "connected-access" not in out @@ -3196,52 +3197,6 @@ def test_cli_sync_lock_timeout_prints_typed_message( assert "Bearer" not in out -# --------------------------------------------------------------------------- -# No-network guard: the module's only network-capable import lives inside -# the injectable default transport, which tests never construct. -# --------------------------------------------------------------------------- - -_NETWORK_ROOTS = {"requests", "httpx", "aiohttp", "socket", "http"} - - -def _imported_module_names(node: ast.AST) -> list[str]: - if isinstance(node, ast.Import): - return [alias.name for alias in node.names] - if isinstance(node, ast.ImportFrom): - return [node.module or ""] - return [] - - -def test_module_has_no_reachable_network_imports(): - source = Path(oura.__file__).read_text(encoding="utf-8") - tree = ast.parse(source) - - # Module level: no network-capable imports at all. urllib.parse is - # pure string handling (query encoding) and allowed; the socket-backed - # urllib.request / urllib.error are not. - for node in tree.body: - for name in _imported_module_names(node): - root = name.split(".")[0] - assert root not in _NETWORK_ROOTS, f"module-level import {name!r}" - assert name not in {"urllib", "urllib.request", "urllib.error"}, ( - f"module-level import {name!r}" - ) - - # Function level: network-capable imports may exist ONLY inside the - # injectable default transport. - for func in ast.walk(tree): - if not isinstance(func, (ast.FunctionDef, ast.AsyncFunctionDef)): - continue - for node in ast.walk(func): - for name in _imported_module_names(node): - root = name.split(".")[0] - if root in _NETWORK_ROOTS or name.startswith("urllib"): - assert func.name == "_default_transport", ( - f"network-capable import {name!r} outside " - f"_default_transport (in {func.name!r})" - ) - - def test_save_records_fetch_windows_and_catalog_reports_known_refetch( tmp_path: Path, monkeypatch ):