diff --git a/README.md b/README.md index 29b9c26..21fd10a 100644 --- a/README.md +++ b/README.md @@ -60,3 +60,9 @@ Available as of [2026-05-08](./CHANGELOG.md#2026-05-08) - `GET /xrpc/com.atproto.sync.getRepoStatus` - `GET /xrpc/com.atproto.sync.listRepos` - `GET /xrpc/com.atproto.sync.listBlobs` + +## Credits + +- [Cocoon](https://github.com/haileyok/cocoon) - a PDS written in Go +- [Tranquil](https://tangled.org/tranquil.farm/tranquil-pds) - a PDS written in Rust +- [PDS Reference Implementation](https://github.com/bluesky-social/atproto/tree/main/packages/pds) diff --git a/docs/specs/lexicon-schemas.md b/docs/specs/lexicon-schemas.md index d29b28e..056cda8 100644 --- a/docs/specs/lexicon-schemas.md +++ b/docs/specs/lexicon-schemas.md @@ -1,9 +1,13 @@ --- title: Lexicon Schema Loading and Generation -updated: 2026-05-07 +updated: 2026-05-16 --- -Tempest needs generic Lexicon validation without baking application schemas into validator code. Record APIs should validate known schemas, tolerate unknown schemas when the request permits it, and give operators a reproducible way to update the known schema set. +Tempest needs generic Lexicon validation without baking application schemas into +validator code. Record APIs should validate known schemas, tolerate unknown schemas +when the request permits it, and give operators a reproducible way to update the known +schema set. The registry should support bundled generated schemas, operator-configured +local schemas, and policy-controlled network resolution. ## Reference Implementation Baseline @@ -19,7 +23,18 @@ Observed behavior: - known `$type` validates the rkey and record body against the schema. - unknown `$type` with `validate: true` fails. - unknown `$type` with `validate` unset returns `validationStatus: "unknown"`. -- The reference implementation has an explicit TODO to replace the static known schema map with automatically fetched and built schemas. +- The reference implementation has an explicit TODO to replace the static known schema + map with automatically fetched and built schemas. + +Other references: + +- Cocoon currently does not appear to implement a real Lexicon schema registry for + record writes. It validates basic request fields, writes records, and marks write + responses `valid` with an inline TODO noting that status may not be true. +- Tranquil PDS separates registry, validator, and optional resolver concerns. Its + resolver path supports DNS/DID/PDS lookup, positive and negative caching, + single-flight resolution, response-size limits, stale-cache behavior, and an offline + mode. Sources checked: @@ -28,16 +43,36 @@ Sources checked: - `packages/lexicon/src/lexicons.ts` - `packages/lexicon/src/validators/complex.ts` - `packages/lexicon/src/validators/primitives.ts` +- `haileyok/cocoon/server/repo.go` +- `haileyok/cocoon/server/handle_repo_create_record.go` +- `tranquil-pds/crates/tranquil-lexicon/src/registry.rs` +- `tranquil-pds/crates/tranquil-lexicon/src/validate.rs` +- `tranquil-pds/crates/tranquil-lexicon/src/resolve.rs` +- `tranquil-pds/crates/tranquil-lexicon/src/dynamic.rs` ## Tempest Design Tempest should keep three concerns separate: -1. Lexicon schema engine: generic validation of documents, definitions, refs, unions, objects, arrays, and primitive constraints. +1. Lexicon schema engine: generic validation of documents, definitions, refs, unions, + objects, arrays, and primitive constraints. 2. Lexicon registry: lookup boundary for known Lexicon documents. -3. Lexicon sources: bundled, generated, configured, or externally resolved schema documents. +3. Lexicon sources: bundled, generated, configured, or externally resolved schema + documents. + +The schema engine must not contain product- or app-specific schemas. Bluesky Lexicons +can be used in tests and can be bundled as generated artifacts, but they should remain +data, not branches in validator code. -The schema engine must not contain product- or app-specific schemas. Bluesky Lexicons can be used in tests and can be bundled as generated artifacts, but they should remain data, not branches in validator code. +The registry exists to answer one runtime question for record writes: given a collection +NSID, can Tempest find a trusted Lexicon document that validates the write? + +The registry should be a deterministic lookup/index boundary over trusted local sources +plus explicitly enabled external sources. It should not silently swallow invalid +documents or duplicate ids at startup/generation time, because operators need failures +that explain why a schema set is unsafe or ambiguous. Externally resolved schemas should +be cached with metadata and should never override bundled or configured local schemas +without an explicit policy. ## Known Schema Sources @@ -45,7 +80,12 @@ Tempest should support these sources in order: - Bundled generated schemas from an explicit atproto source commit. - Operator-configured local schema files for custom applications. -- Later: external Lexicon resolution and caching once a resolution policy exists. +- Policy-controlled external Lexicon resolution and caching. + +Bundled and configured local schemas are the deterministic baseline. External resolution +is part of schema completion, but it must be explicitly configured, bounded, cached, and +safe. If external resolution is disabled or unavailable in optimistic validation mode, +unknown schemas still follow reference fail-open behavior. Bundled schemas should include metadata: @@ -73,18 +113,29 @@ Generated output must be reproducible from the same source commit. ## External Resolution -External resolution is future work and must be policy-driven. +External resolution is part of milestone 09 and must be policy-driven. The +implementation should be behind explicit configuration, use conservative network limits, +and preserve reference validation semantics. -Open policy decisions: +Policy decisions to settle before enabling it by default: - Which NSID authorities can be resolved. -- Whether resolution uses DNS, HTTPS well-known paths, PLC metadata, or another mechanism. +- Whether resolution uses DNS `_lexicon`, DNS `_atproto` fallback, DID documents, + PLC metadata, or another mechanism. - How to verify schema authorship. - Cache TTL and invalidation. - Whether a PDS should allow writes using externally resolved schemas by default. - How to protect against SSRF, oversized schemas, deep refs, and dependency cycles. -Until those decisions are implemented, unknown schemas should follow the reference behavior: accepted as `unknown` unless `validate: true`. +Resolution should follow the published Lexicon model: derive the authority from the +collection NSID, resolve the authority to a DID, resolve that DID to a PDS service +endpoint, then fetch the `com.atproto.lexicon.schema` record whose rkey is the schema +NSID. Fetched documents must match the requested NSID, use supported Lexicon language +version `1`, pass generic document validation, and respect response-size, redirect, +timeout, address-range, cache, and recursion limits. + +If external resolution is disabled, fails, or cannot find a schema, unknown schemas +should follow the reference behavior: accepted as `unknown` unless `validate: true`. ## Validation Semantics @@ -95,14 +146,17 @@ For record writes: - If a known record schema exists and validation is not disabled, validate the record body. - If no schema exists and `validate: true`, reject with `InvalidRequest`. - If no schema exists and validation is unset, return `validationStatus: "unknown"`. -- If `validate: false`, skip schema validation and omit or return unknown validation status according to endpoint compatibility needs. +- If `validate: false`, skip schema validation and omit or return unknown validation + status according to endpoint compatibility needs. ## Adversarial Checks - A schema cannot define refs that escape loader limits or create unbounded recursion. - External schema fetches must not reach private or local addresses. -- A generated schema bundle must be tied to a source commit so compatibility regressions are explainable. -- Unknown Lexicon records cannot bypass generic record safety checks such as `$type`, collection NSID, record key syntax, record size, and CBOR limits. +- A generated schema bundle must be tied to a source commit so compatibility regressions + are explainable. +- Unknown Lexicon records cannot bypass generic record safety checks such as `$type`, + collection NSID, record key syntax, record size, and CBOR limits. - A malicious configured Lexicon must not crash validation or exhaust CPU/memory. ## HTTP Verification diff --git a/docs/tasks/09-lexicon-schemas.md b/docs/tasks/09-lexicon-schemas.md index 75dd853..ef94964 100644 --- a/docs/tasks/09-lexicon-schemas.md +++ b/docs/tasks/09-lexicon-schemas.md @@ -4,31 +4,27 @@ specs: - ../specs/lexicon-schemas.md --- -Goal: load, generate, and manage known Lexicon schemas without hardcoding application schemas in validation logic. +Goal: load, generate, resolve, and manage known Lexicon schemas without hardcoding application schemas in validation logic. ## Tasks - [ ] T09-01: Define Lexicon registry behaviour and document provider boundary. -- [ ] T09-02: Add generic Lexicon document shape validation. -- [ ] T09-03: Add duplicate document id and duplicate definition-ref checks. -- [ ] T09-04: Add deterministic Lexicon manifest format. -- [ ] T09-05: Add generator task for pinned atproto `lexicons/` input. -- [ ] T09-06: Emit source metadata: repo, commit, generated timestamp, and document count. -- [ ] T09-07: Generate bundled known-schema data for selected record schemas. -- [ ] T09-08: Wire generated schemas into the runtime registry. -- [ ] T09-09: Add operator-configured local Lexicon directory support. -- [ ] T09-10: Add startup validation for configured Lexicon directories. -- [ ] T09-11: Add tests for known schema `valid` status. -- [ ] T09-12: Add tests for unknown schema with validation unset returning `unknown`. -- [ ] T09-13: Add tests for unknown schema with `validate: true` failing. -- [ ] T09-14: Add tests for ref cycles, deep refs, oversized schemas, and duplicate ids. -- [ ] T09-15: Document external Lexicon resolution policy before implementing network fetches. -- [ ] T09-16: Add optional external resolver interface behind a disabled-by-default config flag. -- [ ] T09-17: Add SSRF and response-size protections for the external resolver. -- [ ] T09-18: Add cache metadata for externally resolved schemas. -- [ ] T09-19: Add compatibility test against official atproto profile/post/follow record schemas. -- [ ] T09-20: Document schema update workflow for operators. -- [ ] T09-21: Add Hurl smoke test for generated and unknown schema validation behavior. +- [ ] T09-02: Add generic Lexicon document validation, including duplicate ids, duplicate refs, and loader limits. +- [ ] T09-03: Add deterministic Lexicon manifest format with source repo, commit, generated timestamp, counts, and document ids. +- [ ] T09-04: Add generator task for pinned atproto `lexicons/` input. +- [ ] T09-05: Generate bundled known-schema data for selected record schemas and wire it into the runtime registry. +- [ ] T09-06: Add operator-configured local Lexicon directory support with startup validation. +- [ ] T09-07: Preserve record-write validation modes: known `valid`, optimistic unknown, strict unknown failure, and explicit validation skip. +- [ ] T09-08: Add tests for refs, unions, ref cycles, deep refs, oversized schemas, duplicate ids, and duplicate refs. +- [ ] T09-09: Document external Lexicon resolution policy, default configuration, and source precedence. +- [ ] T09-10: Add external resolver interface behind an explicit config flag. +- [ ] T09-11: Implement NSID authority resolution through DNS/DID/PDS `com.atproto.lexicon.schema` records. +- [ ] T09-12: Add SSRF, redirect, timeout, response-size, address-range, and recursion protections for the external resolver. +- [ ] T09-13: Add positive, negative, stale, and single-flight cache behavior for externally resolved schemas. +- [ ] T09-14: Ensure externally resolved schemas cannot override bundled or configured local schemas unless explicitly allowed. +- [ ] T09-15: Add compatibility tests against official atproto profile/post/follow record schemas. +- [ ] T09-16: Add resolver tests for disabled, unknown, success, cache hit, negative cache, oversized response, and private-address rejection paths. +- [ ] T09-17: Document operator schema update workflow and add Hurl smoke tests for generated, resolved, and unknown schema behavior. ## Integration Tests @@ -37,6 +33,9 @@ Goal: load, generate, and manage known Lexicon schemas without hardcoding applic - Unknown schema with `validate: true` fails. - Configured local custom schema validates a custom record. - Duplicate schema ids fail startup or generation. +- Externally resolved custom schema validates when resolver config permits it. +- External resolver failures return `unknown` in optimistic mode and fail when `validate: true`. +- External resolver refuses private/local network targets and oversized schema responses. ## CLI Verification @@ -54,5 +53,6 @@ hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/lexico Expected: - Known generated schema returns `validationStatus: "valid"`. +- Configured external schema resolver returns `validationStatus: "valid"` for a resolvable schema. - Unknown schema with validation unset returns `validationStatus: "unknown"`. - Unknown schema with `validate: true` returns `InvalidRequest`.