diff --git a/docs/specs/README.md b/docs/specs/README.md index c1ea9fc..6bcbfa2 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -23,10 +23,11 @@ Subsystem specifications live in this directory. Milestone task plans live in `. 10. [Admin Operations](admin-operations.md) 11. [Deployment and Observability](deployment-observability.md) 12. [Interop and Integration Testing](interop-testing.md) +13. [Hurl Smoke Tests](hurl-smoke-tests.md) ## Milestones -Milestone tasks live in `docs/tasks/`. Each task file is intended to be small enough to implement and review without guessing at scope. Each milestone must end with a black-box HTTP check using `curl` or `http`. +Milestone tasks live in `docs/tasks/`. Each task file is intended to be small enough to implement and review without guessing at scope. Each milestone must end with a black-box Hurl smoke test. Start with [Milestone 00](../tasks/00-foundation.md), then continue in numeric order. @@ -51,5 +52,6 @@ Research was checked on 2026-05-07 against: - Keep protocol facts separate from project choices. - Every subsystem spec must include HTTP verification. - Every milestone must have integration tests before it is considered done. +- Smoke tests must be Hurl files under `test/smoke/*.hurl` and must run with `hurl --test`. - Use `Req` for outbound HTTP in application code. - Run `mix precommit` before marking implementation work complete. diff --git a/docs/specs/hurl-smoke-tests.md b/docs/specs/hurl-smoke-tests.md new file mode 100644 index 0000000..29f7329 --- /dev/null +++ b/docs/specs/hurl-smoke-tests.md @@ -0,0 +1,86 @@ +--- +title: Hurl Smoke Tests +updated: 2026-05-07 +--- + +Smoke tests for Tempest are Hurl files under `test/smoke/*.hurl`. They exercise the running Phoenix server over HTTP and must fail the build when an endpoint regresses. + +## Layout + +```text +test/smoke/ + 00-foundation.hurl + 01-xrpc-shell.hurl + 02-accounts-sessions.hurl + tempest_basic.hurl + tempest_compat.hurl +``` + +## Run Commands + +Single file: + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/01-xrpc-shell.hurl +``` + +Full suite: + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/ +``` + +Use `--jobs 1` when files create shared accounts or when firehose order matters. + +## File Pattern + +```hurl +GET {{base_url}}/xrpc/_health +HTTP 200 +[Asserts] +header "content-type" contains "application/json" +jsonpath "$.status" exists + +GET {{base_url}}/xrpc/com.atproto.server.describeServer +HTTP 200 +[Asserts] +header "content-type" contains "application/json" +jsonpath "$" exists +``` + +## Chained Auth Pattern + +```hurl +POST {{base_url}}/xrpc/com.atproto.server.createSession +Content-Type: application/json +{ + "identifier": "alice.test", + "password": "correct horse battery staple" +} +HTTP 200 +[Captures] +access_jwt: jsonpath "$.accessJwt" +[Asserts] +jsonpath "$.did" exists + +GET {{base_url}}/xrpc/com.atproto.server.getSession +Authorization: Bearer {{access_jwt}} +HTTP 200 +[Asserts] +jsonpath "$.handle" == "alice.test" +``` + +## Rules + +- Use `{{base_url}}`; do not hard-code localhost into Hurl files. +- Capture tokens and DIDs instead of asking shell scripts to parse JSON. +- Assert status, content type, and protocol-required fields. +- Keep destructive or account-creating suites sequential with `--jobs 1`. +- Store fixture payloads in `test/fixtures/` when request bodies become large. + +## Sources + +- +- +- +- diff --git a/docs/specs/interop-testing.md b/docs/specs/interop-testing.md index d640566..1035f78 100644 --- a/docs/specs/interop-testing.md +++ b/docs/specs/interop-testing.md @@ -3,8 +3,6 @@ title: Interop and Integration Testing updated: 2026-05-07 --- -# Interop and Integration Testing - Protocol work is not done until it passes black-box tests. Unit tests protect parsers and binary code, but every milestone needs an HTTP test that exercises the running Phoenix server. ## Test Layers @@ -12,13 +10,13 @@ Protocol work is not done until it passes black-box tests. Unit tests protect pa 1. Parser and golden tests for repo-core. 2. Context tests for account, identity, record, blob, and sync flows. 3. Phoenix integration tests using `ConnCase`. -4. Running-server smoke tests with `curl` and `http`. +4. Running-server smoke tests with Hurl. 5. External fixture tests against official atproto fixtures. 6. Client compatibility tests against known SDKs or `goat` where useful. -## Smoke Test Script +## Smoke Tests -Create `script/smoke/tempest_basic.sh` once Milestone 02 lands. It should: +Create `test/smoke/tempest_basic.hurl` once Milestone 02 lands. It should: 1. Hit health. 2. Call `describeServer`. @@ -33,6 +31,9 @@ Create `script/smoke/tempest_basic.sh` once Milestone 02 lands. It should: ## Integration Test Rules +- Keep smoke tests in `test/smoke/*.hurl`. +- Run smoke tests with `hurl --test`. +- Pass environment-specific values with Hurl variables, for example `--variable base_url=http://localhost:4000`. - Prefer selectors and structured response assertions over raw HTML. - Use `start_supervised!/1` for test processes. - Avoid `Process.sleep/1`; use monitors or `:sys.get_state/1`. @@ -52,14 +53,14 @@ Create `script/smoke/tempest_basic.sh` once Milestone 02 lands. It should: ## HTTP Verification ```bash -script/smoke/tempest_basic.sh http://localhost:4000 +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/tempest_basic.hurl ``` Expected: -- The script exits non-zero on any failed HTTP check. -- The script prints each endpoint and status. -- The script leaves enough IDs in output to debug failures. +- Hurl exits non-zero on any failed HTTP check. +- Captures expose enough IDs to debug failures. +- Assertions cover status, content type, and required JSON fields. ## Sources @@ -67,3 +68,5 @@ Expected: - - - +- +- diff --git a/docs/tasks.md b/docs/tasks.md index 2ea1c92..dda22ec 100644 --- a/docs/tasks.md +++ b/docs/tasks.md @@ -21,6 +21,6 @@ Milestone task files live in `docs/tasks/`. A milestone is done only when: - all listed tasks are complete; -- the milestone HTTP verification passes against a running server; +- the milestone Hurl smoke test passes against a running server; - integration tests cover the main path and one failure path; - `mix precommit` passes. diff --git a/docs/tasks/00-foundation.md b/docs/tasks/00-foundation.md index 1fd8844..7111d2a 100644 --- a/docs/tasks/00-foundation.md +++ b/docs/tasks/00-foundation.md @@ -15,8 +15,8 @@ Goal: make the project ready for PDS implementation without changing protocol be - [ ] T00-04: Include app version and boot status in health output. - [ ] T00-05: Add test coverage for health success. - [ ] T00-06: Add test coverage for invalid config refusing to boot. -- [ ] T00-07: Document local server startup and smoke-test commands. -- [ ] T00-08: Add a `script/smoke` directory placeholder with README. +- [ ] T00-07: Document local server startup and Hurl smoke-test commands. +- [ ] T00-08: Add a `test/smoke` directory placeholder with README. ## Integration Tests @@ -27,8 +27,7 @@ Goal: make the project ready for PDS implementation without changing protocol be ## HTTP Verification ```bash -curl -fsS http://localhost:4000/xrpc/_health -http GET :4000/xrpc/_health +hurl --test --variable base_url=http://localhost:4000 test/smoke/health.hurl ``` Expected JSON fields: diff --git a/docs/tasks/01-xrpc-shell.md b/docs/tasks/01-xrpc-shell.md index b860aff..91d4572 100644 --- a/docs/tasks/01-xrpc-shell.md +++ b/docs/tasks/01-xrpc-shell.md @@ -17,7 +17,7 @@ Goal: expose the XRPC routing, method registry, and protocol-shaped JSON errors. - [ ] T01-07: Add controller tests for unknown method JSON error. - [ ] T01-08: Add controller tests for wrong HTTP verb. - [ ] T01-09: Add response content-type assertions. -- [ ] T01-10: Add smoke script entries for health and describeServer. +- [ ] T01-10: Add `test/smoke/01-xrpc-shell.hurl` entries for health and `describeServer`. ## Integration Tests @@ -28,9 +28,7 @@ Goal: expose the XRPC routing, method registry, and protocol-shaped JSON errors. ## HTTP Verification ```bash -http GET :4000/xrpc/com.atproto.server.describeServer -http POST :4000/xrpc/com.atproto.server.describeServer -curl -i http://localhost:4000/xrpc/com.atproto.unknown.method +hurl --test --variable base_url=http://localhost:4000 test/smoke/xrpc.hurl ``` Expected: diff --git a/docs/tasks/02-accounts-sessions.md b/docs/tasks/02-accounts-sessions.md index 85b0c08..01dcb6e 100644 --- a/docs/tasks/02-accounts-sessions.md +++ b/docs/tasks/02-accounts-sessions.md @@ -34,16 +34,7 @@ Goal: create local accounts and authenticate XRPC calls. ## HTTP Verification ```bash -http POST :4000/xrpc/com.atproto.server.createAccount \ - handle=alice.test email=alice@example.com password='correct horse battery staple' - -http POST :4000/xrpc/com.atproto.server.createSession \ - identifier=alice.test password='correct horse battery staple' - -TOKEN="$(http --body POST :4000/xrpc/com.atproto.server.createSession \ - identifier=alice.test password='correct horse battery staple' | jq -r .accessJwt)" - -http GET :4000/xrpc/com.atproto.server.getSession "Authorization:Bearer $TOKEN" +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/accounts.hurl ``` ## Done diff --git a/docs/tasks/03-identity-handles.md b/docs/tasks/03-identity-handles.md index 4a41321..c7df8b8 100644 --- a/docs/tasks/03-identity-handles.md +++ b/docs/tasks/03-identity-handles.md @@ -32,10 +32,7 @@ Goal: give local accounts resolvable DID and handle metadata. ## HTTP Verification ```bash -http GET :4000/.well-known/atproto-did Host:alice.test -http GET :4000/xrpc/com.atproto.identity.resolveHandle handle==alice.test -http POST :4000/xrpc/com.atproto.identity.updateHandle \ - "Authorization:Bearer $TOKEN" handle=alice.test +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/identity.hurl ``` ## Done diff --git a/docs/tasks/04-repo-core.md b/docs/tasks/04-repo-core.md index 556dd08..56df936 100644 --- a/docs/tasks/04-repo-core.md +++ b/docs/tasks/04-repo-core.md @@ -36,10 +36,7 @@ Goal: build and prove the binary repository primitives before exposing writes br This milestone has no public write endpoint yet. Add a temporary test-only route or skip public exposure until Milestone 05, but the first user-visible verification must be: ```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.actor.profile rkey=self \ - record:='{"$type":"app.bsky.actor.profile","displayName":"Alice"}' +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/repo.hurl ``` Expected by Milestone 05: diff --git a/docs/tasks/05-record-apis.md b/docs/tasks/05-record-apis.md index 1f4d1e1..f087082 100644 --- a/docs/tasks/05-record-apis.md +++ b/docs/tasks/05-record-apis.md @@ -36,16 +36,7 @@ Goal: persist records in per-account repositories and expose repository XRPC rea ## HTTP Verification ```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.actor.profile rkey=self \ - record:='{"$type":"app.bsky.actor.profile","displayName":"Alice"}' - -http GET :4000/xrpc/com.atproto.repo.getRecord \ - repo==alice.test collection==app.bsky.actor.profile rkey==self - -http GET :4000/xrpc/com.atproto.repo.listRecords \ - repo==alice.test collection==app.bsky.actor.profile +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/records.hurl ``` ## Done diff --git a/docs/tasks/06-car-sync-reads.md b/docs/tasks/06-car-sync-reads.md index bf7a221..be2560a 100644 --- a/docs/tasks/06-car-sync-reads.md +++ b/docs/tasks/06-car-sync-reads.md @@ -30,12 +30,7 @@ Goal: expose repository state through sync read endpoints. ## HTTP Verification ```bash -curl -fsS -D /tmp/tempest-car.headers -o /tmp/alice.car \ - "http://localhost:4000/xrpc/com.atproto.sync.getRepo?did=did:plc:example" - -http GET :4000/xrpc/com.atproto.sync.getLatestCommit did==did:plc:example -http GET :4000/xrpc/com.atproto.sync.getRepoStatus did==did:plc:example -http GET :4000/xrpc/com.atproto.sync.listRepos +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/car-sync.hurl ``` ## Done diff --git a/docs/tasks/07-firehose.md b/docs/tasks/07-firehose.md index 10acf6b..a1b7413 100644 --- a/docs/tasks/07-firehose.md +++ b/docs/tasks/07-firehose.md @@ -31,19 +31,8 @@ Goal: persist and stream repository, identity, and account events. ## HTTP Verification -Terminal 1: - -```bash -curl --no-buffer "ws://localhost:4000/xrpc/com.atproto.sync.subscribeRepos?cursor=0" -``` - -Terminal 2: - ```bash -http POST :4000/xrpc/com.atproto.repo.createRecord \ - "Authorization:Bearer $TOKEN" \ - repo=alice.test collection=app.bsky.feed.post \ - record:='{"$type":"app.bsky.feed.post","text":"firehose test","createdAt":"2026-05-07T00:00:00Z"}' +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/firehose.hurl ``` Expected: diff --git a/docs/tasks/08-blobs.md b/docs/tasks/08-blobs.md index 989c8f6..92427d3 100644 --- a/docs/tasks/08-blobs.md +++ b/docs/tasks/08-blobs.md @@ -34,16 +34,7 @@ Goal: upload, reference, serve, and garbage collect blobs. ## HTTP Verification ```bash -printf 'hello blob' > /tmp/tempest-blob.txt - -http --form POST :4000/xrpc/com.atproto.repo.uploadBlob \ - "Authorization:Bearer $TOKEN" \ - Content-Type:text/plain < /tmp/tempest-blob.txt - -http GET :4000/xrpc/com.atproto.sync.listBlobs did==did:plc:example - -curl -fsS \ - "http://localhost:4000/xrpc/com.atproto.sync.getBlob?did=did:plc:example&cid=bafk..." +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/blobs.hurl ``` ## Done diff --git a/docs/tasks/09-admin-deployment.md b/docs/tasks/09-admin-deployment.md index 76cbe33..ae9d6e0 100644 --- a/docs/tasks/09-admin-deployment.md +++ b/docs/tasks/09-admin-deployment.md @@ -22,7 +22,7 @@ Goal: make Tempest deployable and maintainable as a single-node self-hosted PDS. - [ ] T09-11: Add `mix pds.sequencer.status`. - [ ] T09-12: Add `mix pds.blob.gc`. - [ ] T09-13: Add backup create/restore docs. -- [ ] T09-14: Add smoke script for deployed HTTPS target. +- [ ] T09-14: Add Hurl smoke test for deployed HTTPS target. - [ ] T09-15: Add telemetry events for XRPC, repo writes, blobs, and firehose. ## Integration Tests @@ -35,9 +35,10 @@ Goal: make Tempest deployable and maintainable as a single-node self-hosted PDS. ## HTTP Verification ```bash -curl -fsS https://tempest.example.com/xrpc/_health -http GET https://tempest.example.com/xrpc/_admin/status "Authorization:Bearer $ADMIN_TOKEN" -curl --no-buffer "wss://tempest.example.com/xrpc/com.atproto.sync.subscribeRepos?cursor=0" +hurl --test --jobs 1 \ + --variable base_url=https://tempest.example.com \ + --variable admin_token="$ADMIN_TOKEN" \ + test/smoke/deployment.hurl ``` ## Done diff --git a/docs/tasks/10-compatibility-hardening.md b/docs/tasks/10-compatibility-hardening.md index bf589bb..ab9307f 100644 --- a/docs/tasks/10-compatibility-hardening.md +++ b/docs/tasks/10-compatibility-hardening.md @@ -18,7 +18,7 @@ Goal: close protocol gaps and verify behavior against external clients, fixtures - [ ] T10-08: Add account lifecycle endpoints needed for deactivate/delete/takedown. - [ ] T10-09: Add rate limits for auth, record writes, blob uploads, and identity lookups. - [ ] T10-10: Add interop fixture test suite. -- [ ] T10-11: Add SDK compatibility smoke tests. +- [ ] T10-11: Add Hurl compatibility smoke tests. - [ ] T10-12: Add migration/import tests. - [ ] T10-13: Add abuse cases for oversized records, deep CBOR, and invalid CIDs. - [ ] T10-14: Add external relay/AppView verification notes. @@ -33,13 +33,13 @@ Goal: close protocol gaps and verify behavior against external clients, fixtures ## HTTP Verification ```bash -script/smoke/tempest_basic.sh http://localhost:4000 -script/smoke/tempest_compat.sh http://localhost:4000 +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/tempest_basic.hurl +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/tempest_compat.hurl ``` Expected: -- Both smoke scripts exit successfully. +- Both Hurl smoke tests exit successfully. - Output includes account DID, latest commit, exported CAR size, blob CID, and observed firehose seq. ## Done diff --git a/docs/tasks/README.md b/docs/tasks/README.md index 8d0efd6..cc998bf 100644 --- a/docs/tasks/README.md +++ b/docs/tasks/README.md @@ -20,7 +20,25 @@ Every task should leave one of these behind: - a passing unit test; - a passing Phoenix integration test; -- a passing running-server HTTP check; +- a passing running-server Hurl smoke test; - a small doc update that explains a verified behavior. -Milestone-level HTTP verification is mandatory. +Milestone-level Hurl verification is mandatory. + +## Hurl Rule + +Smoke tests are Hurl files under `test/smoke/*.hurl`. + +Run a single milestone: + +```bash +hurl --test --variable base_url=http://localhost:4000 test/smoke/01-xrpc-shell.hurl +``` + +Run all smoke tests: + +```bash +hurl --test --jobs 1 --variable base_url=http://localhost:4000 test/smoke/ +``` + +Use `--jobs 1` for smoke suites that create shared accounts or depend on event order.