courierbench #

A black-box conformance suite for the Bluesky Push Notification Service spec, built to be used as a coding benchmark.

A candidate implementation is launched as a subprocess and driven through fake infrastructure — a DID directory, a PDS with a full atproto OAuth authorization server, appview and chat services, and Web Push and APNs endpoints. Nothing touches a live service, and no network egress is possible.

75 core cases across seven weighted tiers, plus nine bonus cases. The reference implementation passes all of them in about two minutes.

Why this task #

It was picked to discriminate between models that are close on cheaper benchmarks:

• Cryptography with no partial credit. Web Push payloads are decrypted with the subscription's private key (RFC 8291 key derivation into an RFC 8188 aes128gcm record). VAPID headers, DPoP proofs, and private_key_jwt client assertions are signature-verified. Output either derives to the right key or it does not.
• A stateful, long-horizon service. Grant tiers, dedupe-by-URI, cursor advancement, count-decrease resync, grace-window collection, invalid_grant teardown, dead-subscription pruning, exponential backoff, adaptive cadence. Mistakes compound.
• Adversarial negatives. Replayed service-auth tokens, grants belonging to the wrong account, and a counts-tier chat push that must not contain message text — checked by decrypting the push and looking.

No implementation of this spec exists publicly; the two companion wiki pages are designs, not code. There is nothing to memorize.

Layout #

spec/          the specification and its two companion design documents
CONTRACT.md    the launch interface a candidate is given
TASK.md        the benchmark prompt
crates/
  cb-crypto    ES256 JWT, RFC 8291/8188 decryption, VAPID, P-256, Multikey
  cb-fakes     DID directory, PDS + OAuth AS, appview, chat, Web Push, APNs
  cb-harness   subprocess lifecycle, the test world, the simulated client
  cb-suite     the tiers, and the `score` binary
reference/     a reference implementation, excluded from the bench checkout
scripts/       make-bench-checkout.sh

cb-crypto and reference/courier deliberately share no code: the harness verifies with one implementation and the reference produces with another, so a shared bug cannot verify itself.

Running it #

Grade the in-tree reference (the calibration run — expect 100% core):

cargo run -p cb-suite --bin score

Grade a candidate:

cargo run -p cb-suite --bin score -- \
  --courier-cmd "/path/to/candidate/target/release/courier" \
  --out report.json

Develop the suite:

cargo test -p cb-suite              # one test per tier
CB_FILTER=T4.05 cargo test -p cb-suite   # a single case
CB_KEEP_SCRATCH=1 ...               # keep each case's DATA_DIR for inspection

Producing a benchmark checkout #

bash scripts/make-bench-checkout.sh /tmp/bench

Emits the spec, the contract, the task, and a compiling skeleton crate — no tests, no fakes, no reference. Point the scorer at whatever the model builds there.

The report #

--out writes per-tier JSON, which is the point: a single number tells you nothing if a task bottoms out, and the tier profile is what distinguishes "got the plumbing, failed the crypto" from "never started".

{
  "schema": 1,
  "tiers": [
    { "id": "T4", "name": "Web Push crypto", "weight": 20,
      "passed": 9, "total": 12,
      "cases": [ { "id": "T4.05", "status": "fail", "detail": "…" } ] }
  ],
  "core_score": 0.61,
  "bonus_score": 0.0
}

Tier weights: Discovery 4, Registration 14, Auth handoff 16, Poll cycle 18, Web Push crypto 20, Lifecycle 18, Durability 10. APNs is scored separately.

Calibration #

Two properties matter more than the tests themselves, and both are checked mechanically.

The suite is passable. The reference scores 100% of the core and 100% of the bonus. Any assertion it cannot satisfy is a suite bug, not a hard test.

Cases fail for the reason they name. scripts/calibrate.sh asks the reference to break in one specific way (via a CB_MUTATE hook that exists only in the reference) and checks that exactly the expected case ids go red — no more, no fewer. A case that stays green under its own defect is not testing what it claims; a case that goes red under an unrelated one will misattribute a candidate's failure.

bash scripts/calibrate.sh              # all 12 defects
bash scripts/calibrate.sh fixed-salt   # just one

Defect	Caught by
no-cors	T0.05
no-aud	T1.04
no-lxm	T1.05, T1.09
no-signature	T1.06
no-expiry	T1.07
no-sub-check	T2.12
always-log-scope	T2.05
dedupe-by-count	T3.05
fixed-salt	T4.04, T4.05
no-prune	T5.01, T5.02
flat-cadence	T5.11
plaintext-tokens	T6.04, T6.05

Note what dedupe-by-count shows: only one case distinguishes a URI ledger from a count delta, because every other path is gated behind a count rise that does not recur. Without T3.05 the tier would not catch that defect at all.

Flakiness. Before trusting a comparison between models, run the suite repeatedly against the reference and confirm zero failures. A flake rate of a few percent is indistinguishable from the effect most people are trying to measure. As committed, 12 consecutive full runs produced 0 failures out of 984 case executions.

for i in $(seq 1 20); do cargo run -q -p cb-suite --bin score >/dev/null || echo "FLAKE on run $i"; done

Beyond the benchmark contract #

reference/courier was deployed for real (see impro PR #39, and the live instance it links) — a handful of things the fake infrastructure doesn't exercise, and CONTRACT.md doesn't require, surfaced doing that. None of it touches calibration: cargo run -p cb-suite --bin score still scores 100% core / 100% bonus with these changes in place.

• secp256k1 signing keys. CONTRACT.md's fake world states "all keys... are P-256" (§3), and the reference originally hard-rejected any other key type resolved from a DID document. Real Bluesky accounts overwhelmingly use secp256k1 as their #atproto signing key — the long-standing default — so verifying a real service-auth JWT needs both curves. crypto/key.rs's ServiceAuthKey and crypto/jwt.rs's verify_service_auth add ES256K alongside the existing ES256 (P-256) path, used only for verifying callers' tokens; courier's own OAuth/VAPID keys are still always P-256.
• Two-step OAuth authorization-server discovery. The fake PDS serves /.well-known/oauth-authorization-server directly on its own origin (CONTRACT.md §5); real Bluesky-hosted PDSes don't — the actual authorization server is a separate "entryway" origin (e.g. bsky.social), discovered via RFC 9728 /.well-known/oauth-protected-resource. oauth.rs's resolve_authorization_server does that lookup first, falling back to treating the PDS as its own AS when the endpoint isn't there — which is exactly what keeps the fake-PDS calibration path unchanged.
• scope in client metadata. Real authorization servers (bsky.social included) validate that every PAR-requested scope is a member of the client metadata document's own scope field and reject the request otherwise; the fake AS doesn't enforce this. api.rs's client_metadata now declares the union of both grant tiers.
• A real TLS backend. The workspace's reqwest dependency built with no TLS backend at all (default-features = false, features = ["json"]) — fine for the fake infra's http://127.0.0.1, but every outbound HTTPS call (PLC directory, appview, chat, the real PDS, push services) fails without rustls-tls added.
• Seeding poll baselines at grant time. A brand-new grant on an account with existing unread activity would otherwise see the very first poll as a jump from 0, delivering the entire backlog (up to 50 items) as a flood of pushes for things that predate registration. handoff.rs's complete now seeds last_notif_count/last_chat_count from the account's actual counts at grant time instead of hardcoding 0 — two extra XRPC calls during the OAuth callback that the fake harness tolerates fine, but worth knowing if you're writing calibration assertions that count exact request sequences.
• Capping a notification burst. Delivering dozens of uncollapsed notifications at once is what browsers' abusive-notification heuristics are built to catch, and the cost of tripping one is the origin's push permission — every future notification, not just the noisy ones. Above poll::MAX_INDIVIDUAL_PUSHES new items in a cycle, deliver_notifications sends one summary instead, counting each kind of event and carrying the most recent of each (copy::for_notification_burst) rather than a bare total. Chat previews are deliberately not capped — the user asked for message content, and each push already collapses per conversation, so the stacking problem the cap exists for does not arise there. The suite has no case with a burst that large, so calibration is unaffected.
• The real shape of chat.bsky.convo.getUnreadCounts. CONTRACT.md §6 has the fake answer {"count": N}; the real method answers {unreadAcceptedConvos, unreadRequestConvos}. Reading only count against the real service fails silently — the call succeeds, the poll succeeds, failure_count stays 0, and the total is simply always zero, so no chat notification is ever delivered and nothing anywhere says why. This one shipped to the live deployment and went unnoticed until someone asked why they'd never seen a chat push. poll::unread_chat_count reads either shape.
• messageView.sender has no handle. The fake's chat log carries sender: {did, handle}; the real chat.bsky.convo.defs#messageView carries only sender.did, so a previews-tier push built from the message alone attributes every message to "Someone". poll::sender_handle falls back to resolving the DID's alsoKnownAs through the (already caching) directory resolver.
• Empty response bodies. app.bsky.notification.registerPush / unregisterPush have no defined lexicon output, and some real PDSes proxy back a genuinely empty body on success rather than whatever the downstream service returned. Not a courier-side change (its own {} response is unaffected) — worth knowing if you're writing a client against this contract, since a client that unconditionally parses a JSON response will throw on that empty body.
• Chat baseline seeding had the same silent-shape bug. Grant seeding (handoff::complete) originally read only count from getUnreadCounts — the fake's shape. Against the real service that seeds a permanent zero, so the very first poll after a re-authorization would read the current unread total as a rise from 0 and replay the backlog. It now uses poll::unread_chat_count, the same both-shapes reader the poll cycle uses.
• Adaptive cadence needs activity signals, not just timers. Tracking "active" only from device registration means an account whose user is actively reading (but not re-launching the app) falls to the idle cadence and stays there — pushes arrive minutes late while the account is plainly busy. The poll cycle therefore refreshes last_active_at on the two activity signals a polling service can see without inventing new protocol: a delivered notification/chat message, and an unread count going down (the user read something on another client — the polling service's version of updateSeen). The suite's cadence case (T5.11) still passes: activity-refresh keeps active accounts tight, and an account that genuinely goes quiet still degrades to POLL_IDLE_MS.
• Deep links must follow the record kind, not the notification's own URI. The notification's uri is only a navigable post for replies, quotes, mentions, and subscribed-posts. A like or repost carries the URI of the reaction record in the actor's repo — /profile/<actor>/post/<rkey> built from that is a page no client can render. The deeplink has to follow the reaction's subject to the post, attributed to the subject's repo segment (the post author's DID; client routes accept DIDs where they accept handles). A follow has no post at all and links to the follower. This matches how clients link the same notifications in-app (impro uses record.subject.uri for likes and reposts). copy::deeplink encodes the mapping and has unit tests pinning every reason.
• reasonSubject is reason-dependent — read the appview source before trusting it. For plain like/repost it is the reacted-to post (what clients navigate by); for like-via-repost/repost-via-repost it is the reader's own repost through which the reaction arrived — landing there would be worse than the actor fallback. The -via-repost reasons must use record.subject.uri. Verified against the appview's indexing plugins (packages/bsky/src/data-plane/server/indexing/plugins/{like,repost}.ts), which also show the appview suppresses self-notifications — courier does the same for chat (T3.15).
• Chat pushes must be grouped per conversation. Chat pushes collapse per conversation (tag: chat:{convoId}), so one banner per message replaces the previous banner before it is read — a fast exchange surfaces only its last message. Previews-tier delivery now groups fresh messages per conversation: one push quoting the newest five (older ones counted), so the reader can follow the exchange from the notification itself.
• A rolling context buffer, with the reader's own replies. Grouping within one poll cycle is not enough: messages spaced across cycles still overwrite each other. Each previews-tier account keeps a rolling per-conversation buffer of the newest few messages — the subscriber's own replies included, rendered as "You:" — and every fresh push re-quotes it, so the exchange stays followable even though the banner collapses. This is why T3.11 asserts on the cursor the courier sends to getLog rather than on message text reappearing: re-quoting context is legitimate, re-walking the log from the start is not (calibration mutation drop-cursor pins the latter).
• The subscriber's own chat messages are context, not events. The chat log contains everything said in a conversation; the appview never fires a notification for the acting user, so an own message joins the context buffer but never triggers a push by itself (T3.15).
• Observability. /status serves aggregate counters (polls, deliveries, failures, prunes, registrations, grants, timestamps) with no per-account detail — it is unauthenticated on the same origin as the public config document, so it must never enumerate accounts. Per-account events go to stderr (the journal under systemd).
• A failure about the service is not a failure of the grant. The deployment logged a single invalid_client_metadata from the real authorization server: it had failed to fetch the courier's own /oauth/client-metadata.json while refreshing a token. It was transient — the origin's access log shows every other hourly fetch answering 200, and the next poll refreshed fine — but the handling was not. Every failure spent the same per-account budget, and passing MAX_POLL_FAILURES deletes the subscription. That error says nothing about any particular grant and fails for every account simultaneously, so a sustained version of it would have deleted every subscription and forced the whole user base to re-authorize, over a fault none of their accounts had any part in. poll::FailureScope now separates the two: account-scoped failures spend the budget as before, service-scoped ones (invalid_client_metadata, invalid_client, unauthorized_client — deliberately narrow, since a timeout or a 5xx really could be one account's PDS) get a flat retry delay and keep the grant. T5.13 pins it, and calibration mutation drop-on-service-error restores the old behavior to prove the case catches it.