operations / edge gotchas #

Operational notes that aren't obvious from the code, because the relevant config lives in Cloudflare/Fly dashboards, not the repo.

Cloudflare: Browser Integrity Check is OFF for the public API (/xrpc/) #

The drop-in search API must accept any HTTP client (scripts, server-side consumers, not just browsers) — that's the whole point of being a Bluesky typeahead replacement, and it matches bsky's public-API posture.

Cloudflare's Browser Integrity Check (BIC) was on for the waow.tech zone and returned HTTP 403, body error code: 1010 ("banned based on browser signature") to non-browser fingerprints — e.g. Python's default urllib client. curl and the Zig ingester happened to pass, so it looked fine until tested with a plain script.

Fix (live): a Cloudflare Configuration Rule named xrpc allow (zone waow.tech, id 8c32e078cab2a47dcaeaf5284ca0aa54, phase http_config_settings):

• Match: (http.host eq "typeahead.waow.tech" and starts_with(http.request.uri.path, "/xrpc/"))
• Action: set Browser Integrity Check → Off

Scoped on purpose: BIC stays on for the HTML pages (/, /stats, /docs) and for /admin/* (which is also Bearer-token auth'd). Only the public, read-only /xrpc/* API is opened.

If this rule is deleted or its toggle flips back to On, every non-browser API client breaks with 403 / error code: 1010. Verify with a bare client:

python3 -c "import urllib.request,json; print(json.load(urllib.request.urlopen('https://typeahead.waow.tech/xrpc/app.bsky.actor.searchActorsTypeahead?q=nate&limit=2'))['actors'][0]['handle'])"

Expect a handle (200), not an HTTPError 403.

Note: the cloudflare-api MCP / OAuth token cannot read or write this zone's settings or rulesets (returns 9109 / 10000 / "request is not authorized") even with full-access scopes selected — this rule must be managed in the dashboard (Rules → Configuration Rules).

For volume/abuse control use a Rate Limiting Rule on /xrpc/* (requests/min per IP) — BIC only fingerprints, it does not bound volume.

Cloudflare: Browser Cache TTL must "Respect Existing Headers" #

The worker sets Cache-Control: public, max-age=60 on search responses (src/handlers/search.ts) so the edge + browser cache results for ~1 min and moderation/profile/deletion changes go live within that window.

The waow.tech zone's Browser Cache TTL setting was 14400 (4 hours, a legacy CF default), which overrides the worker's max-age — the served header became max-age=14400 and stale results persisted on the edge per-POP for up to 4h. Symptom: a just-applied moderation change (e.g. a domain block) keeps showing in the UI long after Turso + the search service are correct; curl of a cache-busted URL returns the fresh value but the cached query URL does not.

Fix (live): Browser Cache TTL set to 0 = "Respect Existing Headers" (Caching → Configuration), so the worker's max-age=60 is honored. After changing it, purge the cache to clear the inflated entries.

Unlike the BIC rule above, this is doable via API with a properly-scoped token (the MCP's own token still can't). Mint a zone-scoped token for waow.tech with permission groups Cache Purge, Cache Settings : Edit, Zone Settings : Edit, put it in gitignored .env as CF_CACHE_TOKEN — not CF_API_TOKEN/CLOUDFLARE_API_TOKEN, which bunx auto-loads from .env and wrangler uses for deploy (a cache-scoped token under that name breaks bunx wrangler deploy). Then:

set -a; . ./.env; set +a
ZONE=8c32e078cab2a47dcaeaf5284ca0aa54
# inspect / fix the setting
curl -sS "https://api.cloudflare.com/client/v4/zones/$ZONE/settings/browser_cache_ttl" -H "Authorization: Bearer $CF_CACHE_TOKEN"
curl -sS -X PATCH "https://api.cloudflare.com/client/v4/zones/$ZONE/settings/browser_cache_ttl" -H "Authorization: Bearer $CF_CACHE_TOKEN" -H "Content-Type: application/json" --data '{"value":0}'
# purge stale entries
curl -sS -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE/purge_cache" -H "Authorization: Bearer $CF_CACHE_TOKEN" -H "Content-Type: application/json" --data '{"purge_everything":true}'

Verify with curl -D- of an /xrpc/ query: cache-control should read max-age=60, not 14400. (Cache Purge is a Zone permission — it doesn't appear if the token's permission dropdown is scoped to Account.)

Snapshot builder: R2 channels (don't clobber prod latest.json) #

The indexer (MODE=indexer) writes the snapshot + manifest to R2 and a pointer the search service polls. Production search polls latest.json. A scratch/staging build must NOT overwrite it, or prod will poll an incompatible build and reject it forever (masking real promotions).

INDEX_CHANNEL picks the publish target and defaults to local (no upload), so a build that forgets to declare a channel — or a one-off machine launched without per-run env — can never clobber prod. Read by services/src/index/builder.zig:

INDEX_CHANNEL	publishes to	also requires	use
local (default)	nothing — build stays on the volume	—	pure inspection; read counts in the build dir
staging	staging/<name>/<id>/ + latest.<name>.json	INDEX_STAGING_NAME (non-empty)	a scratch search instance can poll the namespaced latest
prod	builds/<id>/ + latest.json	INDEX_ALLOW_PROD=1	the prod promote — deliberate, separate go/no-go

• Inspection: omit INDEX_CHANNEL (or set local) → manifest + index.db land in the build dir; nothing is uploaded.
• Staging: just indexer-build-staging <branch> → INDEX_CHANNEL=staging INDEX_STAGING_NAME=<branch>. Prod's latest.json is untouched.
• Production promote: INDEX_CHANNEL=prod INDEX_ALLOW_PROD=1. Declaring prod alone is not enough — the arming flag is the second key. Without it the build refuses to publish and stays on disk for a retry.

Recurring snapshot rebuilds (self-sustaining, 2026-06-15) #

The overlay stays bounded only if the snapshot is rebuilt routinely (a 19-day gap let prefix_overlay bloat to 7.9M rows → q=bsky 210s). This is now a fly daily-scheduled machine, not a manual chore:

# build+push the current image
cd services && fly deploy -c fly.indexer.toml --build-only --push   # prints image ref

# one-time volume (≥4× snapshot; ~8.6GB snapshot → 50GB)
fly volumes create typeahead_indexer_data --size 50 --region iad -a typeahead-indexer -y

# daily scheduled prod build+promote (machine 'index-rebuild-daily')
fly machine run <image-ref> -a typeahead-indexer \
  --schedule daily --volume typeahead_indexer_data:/data \
  --vm-memory 2048 --vm-cpus 4 --vm-cpu-kind shared --region iad \
  --env MODE=indexer --env INDEX_CHANNEL=prod --env INDEX_ALLOW_PROD=1 --env INDEX_BUILD_ROOT=/data \
  --env INDEX_SAMPLE_LIMIT=0   # CRITICAL: 0 = full corpus. omitting it defaults to a 1000-actor
                               # SAMPLE build, which ActorCountFloor correctly rejects (never promotes).

App B (typeahead-search, ENABLE_INDEX_PROMOTE=1) polls latest.json and auto-adopts the new build — so scheduled build + existing auto-promote = the snapshot self-refreshes daily and the overlay stays ~1 day small. To run an off-cycle rebuild now: fly machine start <id of index-rebuild-daily>. Do NOT fly deploy this app as a service (loop-rebuilds); the scheduled machine is the only thing that should run it. Remaining defense-in-depth (not yet done): cap the overlay read in overlayPrefixRows, and alert on snapshot age / overlay size.

service identity & lexicons (2026-07-25) #

The service has its own atproto account: @typeahead.waow.tech = did:plc:y6kywgnb7xletjpa5hhxmd4k, currently hosted on the bsky PDS (migratable later; only the DID doc's service entry changes).

• handle verification: _atproto.typeahead.waow.tech TXT (CF zone waow.tech) + the worker serving /.well-known/atproto-did from the SERVICE_DID wrangler var.
• lexicon authority: _lexicon.typeahead.waow.tech TXT → the service DID; schema records live in its repo under com.atproto.lexicon.schema (rkey = NSID).
• to edit/add lexicons: change lexicons/*.json, then ATP_PDS=https://bsky.social ATP_IDENTIFIER=typeahead.waow.tech ATP_PASSWORD=$ATPROTO_PASSWORD uv run scripts/publish-lexicon.py (ATPROTO_PASSWORD is in the gitignored .env; source it, don't read it).
• endpoint changes must update /docs, /llms.txt, README.md, and the homepage API chip together; smoke asserts canonical/alias equivalence.

freshness: what to check when someone says "it's not showing up" (2026-07-31) #

GET /health/freshness — public, no token, 503 when anything is wrong:

{"ok":true,"stale":false,"replica_lag_s":37,"snapshot_age_s":19357,
 "build_id":"...","probe_ok":true,"reasons":[]}

It reports overlay lag (replica_lag_s, the field name predates the replica's removal), snapshot age, and a live query through the real serving path. Check it before Logfire or fly logs — it needs neither, which matters because the search service exported no telemetry at all until 2026-07-31.

Thresholds: overlay lag warns at 10 min, stale at 30 min (sync runs every 5 min, edge cache is 60s). Snapshot stale at 4 days against a 3-day rebuild cadence.

The three layers and how to check each:

layer	check	healthy
relay → Turso	fly logs -a typeahead-ingester --no-tail	+N actors ... pending: 0, cursor advancing
Turso → overlay	/health/freshness → replica_lag_s	< 600s
snapshot	/index/status → build_id	age < 3 days

If the overlay is behind and no sync.* spans are arriving, the sync loop is wedged: the watchdog exits the process within 20 min so fly restarts it. A manual fly machine restart -a typeahead-search is safe — on boot the service re-adopts the on-disk snapshot and resumes projecting from the persisted watermark (sync_meta.last_sync, advanced per page); the worker's backstop covers the seconds in between.

how a request-indexing lands (2026-07-31) #

Measured 1.3–6.4s end to end, of which 80–86% is the single-row Turso UPSERT contending with the live ingester. Searchable ~0.16s after that write returns.

The worker POSTs the row it just wrote to the search service's /admin/refresh, which projects it into the overlay immediately — the overlay being the design's live-update channel. It sends the row, not a DID: an earlier version had the search service re-read the row from Turso, which put the fast path through the most contended component in the system (measured: 28s for an indexed point lookup while a bulk job ran).

Fire-and-forget. The 5-minute sync remains the mechanism of record, so a failed push costs freshness for one actor, never correctness.

Under firehose floods that write can queue behind the ingester's batch transactions — Turso is single-writer. Measured turso_write;dur=10178 (2026-08-04), with spikes past 45s; the smoke gate allows 60s for this check, which is calibration, not a fix. The write stays synchronous on purpose: the response says "indexed", and that has to be true (59f6e39). Moving it off the response path reintroduces the untruthful-"indexed" failure mode; splitting the response into "accepted, landing now" changes the UX contract. Revisit only if user-facing timeouts actually appear — the endpoint's Server-Timing header attributes the phases, so you'll be able to tell.

the worker cron was being killed every hour (found 2026-09-03) #

Workers Logs, 08-27..09-03: 140 scheduled runs, 120 ended exceededWallTime (Cloudflare's 15-minute limit), 20 completed. CPU per run < 1 s; the rest was waiting on the appview and Turso. Phase order was freshness → snapshot → stats → refreshModeration (~8 min, ~1,000 rows) → enrichActors (identity 5,000 + profile 5,000 DIDs), so the last phase got ~7 minutes of a job sized for 40, and the kill was invisible: not an exception, so cron_step_failed never fired. Partial batches persisted, so it degraded rather than broke, for weeks.

What changed:

• enrichActors is gone from the worker. Handles for new actors are typeahead-identity-hourly on heavypad (:20 past the hour, 6,000 DIDs, 40-minute budget); profiles were already typeahead-enrich-backfill there (daily, 3.5 h, ~215k rows/run). The worker keeps freshness, stats, and the moderation refresh.
• the step loop has a 12-minute budget (CRON_BUDGET_MS): a step only starts with time left, and every run logs cron_done with per-phase milliseconds or cron_budget_exhausted naming what it skipped. Silence is no longer an outcome.
• the backfill's Turso calls retry with backoff (2 of 10 daily runs had died on one ReadTimeout from the shared single writer).

profile sources: Bluesky, then Blacksky's appview, then the PDS (2026-09-03) #

Bluesky's appview answers AccountTakedown for accounts it has suspended and serves nothing about them. Blacksky's appview (api.blacksky.community, did:web:api.blacksky.community) indexes the whole network, answers without auth, and keeps serving those accounts, counts included; the PDS record has name, avatar and description but no counts. Enrichment (cron + request-indexing) now goes Bluesky → Blacksky → PDS and writes which one won into actors.profile_source (bsky | blacksky | pds; '' = written before 2026-09-03).

Why the column exists: Blacksky's follower counts run lower than Bluesky's for the same account (2,083 vs 2,531 for zzstoatzz.io; 88k vs 130k for jcsalterego) because it indexes a narrower follow graph, so a quality_score is only comparable with its source known. The report shows it under "stored data" and on the ranking line. Policy unchanged: a Bluesky takedown does not hide an account here (the PDS fallback always worked that way); it just no longer zeroes its rank.

the hidden count is sampling, not enforcement (2026-09-20) #

actors.hidden != 0 was 2,310 rows: 1,687 spam, 203 !hide, 410 from the slur regex, domain blocks, or overrides, and 10 !takedown. Read against Bluesky's daily enforcement volume that looks broken. It is not; it measures something narrower:

• PDS-level suspensions and takedowns reach the relay as #account with active: false. The ingester deletes and tombstones them (handleAccount in services/src/ingest.zig). They never become hidden rows.
• Appview takedowns (getProfiles omits the DID) are re-enriched from Blacksky or the PDS record, whose labels never carry !takedown. By the policy above they stay visible; resolveHidden returns 0.
• Labels only arrive through getProfiles on the refresh cron, which walks the scored-visible pool (~41k rows, 800 per run) plus 200 firehose-touched rows. Nothing subscribes to the labeler stream. A spam label applied to a zero-score account after indexing is not observed until that account posts again.

So the number is a lower bound on labeled rows the cron has revisited. It was dropped from the link-preview card that day and the /stats tile explains itself in its tip. If an enforcement figure is ever wanted, count #account deletes in a counter, or read tombstones (pruned after 7 days); do not present hidden as one. profile_source has no index; grouping on it full-scans 12M rows (a 60 s diagnostic timeout on 2026-09-20).

The card's replacement tile is traffic_sources_total: materializeStats counts traffic_sources rows under TRAFFIC_SOURCE_WHERE (src/metrics.ts, shared with the /stats list) minus the unknown and homepage attribution buckets, and writes it into stats_data. First materialized 07:02 UTC 2026-09-20 at 203. Before that field exists the card counts distinct app domains in traffic_daily, which undercounts (165 that morning) and never overcounts.

/reputation: moderation status across labelers (2026-09-03) #

/reputation?actor=… now answers "what do the moderation services say about this account" next to "why does it rank here". Three facts shaped it:

• the labels typeahead stores are partial by construction. They come from the unauthenticated appview (getProfiles), which reports only self-labels and Bluesky's own moderation service. The page says so next to them (storedLabelsSource).
• the appview cannot be asked for third-party labels without auth. atproto-accept-labelers is ignored on public.api.bsky.app (verified: a Blacksky-labeled account came back label-free however the header was set). So the page asks each labeler directly on its own com.atproto.label.queryLabels endpoint, resolved from the labeler's DID document, 3 s timeout each, in parallel, per report. Nothing is stored; nothing touches the search path.
• the labeler list is discovered, not typed. scripts/discover-labelers.py searches the appview and keeps accounts it flags associated.labeler, filtered to moderation-type names with ≥ 50 subscribers, plus the appview operators' services pinned regardless (Bluesky, Blacksky, Northsky, Eurosky — Eurosky's has 2 subscribers and may not answer). Output is src/labelers.json; rerun the script to refresh, commit the diff.

Negations are resolved client-side (latest record per val wins, neg withdraws, expired labels drop) — applyLabelHistory in src/labelers.ts.

The same change removed the page's two COUNT(*) queries over the visible corpus (the last corpus-proportional read on a request path). The snapshot builder now writes p0..p100 of quality_score into index_meta (score_quantiles), /index/status exposes them, and the percentile is a lookup in 101 numbers. Until the first build after 2026-09-03 promotes, the page says "percentile unavailable".

unattributed bulk clients: the 03:00 UTC spike (2026-09-03) #

The daily spike in the unknown source on /stats is one node script on a Vultr VPS (45.63.115.226, FR) that fires ~200 searchActorsTypeahead requests with limit=100 at 03:00 UTC, every night since 2026-08-17, all Belgian vocabulary (Wallonie, Eupen, chicon, asbl, guindaille…) — someone assembling a Belgian account list by keyword. No X-Client, Origin or Referer, so it cannot be attributed. Found via Workers Logs grouped by cf-connecting-ip + user-agent (the /stats counters have no per-client detail by design).

What it cost: the engine was fine (179 searches, p50 7 ms). The worker was not: a quarter of the burst hit the 60/min per-IP limiter, the rest ran 7–20 s of wall time, and other users' median wall time went from 282 ms to 1.4 s for those two minutes. Two reasons — every cache miss writes a stats row to Turso in waitUntil, and every empty engine answer falls through to a full-corpus Turso FTS query, which rare keywords trigger constantly.

Decision (2026-09-04): the empty-result Turso fallback is OFF unless the worker has TURSO_FALLBACK_ON_EMPTY=1, and even then only for limit ≤ 25. During the 2026-09-04 viral burst (~20x engine volume) the fallback alone put Turso over its connection limit — 1,310 worker requests failed with "Database connections limit exceeded" and 967 more timed out in 90 minutes — and the ingester's /admin/ingest writes failed behind it until its backlog overflowed and dropped events. Serving stayed 2xx throughout; discovery did not. The fallback's reason (a lagging replica) has been gone since the 2026-09-02 replica removal: request-indexing pushes the row into the search box's overlay synchronously. Turso still answers when the engine is unreachable; it no longer answers when the engine says "no such actor".

Same day, second decision: Turso-on-unreachable is also opt-in (TURSO_FALLBACK_ON_UNREACHABLE=1), off by default. Twenty minutes after the first fix the search box panicked and restarted (15 s to adopt its on-disk snapshot). In those 15 s every query went to the full-corpus Turso path, Turso hit its connection limit again (119 errors, 83 timeouts), and the ingester overflowed its backlog again. The engine is one machine; while it is down the worker answers {actors: []} uncached and the next keystroke reaches the restarted engine. Regression test: index.test.ts "an unreachable engine answers empty without touching Turso".

The earlier decision it replaces (2026-09-03): the fallback only ran for limit ≤ 25 (TURSO_FALLBACK_MAX_LIMIT in src/handlers/search.ts). A person typing a handle asks for ~10 rows and keeps the second look for an actor the engine hasn't projected yet; a 100-row keyword sweep takes the engine's answer as final. Nothing changes for browsers or for any client using the default limit; the bot still gets answered. Not chosen: putting unattributed clients on the strict limiter (would break real integrations that never set headers) and leaving it (the fallback is the one corpus-proportional read on the request path, and this let a stranger schedule it). If a client with a legitimate need for limit>25 plus the Turso second look appears, it should identify itself with X-Client and we can allowlist by app.

observability: what the zig services emit (2026-09-02) #

Both fly roles use logfire-zig ≥ 0.3.0, wired the way its README prescribes and which other repos (find-bufo next) should copy:

• pub const std_options = .{ .logFn = logfire.logFn } in main.zig: every std.log.* line ships to logfire as a log record carrying the current span's trace/span id, code.module.name (the std.log scope, e.g. sync), and severity_text — AND still reaches stderr, so fly logs is unchanged. Before this, logs were stderr-only and lost after ~100 lines.
• every span is published as a pending span at start, so a 10-minute sync.incremental pass is visible in the live view from second one.
• the search span is head-sampled 1 in 10 (search_span_sample_rate in index_search.zig), stamped sample_rate=10 so count(*) * sample_rate is still the volume. Decided 2026-09-04: during the viral burst the search box shipped 19k records in one hour, 99% of them this span at p50 3 ms, duplicating what request_buckets already counts. Baseline was ~10–20k records a day; a sustained burst would have been ~14M a month. Searches at or above slow_search_ms (100) log a warning regardless of sampling, so the tail is never lost; errors log regardless too. Volume check: SELECT date_trunc('day', start_timestamp), count(*) FROM records GROUP BY 1.
• span.setStatus(.@"error", …) stamps logfire.level_num = 17; errored spans render red.
• every fatal path goes through services/src/fatal.zig (2026-09-09): fatal.exitForRestart() for both watchdogs, the firehose exit, handler init and cursor failure; pub const panic = std.debug.FullPanic(fatal.panicHook). Each arms a raw-thread deadman BEFORE flushing, then flushes, then exits. The flush ships the last 500 ms of telemetry when the exporter is healthy; the deadman (5 s exit / 3 s panic) guarantees the exit when it is not. otel-zig's forceFlush ignores its timeout at its lock and condvar waits, so a panic on the export thread otherwise hangs the process forever — it did so on 09-03 and again on 09-09 (see docs/retros/). A deadman firing prints fatal: telemetry flush did not finish within N ms to stderr.
• resource: service.name = fly app, service.version = FLY_IMAGE_REF (the deployed image), deployment.environment.name = production.

Query shape: logs are records rows with kind = 'log'; join to spans on trace_id. Pending spans have attributes->>'logfire.span_type' = 'pending_span' and are replaced in the UI by the real span; filter them out of counts.

alerting: overlaps and gaps (reviewed 2026-08-04) #

Four systems page independently — logfire freshness, the typeahead freshness bot, prefect crash automation, pub-search watchdog. Known state:

• overlapping: logfire and the typeahead bot both page on replica staleness, so one condition produces two pages.
• uncovered: enrichment queue depth, and the smoke gate's page-script check failing outside a deploy.

Fixed already: pub-search freshness threshold 30m→60m (overnight publish gaps), cancel-≠-crash orchestration, the typeahead sync-stall class. The principle to hold when adding one: one page per signal, and every page actionable — add coverage only where a firing would change what you do.