diff --git a/docs/solutions/architecture-patterns/favicon-resolution-negative-cache-divergence.md b/docs/solutions/architecture-patterns/favicon-resolution-negative-cache-divergence.md new file mode 100644 index 0000000..1b23297 --- /dev/null +++ b/docs/solutions/architecture-patterns/favicon-resolution-negative-cache-divergence.md @@ -0,0 +1,134 @@ +--- +title: 'Favicon renders in prod but not local: the negative-cache pin' +date: 2026-07-08 +category: architecture-patterns +module: lib/favicon.ts +problem_type: architecture_pattern +component: service_object +severity: low +related_components: + - "app/components/Favicon" + - "app/routes/api/favicon/[domain].ts" + - "lib/static-favicons.ts" +applies_when: + - "A favicon/logo renders in production but the is hidden on local dev with the same codebase" + - "Adding a durable ecosystem domain whose favicon should always render (bundle it in STATIC_FAVICONS instead of relying on the runtime proxy)" + - "Debugging /api/favicon/[domain] returning 404 for a domain whose origin actually serves a valid favicon" + - "A transient favicon fetch failure (timeout or race with SSR compile) appears stuck for ~24h" +symptoms: + - "cosmik.network and stream.place logos render in prod but the is hidden on local dev" + - "/api/favicon/ returns 404 locally while the origin's /favicon.ico returns 200 with a valid image" + - "getFavicon() run standalone succeeds in under 1s and writes a positive cache row, yet the dev-server proxy still 404s until cache is warmed" +resolution_type: code_fix +tags: [favicon, negative-cache, sqlite-cache, env-divergence, static-favicons, ttl] +--- + +# Favicon renders in prod but not local: the negative-cache pin + +## Context + +Airglow renders app favicons in its automation action-flow through a two-tier resolution system, and the boundary between those tiers is where an app logo can render in one environment but silently vanish in another with the exact same code. + +The two tiers, decided in `app/components/Favicon/index.tsx`: + +```tsx +const file = STATIC_FAVICONS[domain]; +const src = file ? `/static/favicons/${file}` : `/api/favicon/${domain}`; +// — a failed load hides silently +``` + +- **Tier 1 — bundled (static).** `lib/static-favicons.ts` exports `STATIC_FAVICONS: Record`, a small hand-maintained map of favicons vendored into `public/static/favicons/` with content-hashed filenames (`..`, e.g. `semble.so.27ea0d97.svg`). A bundled domain renders unconditionally: no network, no cache, no way to fail. +- **Tier 2 — runtime proxy (dynamic).** Any domain NOT in the map falls through to `/api/favicon/`, served by `app/routes/api/favicon/[domain].ts` which calls `getFavicon(domain)` in `lib/favicon.ts`. + +The crux lives in Tier 2's cache. `lib/favicon.ts` caches every result in the `faviconCache` SQLite table, and it caches FAILURES too: + +- A successful fetch is cached for `TTL_MS = 7 days`. +- A "no favicon found" result is stored as an EMPTY marker (`data = ""`) and pinned for `NEGATIVE_TTL_MS = 1 day`, so a domain that failed once is not re-fetched for 24 hours. +- `fetchFavicon` DNS-resolves the domain to an IP (`resolveSafeIP`, which rejects private IPs to block SSRF/DNS-rebinding), then `tryFetch`es `https:///favicon.ico`, then `/favicon.svg`, then `/favicon.png`, each with `Host`+SNI set to the real domain, `redirect:"error"`, a 3-second `AbortSignal.timeout`, requiring a `content-type` starting with `image/`, non-empty, and <=100KB. + +Because the negative result is a durable, timed row in SQLite, the two environments can diverge on cache state alone. A domain that fetched successfully in prod serves for 7 days; the same domain that hit a single transient fetch failure locally is pinned as empty for a full day. Same code, opposite visible behavior. + +## Guidance + +When an app logo renders in one environment but not another (classically: present in prod at `https://airglow.run/u/...`, absent locally at `http://127.0.0.1:5176/u/...`), diagnose in this order. + +**1. Identify which tier the domain uses.** Look at the emitted HTML. A bundled domain emits `/static/favicons/..`; an unbundled domain emits `/api/favicon/`. If both environments emit the `/api/favicon/...` form, the divergence is NOT code, it is Tier 2 cache state, and you should stop looking at the component. + +**2. Probe the proxy directly.** Request `/api/favicon/` on the failing environment. A `404` confirms the proxy is returning "no favicon" — either a live fetch failure or a cached negative marker. + +**3. Inspect `faviconCache`.** Look for a row with an EMPTY `data` and a recent `fetchedAt`. If `fetchedAt` is within `NEGATIVE_TTL_MS` (1 day), the proxy is not even attempting a re-fetch: it is serving a pinned negative from an earlier failure. + +**4. Confirm the origin is actually healthy.** Fetch `https:///favicon.ico` directly. If it returns `200` with an `image/*` content-type, the origin is fine and the local 404 is purely a stale negative cache. (Note some origins 404 on `/favicon.svg` and `/favicon.png` but serve `/favicon.ico`, which is tried first, so a direct `.ico` check is the meaningful probe.) + +Then choose a fix by durability: + +- **Transient fix — warm the cache.** Delete the negative row(s) and trigger any successful fetch so a POSITIVE row is written. The proxy then returns `200` and the logo renders. This lasts only until the 7-day TTL evicts it or the DB is reset, so it is a debugging convenience, not a fix. +- **Durable fix — bundle into `STATIC_FAVICONS`.** Vendor the favicon under `public/static/favicons/..` and add a map entry in `lib/static-favicons.ts`. The logo now renders via Tier 1, unconditionally, with no dependency on the runtime fetch or the negative cache. This is the same treatment `semble.so` and `atmo.pub` already have, and it is the correct move for any first-class ecosystem app whose logo should always render. + +## Why This Matters + +A stale NEGATIVE cache masquerades as a "missing favicon." There is no error, no log, no broken image: `data-hide-on-error` makes a 404 disappear cleanly, so the failure mode is silence. You will not find a bug in the component because there isn't one. + +The negative cache turns a transient blip into a day-long outage. A single fetch that lost a race — most plausibly a 3-second per-path timeout losing to the heavy first-request SSR module-graph compile that Vite does on demand for the `@atproto` tree on a cold dev server — gets written as an empty marker and pinned for 24 hours by `NEGATIVE_TTL_MS`. The origin can be perfectly healthy the entire time; the proxy never checks again until the TTL expires. + +Environment divergence here is cache state, not code. The instinct on "works in prod, broken locally" is to hunt for a config or code difference. For Tier 2 favicons there is none: prod simply cached a success and local cached a failure. Recognizing that the divergence is data, not logic, saves the entire wrong investigation. + +## When to Apply + +- Any "logo or favicon renders in one environment but not another" report for a domain that is NOT in `STATIC_FAVICONS`. +- A favicon that was working and then silently disappeared for a stretch of time (suspect a negative marker written by a transient timeout, pinned for `NEGATIVE_TTL_MS`). +- When onboarding a new ecosystem app whose logo should always render: add it to `STATIC_FAVICONS` up front rather than relying on the runtime proxy and hoping the cache warms. + +## Examples + +Concrete case: `cosmik.network` and `stream.place` favicons rendered in prod (`https://airglow.run/u/vicwalker.dev.br/3moearshafk22`) but not on the equivalent local page (`http://127.0.0.1:5176/u/alice.pds.dev/3mpgzrh3ho222`). Neither domain is in `STATIC_FAVICONS`, so both environments depended on Tier 2. + +Which tier each domain used, read straight from the local HTML: + +``` +/api/favicon/cosmik.network # Tier 2 — unbundled +/api/favicon/stream.place # Tier 2 — unbundled +/static/favicons/semble.so.27ea0d97.svg # Tier 1 — bundled +``` + +The proxy returned `404` locally for both `/api/favicon/cosmik.network` and `/api/favicon/stream.place`, while direct origin fetches proved the sites were healthy: + +``` +cosmik.network/favicon.ico -> 200 image/x-icon 15086 bytes +stream.place/favicon.ico -> 200 image/png 14351 bytes +# (/favicon.svg and /favicon.png 404 on these origins, but .ico is tried first and wins) +``` + +Inspecting `faviconCache` showed EMPTY markers with a same-day `fetchedAt` for both domains, i.e. inside the 1-day negative TTL, which is why the proxy kept returning 404 without retrying. + +Running the fetch standalone confirmed the origins and the code were both fine: + +```sh +bun -e 'import { getFavicon } from "./lib/favicon"; console.log(await getFavicon("cosmik.network"))' +# succeeded in ~365ms and ~908ms, wrote POSITIVE cache rows +``` + +After that write, the dev-server proxy returned `200` for both and the page rendered the logos. The durable follow-up is the `STATIC_FAVICONS` before/after: + +```ts +// lib/static-favicons.ts — before +export const STATIC_FAVICONS: Record = { + "semble.so": "semble.so.27ea0d97.svg", + "atmo.pub": "atmo.pub..svg", +}; + +// after: vendor public/static/favicons/cosmik.network..ico +// and stream.place..png, then add: +export const STATIC_FAVICONS: Record = { + "semble.so": "semble.so.27ea0d97.svg", + "atmo.pub": "atmo.pub..svg", + "cosmik.network": "cosmik.network..ico", + "stream.place": "stream.place..png", +}; +``` + +Related gotcha from the same debugging session: opening the same `bun:sqlite` dev DB from a SEPARATE `bun -e` process while the HonoX/Vite dev server still held it CRASHED the dev server (connection refused afterward). When inspecting the dev DB, stop the dev server first or work against a read-only copy — do not poke the live DB from a second bun process. + +## Related + +- `docs/solutions/tooling-decisions/satori-og-image-rendering-constraints.md` — complementary. Touches the same `lib/favicon.ts` (`getFavicon`) and `lib/static-favicons.ts` loader from the OG-image rendering angle (ICO drop, favicon-module mocking in tests), but says nothing about resolution order, the `/api/favicon` proxy, the `faviconCache` table, or the positive/negative TTLs covered here.