diff --git a/README.md b/README.md index 290833d..02fcc1a 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,9 @@ agents. - `explore-ecosystems` gives every in-scope dependency a cheap ecosystem identity check, then verifies monorepos, siblings, adapters, plugins, specifications, alternatives, and exclusions deeply for material decisions. +- `build-libraries` owns reusable programming models, public APIs, selective + adoption, data-flow shapes, explicit resource ownership, data-oriented hot + paths, library performance, packaging contracts, and restart/resume boundaries. - `build-clis` owns command language, configuration, output, interaction, cancellation, installed artifacts, and CLI verification. - `build-web` classifies hybrid web surfaces and owns shared renderer, diff --git a/deno.lock b/deno.lock new file mode 100644 index 0000000..14dfc5c --- /dev/null +++ b/deno.lock @@ -0,0 +1,16 @@ +{ + "version": "5", + "specifiers": { + "npm:zod@^4.1.12": "4.4.3" + }, + "npm": { + "zod@4.4.3": { + "integrity": "sha512-ytENFjIJFl2UwYglde2jchW2Hwm4GJFLDiSXWdTrJQBIN9Fcyp7n4DhxJEiWNAJMV1/BqWfW/kkg71UDcHJyTQ==" + } + }, + "workspace": { + "dependencies": [ + "npm:zod@^4.1.12" + ] + } +} diff --git a/skills/build-apis/SKILL.md b/skills/build-apis/SKILL.md index 24667c7..d676e5d 100644 --- a/skills/build-apis/SKILL.md +++ b/skills/build-apis/SKILL.md @@ -27,8 +27,10 @@ package currently implements it. and request test. 2. Define runtime schemas and infer application types. Use Standard Schema only where validator-neutral interoperability is an actual boundary. -3. Construct the root app once per deployment boundary. Keep root, service, and - route middleware responsibilities explicit and ordered. +3. Separate endpoint contracts, service-domain capabilities, and resource + implementations. If the repository selects Effect, construct its runtime once + per deployment boundary. In every stack, do not rebuild long-lived pools, + auth clients, log sinks, or equivalent resources per request. 4. Register matching validator middleware before reading `c.req.valid(...)`. 5. Describe full success and error response contracts, not payload-only shapes. 6. Authenticate identity, then authorize organization/tenant/resource access @@ -45,15 +47,23 @@ package currently implements it. ## Reference routing - [service-modules.md](references/service-modules.md): definitions, handlers, - aggregation, registration, middleware, and reachability. + aggregation, service registries, composition, reachability, and independent + deployment boundaries. - [contracts.md](references/contracts.md): Standard Schema, Zod, validation, responses, problems, and OpenAPI. +- [effect-services.md](references/effect-services.md): Effect services, + `Context.Tag`, Layers, typed errors, Scope, configuration, observability, and + request-runtime integration. - [auth.md](references/auth.md): Better Auth, sessions, plugins, organization policy, routes, and import-safe construction. - [queries.md](references/queries.md): filters, sorts, fields, pagination, count strategies, and server-owned constraints. - [runtime.md](references/runtime.md): Hono adapters, middleware order, resources, LogTape, errors, and cleanup. +- [streaming.md](references/streaming.md): SSE framing, cursors, replay, + backpressure, cancellation, authorization, and durable stream sources. +- [deployment.md](references/deployment.md): independent deployability, + resource/config boundaries, health, readiness, shutdown, and contract tests. - [failures.md](references/failures.md): source-grounded failure signatures and correction paths. diff --git a/skills/build-apis/references/auth.md b/skills/build-apis/references/auth.md index 474bf62..2f5cdb1 100644 --- a/skills/build-apis/references/auth.md +++ b/skills/build-apis/references/auth.md @@ -1,39 +1,237 @@ -# Authentication and authorization +# Authentication, organization policy, and Better Auth -## Better Auth ecosystem +## Contents -Inspect server core, database adapter, generated schema/migrations, session and -cookie behavior, plugins, paired client plugins, framework bindings, route mount, -base URL, trusted origins, and deployment host. Community adapters or plugins do -not gain first-party status through naming. +- [Separate identity from authorization](#separate-identity-from-authorization) +- [Better Auth ecosystem preflight](#better-auth-ecosystem-preflight) +- [Construction and route mounting](#construction-and-route-mounting) +- [Organization policy](#organization-policy) +- [Endpoint middleware patterns](#endpoint-middleware-patterns) +- [Cookies, origins, and CSRF](#cookies-origins-and-csrf) +- [Plugins and client symmetry](#plugins-and-client-symmetry) +- [Operational workflows](#operational-workflows) +- [Tests and failure signatures](#tests-and-failure-signatures) -## Identity versus policy +## Separate identity from authorization -Authentication identifies a user/session. Authorization decides whether that -identity can act on an organization, tenant, resource, scope, or operation. -Resolve active organization deliberately and enforce membership and policy at -the server boundary. +| Layer | Question | Example evidence | +|---|---|---| +| Authentication | Who is making the request? | Verified session/user/API key | +| Tenant selection | Which organization/team is active for this operation? | URL/resource relationship plus session preference | +| Membership | Does the user belong to it? | Server-side membership row | +| Authorization | Can this member perform this operation on this resource? | Role/permission/domain policy | +| Data scope | Which rows can the operation observe/mutate? | Server-owned base filter/scope object | -Apply server-owned base filters before user filters. Never trust an organization -ID from URL, body, or client state without policy validation. +Never infer authorization from a client-provided organization ID, UI visibility, +or authentication alone. Do not rely on frontend route guards for backend policy. -## Plugin and route symmetry +## Better Auth ecosystem preflight -Where Better Auth requires them, pair server plugins with browser client plugins -for organization, passkey, OAuth provider, or other capabilities. Align handler -wildcard, issuer, OIDC discovery, OAuth metadata, consent paths, base path, and -cookie scope. Test both root and mounted installations the app supports. +Treat Better Auth as an ecosystem. Inspect the exact installed versions and +official status of: -## Construction +- server core and selected database adapter; +- generated auth schema/migrations; +- framework handler integration; +- client package and renderer-specific client binding; +- organization, OAuth/OIDC provider, admin, passkey, API key, session-management, + or other plugins actually enabled; +- email sender/verification/reset/invitation callbacks; +- base URL/path, trusted origins, cookie configuration, proxy/TLS behavior; +- rate limiting and secondary storage where configured. -Module import must not read required environment values, connect a database, or -configure global logging. Export factories and construct a shared long-lived -database/auth instance at an explicit composition root. Provide close/drain -behavior for tests, CLIs, workers, and graceful shutdown. +Do not infer first-party compatibility from package naming. Community adapters +and plugins need source/version/security review. Do not enable a plugin only on +the server when it requires a paired client plugin for typed client methods. -## Operational completeness +## Construction and route mounting -Do not require email verification before sender, confirmation, expiry, resend, -and failure workflows exist. Test session expiry/revocation, cookie security, -cross-origin requests, organization changes, provider failure, and database -failure. +Export an import-safe factory. Do not read mandatory env, open a database, or +configure global logging at module import: + +```ts +export interface MakeAuthOptions { + readonly baseURL: string + readonly secret: string + readonly database: AuthDatabase + readonly sendInvitation: SendInvitation + readonly trustedOrigins: readonly string[] +} + +export function makeAuth(options: MakeAuthOptions) { + return betterAuth({ + baseURL: options.baseURL, + secret: options.secret, + database: options.database, + trustedOrigins: [...options.trustedOrigins], + plugins: [organization({ sendInvitationEmail: options.sendInvitation })], + }) +} +``` + +Construct one long-lived auth instance at the host composition root. Mount the +handler at the configured wildcard/base path and test the exact deployed prefix. +Align: + +- public base URL and reverse-proxy prefix; +- auth base path and wildcard route; +- callback/redirect URLs; +- issuer, OAuth/OIDC metadata and consent paths; +- secure cookie domain/path/same-site settings; +- trusted origins and CORS credential policy. + +Do not create a new auth instance per request. Do not mount two instances with +different secrets/adapters in nested apps. + +## Organization policy + +Resolve an authorized scope object before domain/data access: + +```ts +export const AuthorizedOrganization = z.object({ + organizationId: z.string(), + membershipId: z.string(), + actorId: z.string(), + role: z.string(), + permissions: z.set(z.string()), +}) +``` + +The middleware/policy service should: + +1. require a valid session; +2. obtain requested organization from route/resource/product rules; +3. look up current membership server-side; +4. evaluate operation/resource policy; +5. produce an immutable scope; +6. ensure queries receive this scope as a base constraint; +7. audit sensitive mutations. + +For a route like `/organizations/:organization_id/imports/:import_id`, verify the +import belongs to the organization in the same server-owned query. Avoid loading +the import globally and checking organization later if existence leakage matters. + +```ts +const result = await db.query.imports.findFirst({ + where: and( + eq(imports.id, importId), + eq(imports.organizationId, scope.organizationId), + ), +}) +``` + +Whether policy lives in middleware or a domain service depends on context, but a +handler must not be able to forget the tenant base filter. Consider a scoped +store/capability that requires `AuthorizedOrganization`. + +Define behavior for suspended/deleted membership, organization switch, resource +transfer, invitation acceptance, and concurrent revocation. A cached session +claim can become stale; decide which high-risk operations re-read policy. + +## Endpoint middleware patterns + +Use explicit variants: + +| Middleware | Guarantee | +|---|---| +| optional auth | `session` may be absent; no policy grant | +| required auth | verified session/user exists | +| optional organization | organization context may resolve; no membership guarantee unless named | +| required organization | current membership and requested scope verified | +| operation policy | named permission/domain predicate satisfied | + +Route order normally authenticates, authorizes scope, then validates user input +and runs the handler. Validate path syntax before an expensive policy lookup only +if failure behavior does not leak resource existence and middleware contracts +remain explicit. + +Do not accept an `organization_id` inside JSON as authoritative when route or +server state already determines scope. If accepted for commands, compare it to +the authorized scope and reject mismatch. + +## Cookies, origins, and CSRF + +Define deployment-specific cookie policy: + +- `Secure` in production HTTPS; +- `HttpOnly` for session cookies; +- appropriate `SameSite` for same-site or cross-site flows; +- narrow domain/path; +- rotation/expiry/revocation; +- proxy-aware secure detection; +- no secret session material in logs. + +Credentialed cross-origin requests require exact allowed origins; wildcard CORS +is invalid/unsafe. CSRF defenses depend on cookie/same-site architecture and +Better Auth's exact integration. Test hostile origins, preflights, redirects, and +state-changing requests through the real proxy. + +Do not place reusable auth tokens in URLs. For SSE where `EventSource` header +constraints matter, prefer same-origin cookies or short-lived resource-scoped +stream tickets. + +## Plugins and client symmetry + +For every plugin build a matrix: + +| Capability | Server plugin/config | Client plugin/binding | Routes/schema | Operational dependency | +|---|---|---|---|---| +| Organization | organization plugin | organization client plugin where required | org/member/invite | email/invitation policy | +| OAuth provider | provider plugin | OAuth client flow | metadata/authorize/token/consent | keys, redirect URLs | +| Passkey | passkey plugin | browser passkey client | credential tables/routes | secure origin | +| API keys | API-key plugin | management client/UI | key records/routes | hashing, scopes, rotation | + +Confirm schema generation/migrations after plugin changes. Keep server and client +versions compatible. Test both root and custom base-path mounts if supported. + +## Operational workflows + +Do not enable a policy gate without its completion path: + +- email verification requires sender, template, expiry, resend, bounce/failure, + and support recovery; +- password reset requires one-time expiry, session revocation decision, and abuse + controls; +- organization invitation requires membership/role policy, expiry, resend, + duplicate invite, wrong-account, and revocation handling; +- OAuth requires provider error, account-linking, redirect, consent, and state + validation; +- session management requires revoke-one/revoke-all and device/session display if + promised; +- account deletion requires organization/data retention and background cleanup. + +## Tests and failure signatures + +Test: + +- import without env/network; +- one auth instance across concurrent requests; +- exact base path, callbacks, cookies, trusted origins, and proxy HTTPS; +- anonymous, expired, revoked, and malformed sessions; +- cross-organization reads/mutations and guessed resource IDs; +- organization switching and membership revocation during a session; +- each role/permission and server-owned base filter; +- plugin server/client symmetry and generated schema; +- sender/provider/database failure; +- redaction of cookie, authorization, password, token, and secret fields; +- SSE/long request authorization expiry policy. + +| Signature | Defect | Correction | +|---|---|---| +| Authenticated user sees another org | Authentication used as authorization | Membership/policy + scoped query | +| Auth methods missing in client | Paired plugin/binding absent | Server/client plugin matrix | +| OAuth callback fails only in production | Base URL/path/proxy mismatch | Deployed route/cookie integration test | +| Import crashes without secret | Module-scope construction | Factory + composition root | +| CORS accepts arbitrary credentialed origin | Unsafe default | Exact origin policy | +| Email verification blocks all users | Operational email flow incomplete | Complete sender/recovery or defer gate | +| Resource 404 leaks across tenant timing/body | Global lookup before scope | Scoped base query | + +## Sources and freshness + +- Better Auth official documentation: https://www.better-auth.com/docs + (primary source, checked 2026-07-17; plugin and framework APIs are version-sensitive). +- Attachment, verified 2026-07-17: `evidence/app/better-auth/utils/auth/` and + `utils/middleware/auth.ts` (observed integration, not the upstream Better Auth monorepo). +- Attachment, verified 2026-07-17: `evidence/app/new-finance/docs/intent-doc.md` + and frontend access middleware (normative ownership and counterexample evidence; + frontend guards are not server authorization proof). diff --git a/skills/build-apis/references/contracts.md b/skills/build-apis/references/contracts.md index da16b10..8950558 100644 --- a/skills/build-apis/references/contracts.md +++ b/skills/build-apis/references/contracts.md @@ -1,48 +1,258 @@ -# API contracts, validation, and problems +# API contracts, validation, OpenAPI, and problems -## Schema ownership +## Contents -Use Zod v4 or the repository's selected runtime schema system for application -contracts and infer TypeScript types from schemas. Use the Standard Schema -`~standard.validate` contract when a reusable utility intentionally accepts -multiple validators. +- [Contract layers](#contract-layers) +- [Zod and Standard Schema ownership](#zod-and-standard-schema-ownership) +- [Request sources](#request-sources) +- [Normalization and coercion](#normalization-and-coercion) +- [Response and problem contracts](#response-and-problem-contracts) +- [OpenAPI completeness](#openapi-completeness) +- [Contract evolution](#contract-evolution) +- [Testing and verification](#testing-and-verification) +- [Failure signatures](#failure-signatures) -Do not create parallel hand-written interfaces for the same boundary. Do not -claim a validator-neutral library requires every application to use multiple -schema systems. +## Contract layers -## Definition and validator pairing +Keep these layers distinct and traceable: -An endpoint definition describes the contract, but `c.req.valid("json")` or an -equivalent accessor is populated only by matching validator middleware. Register -the correct source validator and prove invalid input returns the stable problem -before the handler executes. +```text +wire bytes / URL / headers + -> source-specific transport parser + -> runtime schema validation + -> normalized application input + -> domain capability + -> domain result or typed failure + -> HTTP response variant + -> OpenAPI operation and generated client expectations +``` -Validate path, query, header, cookie, JSON, and form sources independently where -their failure and coercion semantics differ. Normalize only after source-aware -validation. +The schema nearest the wire owns wire syntax. A domain schema owns domain +meaning. They can share fragments, but do not pretend a query-string boolean and +a JSON boolean have the same input representation. -## Full result contract +## Zod and Standard Schema ownership -A response schema should describe what the handler returns across success and -known failures, not only the payload nested inside it. Include status, headers, -content type, body/problem shape, and empty-response behavior as applicable. +Use Zod v4 schemas as the source of truth where the repository has selected Zod, +then infer types with `z.input`, `z.output`, or the repository's helpers: -Map validation failures to stable client problems, unexpected validation defects -to internal diagnostics, and domain failures to documented problem types. Keep a -registry so OpenAPI and runtime responses use the same identifiers and schemas. +```ts +export const CreateImportJson = z.object({ + upload_id: z.uuid(), + mapping: z.record(z.string(), z.string()), +}) -## Safe causes +export type CreateImportJsonInput = z.input +export type CreateImportJson = z.output +``` -Public problems must not copy raw database, provider, filesystem, or network -messages. Preserve the original cause structurally in redacted diagnostics and -return a stable safe detail to the caller. +Input and output types differ when a schema coerces, defaults, transforms, or +brands. Choose deliberately. -## Contract verification +Use Standard Schema at a validator-neutral library boundary: -- parse actual successful and failure responses against published schemas; -- compare registered routes with OpenAPI operations; -- verify status/content-type/header variants; -- send malformed values through every source; -- assert handler non-execution on validation failure; -- assert no raw secret-bearing cause appears in the response. +```ts +export async function validateWith( + schema: TSchema, + input: StandardSchemaV1.InferInput, +): Promise> { + const result = await schema["~standard"].validate(input) + if (result.issues) throw new BoundaryValidationError(result.issues) + return result.value +} +``` + +Do not rewrite application schemas in multiple validators to prove neutrality. +Test the reusable utility against representative Zod and one other Standard +Schema implementation only when multi-validator support is a real requirement. + +## Request sources + +Define separate schemas for each source read by the handler: + +| Source | Wire concerns | Common traps | +|---|---|---| +| Path params | Percent decoding, non-empty identifiers | Treating untrusted ID as authorization scope | +| Query | Repeated keys, strings, empty values, ordering | Boolean/number coercion and ignored unknown keys | +| Headers | Case-insensitive names, combined/repeated values | Expecting original casing after normalization | +| Cookies | Name/value restrictions, signing, scope | Treating client cookie as trusted session data | +| JSON | Content type, size, finite values, unknown fields | Parsing twice or accepting non-JSON content | +| Form/multipart | Repeated fields, files, limits, streaming | Buffering an unbounded upload | + +Register matching middleware before reading a validated Hono source: + +```ts +export const Middleware = [ + createValidator("param", Definition.schemas.Param), + createValidator("query", Definition.schemas.Query), + createValidator("json", Definition.schemas.Json), +] + +const params = c.req.valid("param") +const query = c.req.valid("query") +const body = c.req.valid("json") +``` + +An endpoint definition alone does not populate `c.req.valid(...)`. + +## Normalization and coercion + +Normalize only after source-aware validation. Make absence, empty string, false, +zero, repeated values, and invalid values observably different where the API +contract needs them. + +```ts +export const ListQuery = z.object({ + limit: z.string().regex(/^\d+$/).transform(Number).pipe(z.int().min(1).max(200)).default("50"), + cursor: z.string().min(1).optional(), + include_archived: z.enum(["true", "false"]).transform((value) => value === "true").default("false"), +}) +``` + +This is an example, not a universal coercion policy. If a query parser supplies +arrays for repeated keys, decide whether to reject or support them before the +transform. Do not let JavaScript's `Boolean("false")` or permissive `Number("")` +define the public API. + +Use `.strict()`, `.passthrough()`, or explicit catchalls according to forward +compatibility and security needs. Security-sensitive command inputs usually +benefit from rejecting unknown fields; metadata bags may intentionally retain +them. + +## Response and problem contracts + +Model complete response variants, not the payload alone: + +```ts +const Responses = { + 202: { + contentType: "application/json", + schema: AcceptedImport, + headers: z.object({ location: z.string().url() }), + }, + 401: problem("unauthorized"), + 403: problem("forbidden"), + 409: problem("idempotency-conflict"), + 422: validationProblem, +} as const +``` + +Use an RFC 9457-style problem registry with stable types/codes and safe public +detail. A problem should carry enough stable information for a client decision: + +```json +{ + "type": "https://api.example.test/problems/idempotency-conflict", + "title": "Idempotency key conflict", + "status": 409, + "detail": "The key was already used for a different request.", + "instance": "/imports", + "code": "idempotency_conflict", + "correlation_id": "req_..." +} +``` + +Do not include SQL, filesystem, provider, stack, secret, or raw validation input +in a public problem. Preserve the original cause in redacted diagnostics. + +For validation problems, retain source and path: + +```json +{ + "code": "validation_failed", + "errors": [ + { + "source": "json", + "path": ["mapping", "email"], + "location": "json.mapping.email", + "message": "Expected a column name" + } + ] +} +``` + +## OpenAPI completeness + +OpenAPI is derived from the runtime definition registry or checked against it. +For each operation declare: + +- stable `operationId`, method, path, tags, summary, and description; +- path/query/header/cookie parameters with required and repeated semantics; +- every request body content type and schema; +- every success, empty, redirect, accepted, validation, auth, conflict, + rate-limit, and known domain-error response; +- response headers such as `Location`, `Retry-After`, cursor, ETag, or rate limit; +- auth/security schemes and per-operation requirements; +- examples that validate; +- deprecation and version information; +- streaming media type and event schema where applicable. + +For accepted durable work, document status, cancellation, and event-stream links. +For cursor endpoints, document cursor opacity and invalid/expired behavior. + +OpenAPI compatibility checks should distinguish: + +| Change | Typical classification | +|---|---| +| Remove operation or response field | Breaking | +| Make optional request field required | Breaking | +| Narrow enum accepted by server | Breaking | +| Add optional response field | Usually compatible, but strict clients can fail | +| Add error response | Behaviorally significant; document and test | +| Change operation ID | Client-generation breaking | +| Change default/sort/cursor semantics | Semantic breaking even if schema unchanged | + +## Contract evolution + +Version behavior, not just shapes. Keep a compatibility window between deployed +clients, API hosts, workers, and migrations. Use additive evolution where +possible and explicit version negotiation where semantics cannot remain +compatible. + +Do not embed a service implementation version into every route automatically. +Choose URL, header, media-type, or capability versioning according to external +consumer needs. Internal APIs still need deployment-order compatibility. + +For events and workflow inputs, retain schemas/decoders for in-flight historical +versions or migrate durable records explicitly. An API host can accept a request +under a new schema while an older worker remains responsible for the run. + +## Testing and verification + +- Parse valid/invalid values for every request source. +- Assert handler non-execution after validation failure. +- Verify unknown-key, repeated-key, empty-value, coercion, content-type, and size + behavior. +- Parse every actual success and problem response with its declared schema. +- Assert headers and empty responses, not only bodies. +- Verify safe causes and redaction with hostile provider/SQL error strings. +- Compare declared, registered, documented, and request-tested operation sets. +- Run OpenAPI lint and breaking-change checks against the released baseline. +- Generate a client and execute at least one representative request when client + generation is promised. +- Replay stored workflow/event inputs against compatible decoders. + +## Failure signatures + +| Signature | Cause | Correction | +|---|---|---| +| `c.req.valid()` is empty | Validator middleware absent/wrong source | Pair schemas and route middleware | +| Type compiles but runtime returns string | `z.input`/`z.output` confusion or missing transform | Test parsed output | +| Client cannot handle actual response | Payload-only schema | Declare full variants/status/headers | +| OpenAPI route returns 404 | Generator catalog differs from runtime registry | One registry plus reachability test | +| Validation returns 500 | Transform threw outside normalized validation boundary | Normalize schema and thrown validator errors | +| Cross-org ID passes schema | Validation mistaken for authorization | Server-owned policy after identity | +| Generated client breaks after “additive” change | Strict decoder or operation ID drift | Compatibility test the real client | +| Raw SQL appears in problem detail | Cause copied to public response | Stable problem and redacted diagnostic | + +## Sources and freshness + +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/endpoint/`, + `utils/middleware/validation.ts`, `utils/response/`, and `utils/query/` + (observed executable contracts and tests). +- Standard Schema specification: https://github.com/standard-schema/standard-schema + (primary source; validate the installed specification version). +- Zod v4 documentation: https://zod.dev/ (primary source; examples assume v4). +- OpenAPI Specification: https://spec.openapis.org/oas/latest.html (primary source; + generator/library support can lag the latest specification). +- RFC 9457 Problem Details: https://www.rfc-editor.org/rfc/rfc9457.html (stable standard). diff --git a/skills/build-apis/references/deployment.md b/skills/build-apis/references/deployment.md new file mode 100644 index 0000000..fcb7c75 --- /dev/null +++ b/skills/build-apis/references/deployment.md @@ -0,0 +1,141 @@ +# API deployment and resource boundaries + +## Contents + +- [Deployment contract](#deployment-contract) +- [Configuration boundary](#configuration-boundary) +- [Resource graph](#resource-graph) +- [Health and readiness](#health-and-readiness) +- [Shutdown and draining](#shutdown-and-draining) +- [Independent-deployability matrix](#independent-deployability-matrix) +- [Verification](#verification) +- [Failure signatures](#failure-signatures) + +## Deployment contract + +For every API artifact, record: + +- runtime and executable entrypoint; +- public/internal base URL and route prefix; +- environment/config schema; +- required databases, queues, workflow engines, object stores, auth providers, + network peers, secrets, and certificates; +- migration owner and minimum/maximum compatible schema versions; +- worker processes and task queues required for accepted work; +- observability sinks and failure policy; +- health, readiness, startup, and shutdown semantics; +- expected concurrency, timeouts, body/stream limits, and proxy behavior; +- rollout and rollback order. + +An import-safe library package is not an independently deployable service until +an executable host proves this contract. + +## Configuration boundary + +Resolve configuration once. If c12/defu merge files, environment, and CLI +overrides, complete that merge before constructing service resources. Validate +the final result with Zod and preserve safe provenance for `config explain`. + +Do not: + +- read required env at module import; +- let each database/auth/workflow module invent precedence; +- merge arrays by accidental concatenation; +- treat empty string, false, zero, null, and absence as equivalent; +- log resolved secrets; +- silently default security-sensitive production values. + +Separate deployment config from request input and domain records. Version the +config schema when operators need a migration path. + +## Resource graph + +Inventory each resource: + +| Resource | Owner | Acquire | Health | Drain/close | +|---|---|---|---|---| +| HTTP listener | Host | Start after readiness prerequisites | Accept loop | Stop admission, drain | +| Postgres pool | Host Layer | Once | Lightweight query/pool stats | Close after requests/workers | +| Auth instance | Host | Once per config | Dependency-specific | Close adapter if supported | +| Workflow client | Host | Once | Engine connectivity | Drain/close client | +| Log/trace sinks | Observability Layer | Once | Exporter status | Flush with timeout | +| SSE subscription | Request scope | Per stream | Delivery activity | Cancel on disconnect | + +Resources created inside request handlers need an explicit reason. Transactions, +temporary artifacts, and subscriptions are request-scoped; pools and sinks are +not. + +## Health and readiness + +- Liveness answers whether the process should be restarted. It should not fail + solely because a downstream dependency is temporarily unavailable. +- Readiness answers whether new traffic should be admitted. It can include + required dependency and registry checks. +- Startup checks can validate configuration, migrations, registries, workflow + catalogs, and ports before readiness becomes true. + +Accepted durable-start routes must not report ready when they cannot persist or +submit accepted work according to contract. A read-only degraded mode can remain +ready only if routing/capability status makes that behavior explicit. + +## Shutdown and draining + +Use this order: + +1. mark unready and stop new admission; +2. stop accepting new listeners/streams; +3. ask workers to stop claiming new work; +4. drain in-flight requests, activities, and stream writes within a deadline; +5. release leases or let their durable expiry recover them; +6. flush telemetry; +7. close workflow clients, auth adapters, database pools, and listener; +8. force termination only after the published deadline. + +Test both cooperative shutdown and abrupt termination. Finalizers cannot repair +a SIGKILL; leases, idempotency, and reconciliation must. + +## Independent-deployability matrix + +| Question | Required evidence | +|---|---| +| Can it boot alone? | Minimal production-like config and boot test | +| Can clients discover its contract? | Versioned OpenAPI/event schemas | +| Can it roll forward/back? | Compatibility matrix and migration policy | +| Can it operate without monolith imports? | Artifact dependency inspection | +| Can it expose accepted work honestly? | Worker/engine reachability readiness | +| Can it shut down safely? | Drain and forced-restart tests | +| Can operators diagnose it? | Structured health, logs, traces, metrics, config provenance | + +## Verification + +- Build the exact deployment artifact. +- Boot it with minimal config and with each missing required dependency. +- Send real requests through the deployed adapter, not only `app.request`. +- Compare runtime routes and OpenAPI. +- Exercise auth, CORS, proxy headers, body limits, cancellation, and streaming. +- Run migration compatibility against current and previous application versions. +- Kill during requests and durable starts; verify retry/recovery and no false 202. +- Drain under load and assert resource close/telemetry flush deadlines. +- Scan the artifact for unintended modules, secrets, and development middleware. + +## Failure signatures + +| Signature | Likely defect | +|---|---| +| Health is green but every request 500s | Liveness used as readiness | +| 202 returned while worker absent | Durable admission not part of readiness | +| Service imports monolith root | Boundary is organizational only | +| Deploy requires undocumented env | Import-time or scattered config | +| Rollback fails after migration | No schema compatibility window | +| Shutdown drops accepted work | Admission/drain ordering wrong | +| Tests pass only with shared dev server | No standalone artifact verification | + +## Sources and freshness + +- Attachments, verified 2026-07-17: `evidence/app/new-finance/docs/intent-doc.md`, + `utils/server/`, and `utils/workflows/` (normative boundary plus observed and + incomplete runtime evidence). +- Hono official documentation: https://hono.dev/docs/getting-started/basic + (primary source; deployment adapters differ). +- Deno deployment/runtime documentation: https://docs.deno.com/runtime/ + (primary source; verify selected host and current Deno release). diff --git a/skills/build-apis/references/effect-services.md b/skills/build-apis/references/effect-services.md new file mode 100644 index 0000000..8c2fa9d --- /dev/null +++ b/skills/build-apis/references/effect-services.md @@ -0,0 +1,362 @@ +# Effect services and Layers + +## Contents + +- [When to load this reference](#when-to-load-this-reference) +- [The three-channel model](#the-three-channel-model) +- [Service contracts with Context.Tag](#service-contracts-with-contexttag) +- [Layer construction](#layer-construction) +- [Composition rules](#composition-rules) +- [Typed domain errors](#typed-domain-errors) +- [Scope and finalizers](#scope-and-finalizers) +- [Configuration](#configuration) +- [Observability](#observability) +- [Hono integration](#hono-integration) +- [Testing](#testing) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions and version boundary](#deliberate-exclusions-and-version-boundary) + +## When to load this reference + +Load this reference when a service uses Effect or would materially benefit from +typed failures, explicit requirements, resource lifetime, interruption, +concurrency, retries, or replaceable implementations. Do not rewrite ordinary +pure helpers into Effect solely for stylistic uniformity. + +## The three-channel model + +Read `Effect.Effect` as three independent facts: + +| Channel | Question | Example | +|---|---|---| +| Success | What value can this operation produce? | `Family` | +| Error | Which expected failures must a caller handle? | `FamilyNotFound | StoreUnavailable` | +| Requirements | Which capabilities must be provided? | `FamiliesStore | Tracer` | + +Defects, invariant violations, and process-fatal conditions are not automatically +domain errors. Model expected recovery decisions in the error channel; preserve +unexpected defects as causes and handle them at an owned boundary. + +## Service contracts with Context.Tag + +Define a service around domain capabilities, not a concrete library: + +```ts +import { Context, Effect } from "effect" + +export interface FamilyServiceShape { + readonly get: ( + scope: FamilyScope, + ) => Effect.Effect + readonly updatePreferences: ( + scope: FamilyScope, + patch: PreferencesPatch, + ) => Effect.Effect +} + +export class FamilyService extends Context.Tag("app/FamilyService")< + FamilyService, + FamilyServiceShape +>() {} +``` + +Use stable globally distinctive tag identifiers. Keep the interface small enough +to fake in tests but coherent enough that callers do not assemble business +transactions from low-level store operations. + +An implementation can depend on other services in its Layer construction: + +```ts +export const FamilyServiceLive = Layer.effect( + FamilyService, + Effect.gen(function* () { + const store = yield* FamilyStore + const audit = yield* AuditService + + return FamilyService.of({ + get: (scope) => store.get(scope), + updatePreferences: (scope, patch) => + store.updatePreferences(scope, patch).pipe( + Effect.tap((family) => audit.record({ + actorId: scope.actorId, + action: "family.preferences.updated", + resourceId: family.id, + })), + ), + }) + }), +) +``` + +The example is structural. Confirm the exact Effect version's `Context.Tag`, +`Layer.effect`, and service helper APIs before copying syntax. + +## Layer construction + +Select the constructor from the implementation's behavior: + +| Need | Typical constructor | Lifetime | +|---|---|---| +| Constant fake/config | `Layer.succeed` | No acquisition | +| Effectful initialization without finalizer | `Layer.effect` | Layer build | +| Resource with cleanup | `Layer.scoped` | Scope-bound | +| Map one dependency into another | `Layer.effect`/service transformation | Layer build | +| Combine independent services | `Layer.merge` or `Layer.mergeAll` | Shared graph | +| Supply dependencies to a Layer | `Layer.provide`/`provideMerge` as appropriate | Build graph | + +Do not construct a Layer inside a request handler. Layer memoization applies +within a build graph; rebuilding the graph per request can recreate pools, +clients, and background fibers. + +```ts +export const DatabaseLive = Layer.scoped( + Database, + Effect.acquireRelease( + Effect.tryPromise({ + try: () => openDatabase(config.databaseUrl), + catch: (cause) => new DatabaseStartError({ cause }), + }), + (database) => Effect.promise(() => database.close()), + ), +) +``` + +If finalization can fail, decide whether shutdown records the failure, retries +within a bound, or fails the host. Do not silently discard cleanup errors. + +## Composition rules + +Compose from leaves toward the host: + +```text +Config + -> Log/trace exporters + -> Database and external clients + -> Stores/gateways + -> Domain services + -> Workflow clients + -> HTTP handlers + -> Host runtime +``` + +At every Layer boundary, answer: + +- Which tags does it provide? +- Which tags does it require? +- Is it shared or request-scoped? +- What does it acquire and release? +- Which configuration has already been parsed? +- Can it start fibers, and who supervises them? + +Use a single application Layer exported from the service's runtime module: + +```ts +export const AccountsLive = Layer.mergeAll( + FamilyServiceLive, + PreferencesServiceLive, + ImportClientLive, +).pipe( + Layer.provideMerge(StoreLive), + Layer.provideMerge(DatabaseLive), + Layer.provideMerge(ObservabilityLive), +) +``` + +The exact dependency direction can differ. Typecheck the final graph and inspect +it for duplicate resource Layers rather than adding `provide` calls until the +type error disappears. + +## Typed domain errors + +Give expected errors stable discriminants and safe fields: + +```ts +import { Data } from "effect" + +export class FamilyNotFound extends Data.TaggedError("FamilyNotFound")<{ + readonly familyId: string +}> {} + +export class PreferencesConflict extends Data.TaggedError("PreferencesConflict")<{ + readonly familyId: string + readonly expectedRevision: number +}> {} +``` + +Translate errors once at the transport boundary: + +```ts +const program = FamilyService.pipe( + Effect.flatMap((service) => service.updatePreferences(scope, patch)), + Effect.catchTags({ + FamilyNotFound: () => Effect.succeed(notFoundProblem), + PreferencesConflict: (error) => Effect.succeed(conflictProblem(error)), + }), +) +``` + +Do not collapse every failure into `Error`, stringify a `Cause`, or expose a raw +database/provider message. Preserve structured causes in redacted diagnostics. +Use retry schedules only for failures classified as transient; invalid input, +authorization denial, conflicts requiring user decisions, and deterministic +bugs should not retry blindly. + +## Scope and finalizers + +Use Scope when an operation owns a resource whose lifetime is smaller than the +whole process: transaction, temporary file, stream subscription, lease, +connection, or child fiber group. + +```ts +export const withExportFile = ( + use: (file: ExportFile) => Effect.Effect, +) => Effect.scoped( + Effect.acquireRelease( + createExportFile, + (file) => removeExportFile(file).pipe(Effect.orDie), + ).pipe(Effect.flatMap(use)), +) +``` + +Finalizers run on success, typed failure, defect, and interruption. Still test +the actual host/runtime adapter: a hard process kill cannot execute in-process +finalizers, so durable leases and reconciliation must cover abandoned work. + +Distinguish: + +- host scope: pools, sinks, clients, worker supervisors; +- request scope: disconnect signal, request span, temporary resources; +- transaction scope: one atomic database unit; +- workflow activity scope: activity-local clients/temporary artifacts; +- stream scope: subscription, heartbeat, encoder, and cancellation listener. + +## Configuration + +Resolve config once at the host boundary. A Layer may consume a validated config +service, but should not independently reload `.env`, c12 files, or process env. + +```ts +export class AccountsConfig extends Context.Tag("app/AccountsConfig")< + AccountsConfig, + z.output +>() {} + +export const AccountsConfigLayer = (input: unknown) => + Layer.succeed(AccountsConfig, AccountsConfigSchema.parse(input)) +``` + +Keep secrets redacted. Record config provenance when operators need to explain a +resolved value, but never attach secret values to logs or traces. If c12/defu own +authoring and merging, they run before the Effect config value is provided; do +not create a second precedence system inside Layers. + +## Observability + +Effect provides logging, tracing, metrics, and OpenTelemetry integration. The +repository may use LogTape as its runtime transport. Integrate them through one +owned bridge or sink policy rather than emitting duplicate records through both. + +Preserve these fields across HTTP, services, activities, and stores: + +- trace/span and correlation IDs; +- actor, organization/tenant, and request IDs where policy permits; +- service/module/operation name; +- workflow/run/activity/attempt IDs when present; +- error tag and safe cause class; +- duration, retry count, outcome, and cancellation status. + +Do not log the same failure in every layer. A lower layer should add structured +context to the error/cause; the owned boundary emits one diagnostic unless an +intermediate retry/compensation event is operationally meaningful. + +## Hono integration + +Choose one integration model: + +| Model | Use when | Risk | +|---|---|---| +| Host `ManagedRuntime` | Many handlers execute Effects against one graph | Must dispose at shutdown | +| Explicit capability object in Hono context | Small service or gradual migration | Loses compile-time requirement graph at handler boundary | +| Effect-native HTTP platform | Whole host is designed for it | Larger framework change; do not mix casually with Hono | + +For Hono with a host runtime: + +```ts +const runtime = ManagedRuntime.make(AccountsLive) + +app.use("*", async (c, next) => { + c.set("runEffect", (effect) => runtime.runPromise(effect)) + await next() +}) + +const close = async () => { + stopAdmission() + await drainRequests() + await runtime.dispose() +} +``` + +Connect request cancellation to Effect interruption when the runtime adapter +supports it. Do not let a disconnected upload, query, or SSE consumer continue +indefinitely by default. + +## Testing + +Provide small deterministic Layers: + +```ts +export const FamilyServiceTest = Layer.succeed(FamilyService, { + get: (scope) => Effect.succeed(familyFixture(scope.familyId)), + updatePreferences: (_scope, patch) => Effect.succeed(updatedFamily(patch)), +}) +``` + +Test: + +- every typed error branch; +- missing Layer requirements as a compile-time or construction failure; +- resource acquisition exactly once; +- finalization on success, failure, and interruption; +- request cancellation interrupts owned work; +- retry policy excludes permanent errors; +- log/trace fields and redaction; +- a composed request through the production Layer graph with test resources. + +## Failure signatures + +| Signature | Likely cause | Correction | +|---|---|---| +| Pool count grows per request | Layer/runtime rebuilt in middleware | Build once at host root | +| `Effect<_, never, _>` around fallible I/O | Errors converted to defects or swallowed | Model expected failure channel | +| Every error becomes HTTP 500 | No tagged boundary mapping | Map expected tags to stable problems | +| Duplicate logs for one exception | Every Layer logs and rethrows | One emission boundary plus structured cause | +| Shutdown hangs | Unscoped fiber or missing finalizer | Supervise and bound drain | +| Test needs production env | Config/resource acquisition at import | Parameterized Layer/factory | +| `Layer.provide` maze compiles but creates duplicates | Graph assembled by trial and error | Inventory provides/requires and inspect sharing | +| Request abort has no effect | AbortSignal not connected to interruption | Add scoped cancellation bridge | + +## Deliberate exclusions and version boundary + +- Do not present Effect as durable persistence by itself. Ordinary fibers and + retries disappear with the process. +- Do not put Hono `Context` into domain service interfaces. +- Do not use `Effect.catchAll` to turn every failure into success. +- Do not wrap an already managed pool with a finalizer that closes it per call. +- Do not mix multiple logging transports without an explicit routing owner. + +The uploaded code pins `effect` 3.21.x and `@effect/workflow` 0.18.x while its +Deno imports use major ranges. Current official Effect material describes +Workflows as alpha. Confirm exact APIs against the installed versions and pin a +tested compatibility set before production use. The patterns in this reference +are stable architectural guidance; copied API syntax still requires typecheck. + +## Sources and freshness + +- Effect official documentation: https://effect.website/docs/ and monorepo + https://github.com/Effect-TS/effect (primary sources, checked 2026-07-17). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/`, + especially tests using `Layer`, `ManagedRuntime`, and `WorkflowEngine.layerMemory` + (observed source; not production durability evidence). +- Version boundary: uploaded manifests pin `effect` `^3.21.3`. `Context.Tag`, + `Layer`, `ManagedRuntime`, Scope, and helper signatures are version-sensitive; + typecheck every example against the repository lockfile. diff --git a/skills/build-apis/references/failures.md b/skills/build-apis/references/failures.md index eb9752c..a4cebd8 100644 --- a/skills/build-apis/references/failures.md +++ b/skills/build-apis/references/failures.md @@ -1,19 +1,223 @@ -# API failure signatures +# API failure diagnosis, recovery, and reachability -| Signature | Likely defect | Correction evidence | +Use this reference to diagnose API behavior that is wrong, unreachable, unsafe, duplicated, leaking internals, or operationally incomplete. Start from the executable request path. Definitions, generated OpenAPI, types, and handler files can all exist while the capability remains unreachable. + +## Contents + +- Evidence ladder +- Reachability failures +- Validation and contract failures +- Middleware and composition failures +- Authentication and authorization failures +- Resource and dependency failures +- Error and observability failures +- Workflow/stream boundary failures +- Recovery protocol +- Failure-injection matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness + +## Evidence ladder + +Classify a capability honestly: + +```text +described -> defined -> registered -> bootable -> reachable + -> behaviorally correct -> authorized -> observable -> recoverable +``` + +Evidence needed: + +- description: documentation only; +- definition: method/path/request/response contract exists; +- registration: definition has a matching runtime handler/middleware mapping; +- bootable: the composed service starts with resolved dependencies; +- reachable: an actual request hits the handler; +- correct: response and side effects match the contract; +- authorized: positive and negative tenant/resource policy tests pass; +- observable: one correlated diagnostic and stable client failure; +- recoverable: dependency/process interruption has defined retry/repair behavior. + +Never upgrade a lower rung into a higher claim. + +## Reachability failures + +| Signature | Likely defect | Evidence to inspect | Correction proof | +|---|---|---|---| +| OpenAPI lists route but request is 404 | definition included in docs but omitted from runtime registry | definition aggregate, handler map, registration loop | startup conformance plus real request | +| Handler exists but never runs | name/key/method/path mismatch | `Definition.Name`, handler keys, base path, mount | invocation counter/request fixture | +| Startup only warns about missing handler | partial registry tolerated | startup validation policy | fail boot or explicitly mark capability unavailable | +| Empty 200/204 from stub | unavailable behavior hidden as success | handler body, TODO/stub, side-effect oracle | unregister, explicit 501/capability response, or implement | +| Middleware executes twice | nested app/root factory or duplicate mount | app construction tree and middleware counters | one root and exact-once request test | +| Service imports but cannot boot alone | hidden root dependency/env/global side effect | import graph, config/resource creation | standalone composition-root boot | +| Correct route under wrong prefix | base path/function/mount drift | host routing and generated URLs | deployed-path request | + +The retained service guides make `mod.ts` the contract registry and `index.ts` the runtime composition/handler registry. Those names are local design evidence, not a universal framework requirement. Preserve the separation and add conformance at the consumer's actual registration seam. + +## Validation and contract failures + +| Signature | Likely defect | Correction | +|---|---|---| +| `c.req.valid('json')` empty/unsafe | matching validator middleware missing or wrong source | register exact source schema before handler; request test | +| Query accepted but ignored | disabled feature silently normalizes/handler ignores | reject or document explicit disabled semantics | +| Header validation fails by casing | transport normalization mismatch | use framework/header canonical behavior and tests | +| Form/file request consumes body twice | multiple parsers/middleware ownership | one body parsing owner and size limits | +| OpenAPI response passes but client fails | schema models payload, not status/headers/envelope variants | contract actual response tuples/variants | +| Type uses schema object rather than inferred data | `typeof Schema.Input`/similar misconception | verified schema-library inference helper | +| Default sort/count changes unnoticed | semantic contract not in schema diff | behavioral compatibility snapshots | +| Validation error becomes 500 | expected issues cross wrong error boundary | stable 400/422 mapping and issue paths | + +If using Standard Schema, inspect `~standard.validate` result shape and async behavior at the installed implementation. If using Zod or another owner, preserve its supported inference and error APIs. Do not force either. + +## Middleware and composition failures + +Middleware executes as an ordered/onion system. Establish ownership: + +```text +host/root: proxy trust, request ID, correlation, access diagnostics, CORS/security, final error boundary +service: long-lived/request-scoped dependency adaptation and service policy +route: authentication/authorization, validation, rate/capability policy +handler: domain call and response shaping +``` + +Failure signatures: + +- response diagnostics missing because middleware does not await `next()`; +- error logged multiple times at route, service, and root; +- request ID generated twice and changes mid-request; +- wildcard CORS plus credentials or unsafe “production default”; +- pretty response formatting modifies stable result/download/stream output; +- service-wide DB/client created per request; +- framework app factory called in nested endpoint groups and installs root middleware again; +- thrown exception bypasses stable problem mapping; +- proxy headers trusted from untrusted peers. + +The retained finance server is a useful counterexample: it documents default CORS as allowing all origins, configures LogTape at module import, uses top-level await, and contains visible patch-marker lines in a doc comment. Do not copy it as a production default. Preserve the consuming application's logging/configuration owners and import safety. + +## Authentication and authorization failures + +Authentication says who the session represents. Authorization says whether the current principal may act on the current resource now. + +| Signature | Defect | Required proof | +|---|---|---| +| User sees another organization's row | request org ID used as authority | current membership plus source query scoped by org/resource | +| Valid session retains revoked access | membership cached/assumed for session lifetime | revocation test and policy re-read | +| Search result grants access | projection used as permission authority | authoritative membership/resource check | +| Status endpoint leaks execution existence | start route protected but status/cancel not equally scoped | every workflow endpoint policy matrix | +| SSE continues after access revoke indefinitely | long-lived auth policy undefined | reconnect/revalidation/termination test | +| Admin/provider webhook trusts body identity | authenticity/authorization absent | signature/replay verification and source policy | + +Choose and document 403 versus non-disclosing 404. Apply the same policy to counts, errors, timings where practical, streams, exports, and workflow controls. + +## Resource and dependency failures + +| Signature | Cause | Repair evidence | |---|---|---| -| `c.req.valid()` is empty or unsafe | Matching validator middleware missing | Definition, middleware registration, request test | -| Middleware fires twice | Nested root server construction | One composition root and invocation count | -| OpenAPI lists an endpoint that returns nothing | Definition registered without reachable handler | Registry-to-request trace | -| Stub returns empty 200 | Unavailable capability hidden as success | Unregister or explicit 501/unavailable contract | -| Authenticated user sees another org | Auth mistaken for authorization | Membership policy and server base filter | -| Import fails without env | Resource construction at module scope | Import-safe factory and explicit composition root | -| Client receives SQL/provider message | Raw cause leaked | Stable problem plus redacted structured log | -| CORS allows every origin unexpectedly | “Production” defaults are unsafe | Explicit origin/credential policy | -| Response schema passes but client fails | Payload-only schema omits status/headers/result wrapper | Parse actual response variants | -| Type exported with `typeof Schema.Input` | Schema object type mistaken for inferred data | Infer through schema/type helper | -| Missing handler only warns | Registry tolerates partial reachability | Startup validation or explicit capability status | -| DB pool never closes | Lifetime handle hidden | Composition-root close/drain test | - -Treat documentation markers, workspace globs, and generated OpenAPI as evidence -to inspect, not proof that the runtime path is complete. +| Import fails without env/network | construction at module scope | import-safe contract modules and explicit composition root | +| Connections grow per request | pool/client/runtime built in middleware/handler | acquisition count and host-owned lifetime | +| Shutdown hangs | hidden close/drain or intake still active | stop intake, abort/drain, dispose resources | +| Healthy endpoint while DB schema missing | liveness used as readiness | dependency/schema/capability readiness | +| Timeout returns but query continues | request abort not propagated | cancellation trace and resource release | +| Retry duplicates mutation | blanket request retry without idempotency | idempotency key and side-effect oracle | +| Test fake passes while real adapter fails | fake omits transport/driver semantics | container/deployed integration test | + +Health, readiness, and startup are different. Liveness should not flap for a transient downstream; readiness should prevent serving a capability whose required dependency/schema is unavailable. + +## Error and observability failures + +One failure should produce: + +- one stable client problem without secrets/internal messages; +- one primary correlated diagnostic at the owner boundary; +- structured cause chain retained internally; +- trace/request ID shared across middleware, dependency calls, workflow start, and stream where applicable; +- retryability and operator action classification. + +Failure signatures: + +- raw SQL/provider/stack text in response; +- every layer logs the same error; +- handler catches unknown error and returns empty success; +- 500 converted to 200 by generic response helper; +- stable result body receives diagnostic prefixes; +- correlation context disappears across async callbacks; +- problem `instance` exposes an internal path or sensitive query; +- errors use unstable human strings as machine codes. + +Do not require LogTape. If it is selected, configure it once at the application composition root, not in a reusable server module. Otherwise use the selected observability owner with the same category/correlation/redaction contract. + +## Workflow/stream boundary failures + +| Signature | Defect | Proof | +|---|---|---| +| Start returns 202 but no durable record/queue | acceptance precedes durable commit | execution/status immediately readable; restart test | +| Status says success while required sink failed | global workflow state collapses per-stage results | stage/sink manifest and response policy | +| Cancel endpoint returns success but runtime cannot cancel | capability inferred from definition | runtime adapter/conformance and terminal-state test | +| SSE reconnect loses events | no durable event ID/replay boundary | `Last-Event-ID` replay fixture | +| Slow stream client grows memory | no backpressure/bounds | slow-reader memory oracle | +| Request abort leaves work running unintentionally | cancellation ownership absent | abort trace and explicit detach policy | +| Workflow runtime adapter returns “not implemented” | public route exposed before capability | unregister/501/readiness until implemented | + +## Recovery protocol + +1. Identify the first failed rung in the evidence ladder. +2. Preserve request/trace ID, deployment version, resolved config (redacted), route inventory, dependency state, and affected durable IDs. +3. Reproduce with the smallest real request and record status/headers/body/side effects. +4. Determine whether the request was rejected before commit, committed, partially committed, or completed with response loss. +5. Retry only if idempotency and commit evidence make it safe. +6. Repair registry/config/schema/dependency ownership at the failing boundary. +7. Run positive, negative, interruption, and restart tests. +8. Update readiness/capability reporting so the same partial state is visible. + +For ambiguous mutations, query by idempotency/request/execution identity before retrying. + +## Failure-injection matrix + +Inject: + +- missing handler/definition/middleware mapping; +- invalid each-source payload, oversized input, repeated keys, wrong content type; +- expired/tampered cursor; +- membership revoke between auth and query; +- cross-tenant resource IDs; +- dependency connect/query timeout, restart, malformed result; +- process loss before/after authoritative commit; +- request abort before and during dependency call; +- duplicate idempotency key and concurrent request; +- response serialization failure after side effect; +- SSE slow reader, disconnect, replay gap, and revocation; +- workflow start/status/cancel with unavailable runtime; +- shutdown with active requests and streams. + +## Executable verification + +Verification sequence: + +```text +import contract module without env/network + -> boot service from explicit config and disposable dependencies + -> compare definitions, handlers, middleware, OpenAPI, and route inventory + -> issue real positive/negative requests + -> inspect authoritative side effects + -> inject dependency/process failures + -> restart and reconcile by request/execution ID + -> stop intake and prove clean shutdown +``` + +Use repository-native tasks and a real HTTP client. Assert exact middleware invocation counts, response status/headers/body, current authorization, side effects, stable problem shape, and one diagnostic owner. A handler unit test alone does not prove routing, middleware, serialization, or host configuration. + +## Deliberate exclusions + +- Do not force Hono, Zod, Standard Schema, LogTape, Effect, Drizzle, or Better Auth. +- Do not call a route implemented from definition/OpenAPI existence. +- Do not keep stubs reachable as successful capabilities. +- Do not treat valid sessions as current resource authorization. +- Do not retry ambiguous mutations without idempotency/commit evidence. +- Do not configure global logging or create pools at reusable-module import. +- Do not log the same expected failure at every layer. +- Do not report structural/type checks as executable API verification. + +## Sources and freshness + +Grounded in the retained finance service-module authoring/structure guides; endpoint, validation, query, response, server, auth, middleware, and workflow utilities; and observed counterexamples including wildcard CORS defaults, import-time LogTape configuration, registry reachability risks, private schema inference mistakes, and incomplete workflow adapters, reviewed 2026-07-17. Framework, validation, auth, observability, and runtime APIs are version-sensitive; inspect the consuming repository and run actual requests before asserting capability. diff --git a/skills/build-apis/references/queries.md b/skills/build-apis/references/queries.md index 397835a..cf27f0c 100644 --- a/skills/build-apis/references/queries.md +++ b/skills/build-apis/references/queries.md @@ -1,39 +1,296 @@ -# Query contracts and pagination +# API query contracts and collection endpoints -## Normalize sources +Use this reference when an HTTP/API endpoint exposes filtering, sorting, field selection, pagination, counts, search, or graph queries. It owns public syntax and semantic compatibility. Compose with `build-data/references/queries.md` for storage compilation and execution. -Query, JSON, and form inputs may use different source syntax but should normalize -into one validated query request. Keep filters, sorts, selected fields, -pagination, and count policy explicit. +## Contents -Disabled features should normalize to `null` or a rejected input according to -the public contract. Do not silently accept an ignored filter or sort. +- Public ownership model +- Source decoding and normalization +- Endpoint configuration +- Filtering +- Sorting +- Field selection +- Pagination and cursors +- Count and response metadata +- Authorization and policy +- Compatibility and OpenAPI +- Integration example +- Failure and error mapping +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Server-owned policy +## Public ownership model -Apply tenant, organization, soft-delete, visibility, and other server-owned base -constraints before user-controlled filters. Validate allowed fields/operators and -map public names to storage expressions rather than interpolating arbitrary input. +An API query contract includes more than a request schema: -## Pagination +- accepted source and exact wire syntax; +- supported public fields/operators/directions; +- defaults and disabled-feature behavior; +- maximum complexity and page size; +- stable ordering and cursor semantics; +- count semantics; +- server-owned tenant/visibility policy; +- response envelope, links, pagination metadata, status, and headers; +- stable errors for invalid, expired, forbidden, too-complex, timed-out, and unavailable queries; +- versioning and generated-client/OpenAPI behavior. -Treat cursor and offset pagination as distinct discriminated contracts. Define -stable ordering and tie-breakers before cursors. Specify behavior for invalid, -expired, or cross-filter cursors and for inserts/deletes between pages. +Changing default order, cursor payload interpretation, count from exact to estimated, wildcard expansion, or operation ID can break clients even when the JSON schema still validates. -## Count strategy +## Source decoding and normalization -| Strategy | Use when | Limitation | +Query parameters, JSON bodies, and form data have different wire shapes. Decode one selected source and normalize into one internal query specification: + +```text +query: filter[status][in]=open,blocked&sort=-created_at&fields=id,status + -> transport decoder + -> normalized filters/sorts/fields/pagination +``` + +Do not merge sources by an undocumented precedence such as JSON over query over form. If an endpoint accepts multiple sources, define whether duplicate components conflict, one source wins, or the request is rejected. + +The retained finance utilities provide separate query/JSON/form adapters for filters, sorts, fields, and pagination, and a composite factory. Treat those exact function names as repository-local unless their public package exports are confirmed. + +Parsing and policy are separate: + +- decoder recognizes bracket/CSV/JSON syntax; +- validator enforces limits and public registries; +- handler receives normalized values only after matching middleware ran; +- data layer maps approved public fields to storage expressions. + +## Endpoint configuration + +Make each collection endpoint's capabilities inspectable: + +```ts +const ListAccountsQuery = defineCollectionQuery({ + source: 'query', + filters: { + max: 12, + fields: { + status: { type: 'enum', operators: ['eq', 'in'], values: ACCOUNT_STATUS }, + openedAt: { type: 'date', operators: ['gte', 'lte'] }, + }, + }, + sorts: { + allowed: ['createdAt', 'name'], + default: [{ field: 'createdAt', direction: 'desc' }], + tiebreaker: 'id', + max: 3, + }, + fields: { + allowed: ['id', 'name', 'status', 'createdAt'], + defaults: ['id', 'name', 'status'], + }, + pagination: { modes: ['cursor'], defaultLimit: 20, maxLimit: 100 }, + count: 'none', +}) +``` + +This is a conceptual contract. Use the consumer's selected schema/query library and verified APIs. + +Disabled components must either reject supplied parameters or document that the component is unavailable and normalizes to `null`. Silently accepting ignored filters is a false API. + +## Filtering + +Useful URL syntax includes: + +```text +filter[status]=active +filter[amount][gte]=50 +filter[amount][lte]=200 +filter[deleted_at]=null +filter[status][in]=open,blocked +``` + +Define: + +- whether missing operator means `eq`; +- null keywords versus literal strings; +- escaping/delimiter rules for arrays (CSV is ambiguous when values contain commas); +- repeated-key behavior; +- AND/OR semantics; do not imply arbitrary boolean groups if only AND exists; +- supported operators per field; +- scalar types and enum values; +- maximum filters and values; +- date/timezone normalization; +- empty list behavior. + +Do not expose storage column names by accident. Public fields map through an allowlist. Reject unknown operators before the storage layer. + +## Sorting + +Define one public syntax, for example `sort=-createdAt,name`, and whether `-` means descending. Each endpoint owns allowed fields, defaults, maximum keys, and the immutable unique tiebreaker. + +A sort contract is semantically breaking if its default or null/collation behavior changes. If clients can sort by a non-unique field, the API still appends a tiebreaker for cursor stability, whether or not it is displayed in public metadata. + +Reject a client attempt to mark an arbitrary field as the cursor tiebreaker. That is server-owned configuration. + +## Field selection + +Support only what the response model can honor: + +```text +fields=id,name,status +fields[accounts]=id,name +``` + +Define wildcard, default, duplicate, unknown-field, and cross-resource behavior. Keep relationship inclusion separate from scalar field selection. Always load internal fields needed for authorization/cursor construction, then omit them from the response if they are not public. + +Never make `*` mean “every current database column.” Expand against an explicit public registry, so a newly added secret/internal column cannot become public. + +## Pagination and cursors + +Expose cursor and offset as different contracts, not interchangeable fields. + +Cursor response example: + +```json +{ + "data": [], + "pagination": { + "type": "cursor", + "limit": 20, + "next": "opaque-token", + "previous": null, + "expiresAt": "2026-07-18T14:22:33Z" + } +} +``` + +Define: + +- stable sort and tiebreaker; +- opaque authenticated token policy; +- TTL and expired status (the retained finance utility maps expiry to 410; confirm this public choice); +- invalid/tampered/cross-endpoint status; +- binding to filters, sort, resource, API version, and relevant authority context; +- next and previous direction semantics; +- behavior under inserts, deletes, and updates; +- page size and `limit + 1` continuation detection. + +The retained cursor schema contains sort field/value, tiebreaker/value, direction, and created time. Its inspected shape does not include a filter/resource digest. Do not copy it unchanged for a contract where cross-filter replay would be wrong. + +Offset/page pagination must define maximum offset and concurrent-write instability. It can be appropriate for bounded administration or snapshot-stable results. + +## Count and response metadata + +Publicly distinguish: + +- `exact`: same authoritative filtered relation; +- `planned`: planner estimate for the filtered query; +- `estimated`: coarse relation/table statistic; +- `none`: no total promised. + +Use names clients cannot mistake: + +```json +{"count":{"mode":"estimated","value":120000,"asOf":"2026-07-17T14:22:33Z"}} +``` + +Do not put an estimate in `total` if generated clients/UI treat it as exact. If page rows and count are not read from one snapshot, document that they may diverge under concurrent writes. + +Response contracts include status and headers as well as payload. Model success variants and stable problem details. A payload-only schema can pass while the actual tuple/wrapper/status is wrong. + +## Authorization and policy + +Authentication identifies a session/user; it does not authorize requested organization, family, account, or dataset. The endpoint should resolve current authority and pass server-owned constraints to execution: + +```text +valid session + -> current membership/permission + -> requested resource belongs to authorized organization + -> base query predicate + -> client filters within that scope +``` + +Do not accept `organization_id` as a normal client filter that can broaden results. Avoid counts that reveal out-of-scope records. Search/SPARQL projections cannot substitute for current membership unless the architecture explicitly owns and validates that authority. + +## Compatibility and OpenAPI + +Document the exact query syntax in OpenAPI using the capabilities of the chosen generator. Complex bracket notation or JSON-encoded filters may need examples and manual parameter definitions. Verify generated clients; some generators cannot represent arbitrary deep-object query syntax consistently. + +Treat these as possible breaking changes: + +- rename/remove field or operator; +- alter coercion, case, date/timezone, null, or empty semantics; +- change default sort/tiebreaker; +- change cursor version, signing, TTL, or context; +- change count mode/meaning; +- lower limits clients relied on; +- change operation ID or response metadata shape; +- change disabled parameters from reject to ignore or vice versa. + +Version cursor payloads and reject unsupported versions rather than guessing. + +## Integration example + +```ts +export const Middleware = [ + requireSession(), + requireOrganizationMembership(), + validate('query', ListAccountsQuery), +] + +export async function Handler(c: Context) { + const query = c.req.valid('query') + const authority = c.get('organizationAuthority') + const result = await accounts.list({ query, authority, signal: c.req.raw.signal }) + return c.json(toCollectionResponse(result), 200) +} +``` + +Names are illustrative. If the consumer uses Hono validators, middleware registration must match `c.req.valid(...)`. If another framework is selected, preserve the same ownership and cancellation path. + +## Failure and error mapping + +| Failure | Public contract | Internal evidence | |---|---|---| -| Exact | Correct total is required and affordable | Can dominate query cost | -| Planned | Approximation should reflect the filtered query | Planner estimate, not exact | -| Estimated | Coarse relation-level UI hint is enough | May ignore active filters | -| None | Current page/continuation is sufficient | No total pages | +| unknown field/operator | 400/422 stable validation problem | normalized issue path and endpoint config | +| malformed cursor/signature | 400 invalid cursor | safe reason, cursor version/key ID | +| expired cursor | documented 410 or selected status | age/TTL without token contents | +| wrong filter/resource/version cursor | 400 invalid context | expected/actual digest metadata safely | +| unauthorized organization/resource | 403 or non-disclosing 404 by policy | current membership decision and trace | +| query timeout/engine unavailable | stable timeout/unavailable problem | dependency, duration, retryability | +| result decoding defect | 500 generic problem | schema/driver cause, never raw values to client | + +## Test matrix + +Test: + +- query, JSON, and form sources only where endpoint declares them; +- source conflicts and repeated keys; +- every allowed/forbidden field/operator/type; +- null, arrays, delimiters, Unicode, wildcards, malformed dates/numbers; +- complexity, page, offset, and value-list limits; +- disabled components with supplied parameters; +- default and compound sorts with duplicates; +- cursor round trip, tampering, expiry, rotation, wrong route/filter/sort/version; +- inserts/deletes/updates between pages; +- exact/planned/estimated/no-count response semantics; +- fields defaults/wildcard/internal-field exclusion; +- current membership and cross-tenant adversarial requests; +- OpenAPI output and at least one generated/real client encoding; +- response status, headers, envelope, links, and problem variants; +- timeout/cancellation and dependency errors; +- semantic compatibility snapshots for defaults and operation IDs. + +## Executable verification + +Start the actual service, issue encoded requests with `curl` or the repository client, and inspect the generated OpenAPI document. Traverse a fixture collection through all cursor pages and assert every authorized ID appears once. Repeat while inserting/deleting records. Attempt cross-organization filters and cursors. Compare exact counts with the same authoritative predicate. Verify cancellation reaches the storage request. + +## Deliberate exclusions -Do not label a relation estimate as the exact filtered total. +- Do not force the finance query utility, Hono, Zod, Drizzle, or SPARQL library. +- Do not advertise operators the selected backend cannot implement consistently. +- Do not accept and ignore unsupported query parameters. +- Do not expose raw database fields through wildcard selection. +- Do not trust client tenant filters or cursor contents as authorization. +- Do not call estimated counts exact. +- Do not assume OpenAPI schema compatibility means semantic compatibility. +- Do not promise stable pagination without total ordering and context-bound cursors. -## Verification +## Sources and freshness -Test server base filters, allowed/forbidden fields, compound sort stability, -cursor round trips, empty pages, duplicates, concurrent writes, each count mode, -database errors, and the actual generated query or protocol request. +Grounded in the retained finance `utils/query` implementation and extensive tests, endpoint/validation/response utilities, service-module authoring guides, and PopModern search/SPARQL consumers, reviewed 2026-07-17. Exact finance utility exports are observed private/workspace evidence unless independently published. Cursor, validator, Hono, OpenAPI, and generated-client behavior is version-sensitive and must be tested at installed versions. diff --git a/skills/build-apis/references/runtime.md b/skills/build-apis/references/runtime.md index 16b8937..dda2268 100644 --- a/skills/build-apis/references/runtime.md +++ b/skills/build-apis/references/runtime.md @@ -1,34 +1,197 @@ -# Runtime, middleware, and resources +# HTTP runtime, middleware, errors, and resources -## Runtime adapters +## Contents -Hono portability does not imply every runtime adapter, middleware package, RPC -client, validator, streaming mode, websocket path, or OpenAPI integration has -identical behavior. Inspect the selected host and deployed entrypoint. +- [Adapter preflight](#adapter-preflight) +- [Middleware ownership and order](#middleware-ownership-and-order) +- [Request context](#request-context) +- [Error boundary](#error-boundary) +- [Resource lifetime](#resource-lifetime) +- [Timeout and cancellation](#timeout-and-cancellation) +- [Logging and tracing](#logging-and-tracing) +- [Security defaults](#security-defaults) +- [Verification](#verification) -## Resource ownership +## Adapter preflight -Construct long-lived database pools, auth instances, HTTP clients, logger sinks, -and workflow clients once at the host composition root. Adapt them into request -context without recreating them per request. Expose shutdown and drain behavior. +Hono's standards-oriented API does not make every host identical. Inspect the +actual deployed adapter for: -Avoid import-time configuration and logs. Importing a module should be safe in -tests, code generation, and tooling without production environment values. +- request/response streaming and disconnect signals; +- raw request access required by webhook signature verification; +- connection and execution duration limits; +- body size and multipart behavior; +- WebSocket/SSE support; +- remote address and trusted proxy handling; +- serverless instance/resource reuse; +- graceful shutdown hooks; +- TLS and HTTP protocol ownership. -## Defaults and security +Run adapter-specific integration tests. An `app.request(...)` test proves route +composition, not the host's networking or lifetime behavior. -Do not call a default “production optimized” while enabling wildcard CORS, -pretty JSON, verbose diagnostics, or development middleware. Defaults must be -explicitly justified by deployment and threat model. +## Middleware ownership and order + +Use three levels: + +| Level | Examples | Rule | +|---|---|---| +| Root | correlation, security headers, tracing, error boundary, access log | Install exactly once | +| Service/group | service capability context, shared org policy, rate limits | Mount on explicit prefix/group | +| Route | auth requirement, resource authorization, source validation | Keep visible beside handler | + +A typical order is: + +1. request ID/correlation and safe request context; +2. trusted proxy/origin/security policy; +3. tracing, timing, and access-log frame; +4. CORS where cross-origin browser access is intended; +5. session authentication; +6. capability adapters/resource context; +7. organization/resource authorization; +8. request-source validation; +9. handler; +10. centralized result/error normalization and final access record. + +The framework's onion execution order means “registered first” and “response +finalizes first” can differ. Test both request and response phases. Do not assume +nested Hono apps deduplicate middleware. + +## Request context + +Put guarantees, not arbitrary bags, into context: + +- parsed correlation/request ID; +- authenticated session or explicit anonymous marker; +- authorized organization/resource scope; +- request logger/span already bound to safe fields; +- capabilities or an Effect runner backed by host-owned resources; +- request deadline/disconnect signal. + +Make context types reflect middleware preconditions. A route that needs an +organization should not accept the optional environment type and then assert +non-null inside the handler. + +Do not store raw request bodies, auth headers, cookies, passwords, or unredacted +provider payloads in generic context or log properties. ## Error boundary -One root boundary owns unexpected failure logging, correlation ID, safe problem -mapping, and response completion. Handlers should not log a public failure and -then rethrow it into a second public log. +One root boundary owns unexpected failure completion. Translation sequence: + +```text +expected domain error + -> stable problem variant +unexpected typed infrastructure error + -> safe 5xx problem + structured redacted cause +defect / unknown throw + -> safe 500 + Cause/stack in controlled diagnostics +response already committed + -> abort/close stream + diagnostic; never attempt second response +``` + +Handlers can map expected domain errors close to the route, but they should not +emit the same final error log and rethrow into the root logger. Validation errors +normally produce 422 without running the handler. Authentication uses 401; +known authenticated-but-disallowed policy uses 403; hide resource existence with +404 only when that is the explicit security contract. + +Preserve one correlation ID in public problems and diagnostics. Redact before a +record reaches any sink, including JSON/file/telemetry exporters. + +## Resource lifetime + +| Lifetime | Resources | +|---|---| +| Process/host | pools, logger sinks, auth, workflow clients, DNS/TLS clients | +| Request | spans, authorized scope, transaction when truly request-wide | +| Operation | temporary files, one upload decoder, one provider request | +| Stream | subscription, encoder, heartbeat, backpressure queue | + +Construct host resources once and provide them through an explicit root. Avoid +module-scope acquisition because imports are used by tests, OpenAPI generation, +codegen, and CLIs. Export factories and pure registries from libraries. + +If a serverless adapter reuses isolates, cache only resources the adapter permits +and handles safely. Do not infer that a global pool is safe on every edge host. + +## Timeout and cancellation + +Distinguish: + +- client/request deadline; +- server handler timeout; +- database statement timeout; +- external provider connect/request timeout; +- workflow start RPC timeout; +- overall durable workflow/activity timeout. + +Propagate the earliest relevant deadline or AbortSignal. A timeout response does +not prove downstream work stopped. Connect cancellation to fetch/database/Effect +interruption where supported, or persist a cancel/abandon policy. + +Do not cancel durable work merely because the start request disconnects after +durable acceptance. Do cancel request-scoped queries, upload decoding, and SSE +delivery unless the contract says otherwise. + +## Logging and tracing + +If LogTape is the selected transport, use categories to separate stable results +from operational diagnostics and configure it once. HTTP APIs normally return +results as responses, while LogTape handles diagnostics; CLI raw result sinks are +a separate contract. + +Recommended request diagnostic fields: + +- correlation, trace, and request IDs; +- method and route template, not sensitive raw URL; +- service/operation and status; +- duration and response bytes where available; +- authenticated actor/organization only under policy; +- failure tag, retry/cancellation classification; +- workflow run ID for accepted work. + +Avoid logging entire request/response bodies. Sample intentionally and apply +redaction before serialization. Make replay-aware logging choices inside durable +workflow runtimes so replay does not duplicate operational events. + +## Security defaults + +Production defaults are explicit. Do not label a preset secure while it enables +wildcard credentialed CORS, verbose errors, pretty development middleware, +trusted arbitrary proxy headers, unlimited bodies, or public diagnostics. + +Define: + +- exact allowed origins, methods, and credential behavior; +- trusted proxy count/ranges; +- security headers and CSP ownership; +- body/file limits and accepted content types; +- cookie security and CSRF policy; +- rate/admission limits at the correct identity scope; +- error-detail/redaction policy; +- health and metrics exposure. + +## Verification + +- Assert middleware invocation and order for success, validation, auth, handler + failure, and committed streaming response. +- Test malformed proxy/origin/request IDs. +- Cancel a live database/provider call on disconnect. +- Exhaust the pool and verify bounded safe failure. +- Run concurrent requests against one shared resource graph. +- Import registries without env or network access. +- Start and drain the real adapter. +- Verify one correlated diagnostic per public failure and zero secret leakage. +- Test production CORS, body limit, timeout, and security-header configuration. -## Operational verification +## Sources and freshness -Run concurrent requests, cancellation/disconnect, graceful shutdown, pool -exhaustion, provider timeout, malformed response, and middleware exception tests. -Assert resources close and every public failure has one correlated diagnostic. +- Hono official documentation: https://hono.dev/docs/ (primary source, checked + 2026-07-17; runtime adapter and streaming support are version-sensitive). +- Effect official documentation: https://effect.website/docs/ (primary source; + interruption/runtime APIs require installed-version verification). +- LogTape official documentation: https://logtape.org/ (primary source; use only + when the repository selects LogTape as transport). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/server/` and + `utils/middleware/` (observed source and request tests). diff --git a/skills/build-apis/references/service-modules.md b/skills/build-apis/references/service-modules.md index 5611f30..3b96dd6 100644 --- a/skills/build-apis/references/service-modules.md +++ b/skills/build-apis/references/service-modules.md @@ -1,53 +1,397 @@ -# Service module architecture +# Service-module architecture -## Suggested ownership +## Contents -- `definition.ts` owns method, route, input schemas, response/result schemas, - metadata, and OpenAPI description. -- `handler.ts` owns endpoint behavior against explicit services. -- Group and service `mod.ts` files aggregate endpoint definitions. -- Service `index.ts` aggregates handlers, constructs long-lived dependencies, - creates one root app, and registers definitions dynamically or statically. -- `workflows/` owns business orchestration; `runtime/` owns worker or engine - bootstrap. Avoid an ambiguous `orchestrator/` bucket. +- [Outcome](#outcome) +- [Terms and ownership](#terms-and-ownership) +- [Target module shape](#target-module-shape) +- [Definition and handler contract](#definition-and-handler-contract) +- [Group and service registries](#group-and-service-registries) +- [One composition root](#one-composition-root) +- [Domain and data seams](#domain-and-data-seams) +- [Workflow ownership](#workflow-ownership) +- [OpenAPI and reachability](#openapi-and-reachability) +- [Independent deployment](#independent-deployment) +- [Deliberate exclusions](#deliberate-exclusions) +- [Failure signatures](#failure-signatures) +- [Tests and verification](#tests-and-verification) +- [Evidence boundary](#evidence-boundary) -Names may differ, but each contract needs one owner. +## Outcome -## Reachability trace +A service module is complete when a reviewer can find its public contracts, +middleware guarantees, behavior, domain capabilities, data access, resources, +registration, deployment entrypoint, and executable requests without reverse +engineering a generic framework. A file named `definition.ts` or a generated +OpenAPI operation is not implementation evidence by itself. + +## Terms and ownership + +| Term | Owns | Does not own | +|---|---|---| +| Service | One top-level runtime/deployment boundary such as accounts or ledger | Every related concept in the product | +| Service module | A coherent endpoint/domain slice inside a service | A separate server by default | +| Endpoint definition | Static transport contract and documentation | Business behavior or resource construction | +| Endpoint handler | Translation from validated transport input to a capability call | Database construction, global middleware, or workflow worker boot | +| Domain capability | Business operation and typed domain failures | Hono context or HTTP response tuples | +| Data adapter | Queries and persistence against an explicit client/transaction | Tenant authorization policy unless the policy is encoded as a required scope | +| Composition root | Configuration, resources, Layer graph, app construction, registration, shutdown | Domain policy | + +Use “service module” consistently. A service is a top-level executable unit; +`accounts/preferences` can be a module within the `accounts` service. Do not +split a module into a network service solely because it has a folder. + +## Target module shape + +Start with the smallest shape that owns the current behavior: ```text -definition - -> exported group/service registry - -> handler registry - -> root server registration - -> middleware chain - -> deployed adapter - -> executable request +services/accounts/ +├── endpoints/ +│ └── preferences/ +│ ├── get/ +│ │ ├── definition.ts +│ │ └── handler.ts +│ ├── update/ +│ │ ├── definition.ts +│ │ └── handler.ts +│ ├── mod.ts +│ └── index.ts +├── domain/ +│ ├── preferences.ts +│ └── errors.ts +├── data/ +│ └── preferences.ts +├── mod.ts +└── index.ts +``` + +Add durable-work folders only when the service owns durable behavior: + +```text +services/imports/ +├── endpoints/ +├── workflows/ +├── activities/ +├── runtime/ +├── domain/ +├── data/ +├── mod.ts +└── index.ts +``` + +- `endpoints/` owns HTTP, webhook, admin, status, signal, and cancellation + adapters. +- `workflows/` owns replay-safe business coordination. +- `activities/` owns side effects called by workflows. +- `runtime/` owns engine adapters, registrations, and worker bootstrap. +- `domain/` owns business schemas, decisions, and typed errors. +- `data/` owns storage-specific queries and mappings. + +Do not create an `orchestrator/` bucket. It obscures whether code is business +coordination, an engine adapter, a worker host, or ordinary domain behavior. + +## Definition and handler contract + +`definition.ts` should answer: + +- stable endpoint name; +- method and relative route; +- schemas for every request source actually read; +- complete response variants; +- public description, operation identifier, tags, and security requirements; +- whether the route is synchronous or accepts durable work. + +Prefer a definition helper that retains literal route and method types: + +```ts +import { z } from "zod" + +export const Params = z.object({ family_id: z.uuid() }) +export const Json = z.object({ currency: z.string().length(3) }) +export const Result = z.object({ family_id: z.uuid(), currency: z.string() }) + +export const Definition = defineEndpoint({ + name: "update-family-preferences", + method: "PATCH", + route: "/families/:family_id/preferences", + schemas: { Param: Params, Json }, + responses: { + 200: jsonResponse(Result, "Preferences updated"), + 401: problemResponse("unauthorized"), + 403: problemResponse("forbidden"), + 404: problemResponse("family-not-found"), + 422: validationProblemResponse, + }, + security: [{ session: [] }], +}) ``` -A definition or handler file that never reaches the root registry is not an -implemented endpoint. Do not silently skip missing handlers with a warning while -advertising the definition as available. +The exact helper names are repository-specific. Preserve these semantics even +when a framework uses a different shape. + +`handler.ts` should read only middleware-guaranteed state, call a capability, +and translate its result: + +```ts +export const Middleware = [ + requireSession(), + requireOrganization({ param: "family_id" }), + createValidator("param", Definition.schemas.Param), + createValidator("json", Definition.schemas.Json), +] + +export const Handler: EndpointHandler = async (c) => { + const session = c.get("session") + const { family_id } = c.req.valid("param") + const input = c.req.valid("json") + const preferences = await c.get("accounts").updatePreferences({ + actorId: session.user.id, + familyId: family_id, + input, + }) + + return c.json(...ok(preferences)) +} +``` + +Do not let `Definition.schemas` imply validation that the route never registers. +Do not read raw request input after a validated source exists. + +## Group and service registries + +Use explicit registries so definitions and handlers can be compared at startup: + +```ts +export const PreferenceDefinitions = [ + GetPreferenceDefinition, + UpdatePreferenceDefinition, +] as const + +export const PreferenceHandlers = { + "get-family-preferences": GetPreferenceHandler, + "update-family-preferences": UpdatePreferenceHandler, +} satisfies HandlerRegistry +``` + +The group `index.ts` registers each definition with its handler and middleware on +an app supplied by the service. The service `mod.ts` exports its definition +catalog without constructing resources. The service `index.ts` builds or mounts +the app exactly once. + +Startup validation must reject: + +- a duplicate definition name or method/path pair; +- a definition without a handler; +- a handler without a definition; +- a referenced schema source without matching middleware; +- an operation documented as enabled but disabled at runtime; +- an endpoint that requires a capability absent from the service Layer. + +Silently skipping a missing handler makes generated documentation lie. ## One composition root -Call the server factory once per deployment boundary. Nested groups should mount -routes or middleware on the existing app, not create another root that repeats -correlation, logging, CORS, error, or auth middleware. +The host entrypoint owns this sequence: + +1. resolve and validate configuration; +2. construct long-lived resources; +3. assemble the Effect Layers or explicit capability object; +4. create one Hono root for this deployment boundary; +5. install root middleware once; +6. register service groups and endpoint middleware; +7. validate the registry/OpenAPI/reachability contract; +8. start listening and publish readiness; +9. on shutdown, stop admission, drain, close resources, and flush sinks. + +```ts +export async function startAccountsHost(input: AccountsHostInput) { + const config = AccountsConfig.parse(input.config) + const resources = await makeAccountsResources(config) + const runtime = ManagedRuntime.make(AccountsLive(resources)) + const app = createServer({ logger: resources.logger }) + + registerAccountsEndpoints(app, { runtime }) + assertEndpointRegistry(app, AccountsDefinitions) + + return serve({ + app, + port: config.port, + close: async () => { + await runtime.dispose() + await resources.close() + }, + }) +} +``` + +This is illustrative, not a mandate to use `ManagedRuntime`. The invariant is +one owned runtime/resource graph with explicit lifetime. + +Nested route groups must not call the server factory again. Recreating the root +commonly duplicates CORS, correlation IDs, request logs, error boundaries, and +resource construction. + +## Domain and data seams + +Handlers depend on capabilities, not concrete clients. A capability should use +domain terms and accept the policy facts it needs: + +```ts +export interface AccountsService { + readonly updatePreferences: (input: UpdatePreferencesInput) => + Effect.Effect +} +``` + +The data adapter receives a server-owned scope rather than an arbitrary client +filter: + +```ts +export interface FamilyScope { + readonly actorId: string + readonly familyId: string + readonly membershipId: string +} + +export interface PreferencesStore { + readonly update: ( + scope: FamilyScope, + input: PreferencesPatch, + ) => Effect.Effect +} +``` + +Do not name every data module `repository` automatically. A direct query module, +store, gateway, or adapter can be clearer. Do not leak Drizzle rows or provider +errors across the domain boundary. + +## Workflow ownership + +A synchronous handler may call a domain capability directly. A durable start +handler should validate and authorize, persist/submit through the workflow +control plane, and return an accepted result with stable inspection links: + +```ts +const acceptedRun = await workflows.start({ + workflow: "publish-import", + idempotencyKey: `${organization.id}:${upload.id}`, + input: { organizationId: organization.id, uploadId: upload.id }, +}) + +return c.json(...accepted({ + run_id: acceptedRun.id, + status_url: `/imports/${acceptedRun.id}`, + events_url: `/imports/${acceptedRun.id}/events`, +})) +``` + +The HTTP request does not keep the work alive. The accepted durable record or +engine command must exist before the 202 response. Business services own product +routes and authorization even when a shared workflow host owns execution. + +## OpenAPI and reachability + +OpenAPI generation consumes the same endpoint registry that runtime registration +consumes. It must include: + +- stable operation IDs; +- every request source and content type; +- every success and known problem variant; +- auth/security scheme requirements; +- pagination/cursor contracts; +- 202 status and status/cancel/event links for durable starts; +- examples that parse under the schemas. + +Then compare three sets: + +```text +declared operations == registered runtime operations == request-tested operations +``` + +Generated schema equality is insufficient. Send actual requests to the composed +app and parse actual responses. + +## Independent deployment + +A service is independently deployable only if its artifact declares and tests: + +- executable entrypoint and host adapter; +- config schema and required secrets; +- database, auth, workflow, object-storage, and network dependencies; +- migration ownership and compatibility window; +- readiness versus liveness checks; +- worker processes or task queues it requires; +- startup validation and graceful shutdown; +- public/internal ports and base paths; +- OpenAPI and event contract versions; +- deployment ordering and rollback compatibility. + +Independent deployability does not require deploying every service separately. +It means boundaries are explicit enough that a separate deployment is possible +without importing a monolithic hidden composition root. + +## Deliberate exclusions + +- Do not create a microservice for every endpoint group. +- Do not place runtime construction in `mod.ts` barrel exports. +- Do not expose a generic workflow API that bypasses service authorization. +- Do not generate handler behavior from OpenAPI unless the generated runtime is + the reviewed source of truth. +- Do not treat a workspace glob, future-plan document, or empty directory as an + implemented service. +- Do not force Effect into a simple pure helper; use it where typed failures, + resources, concurrency, cancellation, or composition justify it. + +## Failure signatures + +| Signature | Likely defect | Next proof | +|---|---|---| +| Definition exists but request returns 404 | Registry or deployed adapter omission | Trace definition to composed app and real request | +| Missing handler logs a warning | Partial registry accepted | Fail startup or mark capability unavailable | +| Middleware executes twice | Nested root construction | Count middleware invocation in request test | +| Import requires production env | Module-scope resource construction | Import test with empty env | +| Handler imports DB singleton | Composition boundary bypassed | Inject capability/client from root | +| OpenAPI advertises only 200 | Error and async contracts omitted | Compare actual status variants | +| One service cannot start alone | Hidden config/resource dependency | Minimal host fixture | +| Worker package exists but route uses legacy start | Durable path unreachable | Route-to-worker reachability test | +| Tenant ID appears only in user filter | Authorization delegated to caller | Server-owned scope/base constraint | + +## Tests and verification + +Required test layers: + +1. definition schema tests for valid and invalid sources; +2. domain capability tests without HTTP; +3. adapter tests against the real database/protocol where risk warrants it; +4. registry conformance tests; +5. composed request tests with real middleware; +6. OpenAPI/runtime/request set equality; +7. auth and cross-organization isolation tests; +8. startup, readiness, drain, and shutdown tests; +9. standalone service artifact boot test; +10. durable-start reachability test when workflows are present. + +Verification commands are repository-specific, but the evidence must include a +typecheck, focused tests, generated-contract validation, a standalone boot, and +at least one executable request for every operation changed. -## Middleware ownership +## Evidence boundary -Use an explicit order such as: +The uploaded service-module authoring guides define the target architecture. The +uploaded utility packages provide executable endpoint schemas, Standard Schema +validation, RFC problem helpers, query utilities, server construction, and a +workflow control-plane seam. Some referenced service directories are absent from +the supplied archives, and the SQL-backed Effect workflow adapter explicitly +returns not-implemented failures. Treat those surfaces as design evidence, not +proof of production reachability. -1. request identity/correlation and context; -2. trusted proxy/origin and security policy; -3. logging and timing; -4. CORS where required; -5. authentication/session loading; -6. service dependency adapters; -7. route authorization and validation; -8. handler; -9. centralized error/response normalization. +## Sources and freshness -The exact order is repository-specific. Test ordering rather than assuming Hono -or another framework deduplicates repeated middleware. +- Attachments, verified 2026-07-17: `evidence/app/better-auth/docs/service-module-authoring.md`, + `docs/service-module-structure.md`, `utils/endpoint/`, `utils/middleware/`, and + `utils/server/` (normative plus observed source; target services are incomplete). +- Attachment, verified 2026-07-17: `evidence/app/new-finance/.agents/plans/services-and-service-modules.md` + and `utils/workflows/` (planned ownership plus executable/counterexample evidence). +- Hono official documentation: https://hono.dev/docs/ (version-sensitive adapter behavior; + verify against the installed release). diff --git a/skills/build-apis/references/streaming.md b/skills/build-apis/references/streaming.md new file mode 100644 index 0000000..66d58bf --- /dev/null +++ b/skills/build-apis/references/streaming.md @@ -0,0 +1,252 @@ +# SSE and streaming API contracts + +## Contents + +- [When SSE fits](#when-sse-fits) +- [Authority and event identity](#authority-and-event-identity) +- [Wire contract](#wire-contract) +- [Connection procedure](#connection-procedure) +- [Replay and cursors](#replay-and-cursors) +- [Backpressure and slow consumers](#backpressure-and-slow-consumers) +- [Cancellation and cleanup](#cancellation-and-cleanup) +- [Authentication and authorization](#authentication-and-authorization) +- [Proxies, heartbeats, and deployment](#proxies-heartbeats-and-deployment) +- [Failure signatures](#failure-signatures) +- [Tests and verification](#tests-and-verification) + +## When SSE fits + +Use Server-Sent Events for an ordered server-to-client event feed over HTTP when +the browser does not need bidirectional frames on the same connection. Common +cases include workflow timelines, import progress, report completion, and +monitoring updates. + +Choose another mechanism when: + +- the client must stream substantial data to the server; +- bidirectional low-latency messages are fundamental; +- binary frames are required; +- a durable broker subscription, not an HTTP projection, is the public contract; +- polling is simpler and meets latency/load requirements. + +SSE is a delivery transport, not durability. A durable event/timeline store must +exist if reconnect and replay are promised. + +## Authority and event identity + +Before writing an endpoint, specify: + +- authoritative event source: append-only timeline, outbox, engine history, or + ephemeral pub/sub; +- stream scope: workflow run, organization, user, or filter; +- monotonically ordered cursor within that scope; +- retention and replay window; +- event schema/version; +- visibility policy for every event; +- behavior when cursor is unknown, expired, from another scope, or ahead. + +Prefer an opaque cursor externally even if the implementation currently uses a +timeline sequence. Bind or validate the cursor against the authorized scope so a +cursor cannot move a client into another organization's stream. + +## Wire contract + +An SSE response uses `text/event-stream`. Each event is separated by a blank +line; multiline data uses one `data:` line per line. Keep JSON payloads on one +serialized line when possible: + +```text +id: eyJydW5faWQiOiJydW5fMTIzIiwic2VxIjo0Mn0 +event: workflow.progress.v1 +data: {"run_id":"run_123","stage":"detect","completed":120,"total":500} + +``` + +Define a stable envelope: + +```ts +export const WorkflowStreamEvent = z.discriminatedUnion("type", [ + z.object({ + type: z.literal("workflow.progress.v1"), + sequence: z.int().nonnegative(), + occurred_at: z.iso.datetime(), + data: z.object({ stage: z.string(), completed: z.int(), total: z.int().nullable() }), + }), + z.object({ + type: z.literal("workflow.completed.v1"), + sequence: z.int().nonnegative(), + occurred_at: z.iso.datetime(), + data: z.object({ result_url: z.string().url().optional() }), + }), +]) +``` + +Do not send internal engine records, raw errors, secrets, or unrestricted result +payloads merely because the stream is authenticated. + +## Connection procedure + +Use this ordering to avoid a subscribe/replay race: + +1. authenticate and authorize the requested stream scope; +2. validate cursor and filter; +3. establish the live subscription or capture a high-water mark; +4. replay `(cursor, high-water]` from the durable source; +5. forward live events after the high-water mark; +6. emit periodic heartbeat comments if infrastructure needs them; +7. close after terminal state when the contract is run-scoped, or remain open for + an explicitly continuous feed. + +The exact subscribe-before-replay algorithm depends on the event source. The +invariant is no event lost between the replay query and live subscription. + +Pseudo-implementation: + +```ts +return streamSSE(c, async (stream) => { + const scope = await authorizeRun(c, c.req.param("run_id")) + const cursor = await cursors.parse(c.req.header("Last-Event-ID"), scope) + + await using subscription = await events.subscribe(scope) + const highWater = await events.highWater(scope) + + for await (const event of events.replay(scope, cursor, highWater)) { + await stream.writeSSE(encode(event)) + } + + for await (const event of subscription.after(highWater)) { + await stream.writeSSE(encode(event)) + if (event.terminal) break + } +}) +``` + +Treat `await using` and helper names as illustrative. Implement cleanup using the +runtime/framework's real APIs. + +## Replay and cursors + +Browsers can send the last received ID as `Last-Event-ID` when reconnecting. A +custom client may use an explicit query/header cursor. Pick one public contract +and document precedence rather than accepting conflicting cursors silently. + +Cursor decisions: + +| Condition | Recommended behavior | +|---|---| +| No cursor | Start at current snapshot/high-water or retained beginning, as documented | +| Valid cursor | Replay strictly after it | +| Cursor belongs to another scope | Reject without revealing that scope | +| Expired cursor | Return explicit 409/410 resync problem or emit a reset event | +| Cursor ahead of high-water | Reject as invalid | +| Duplicate delivery after reconnect | Client deduplicates by event ID; server maintains stable IDs | + +If state snapshots are cheaper than full history, send a versioned snapshot and +then events after its high-water mark. Do not call an ephemeral “current status” +query replay. + +## Backpressure and slow consumers + +Bound every queue between the durable source and socket. A slow browser must not +cause unbounded process memory. + +Define: + +- per-connection queue capacity; +- per-organization connection and throughput limits; +- maximum event size; +- write timeout; +- slow-consumer behavior: disconnect with resumable cursor, coalesce replaceable + progress events, or drop explicitly non-authoritative heartbeats; +- fairness between consumers; +- retention pressure and compaction behavior. + +Never drop terminal, audit, or state-transition events without a durable replay +path. Progress samples may be coalesced only when the public contract says they +are replaceable and the eventual authoritative state remains available. + +## Cancellation and cleanup + +Client disconnect cancels delivery, not necessarily the underlying workflow. +Expose workflow cancellation as an authorized explicit command. + +On stream cancellation: + +- detach broker/listener subscriptions; +- interrupt pending replay reads and heartbeat fibers; +- release database cursors and timers; +- stop writes immediately; +- record metrics without logging routine disconnects as server failures; +- preserve the last durable cursor, not an optimistic unsent cursor. + +Connect `Request.signal` or the framework disconnect hook to the scoped stream +program. Test cancellation while replay is blocked and while a write is blocked. + +## Authentication and authorization + +Authenticate before opening the stream. Re-authorize the requested run/resource +against server-owned organization membership. Decide whether long-lived streams +must revalidate session/membership during the connection or only on reconnect. + +Avoid bearer tokens in query strings because URLs leak into history, logs, and +referrers. Browser `EventSource` has header constraints; prefer same-origin secure +cookies, a short-lived scoped stream ticket, or a fetch-based SSE client where +custom authorization headers are required. + +Bind stream tickets to user, organization, resource, expiry, and one intended +audience. A ticket is not a general API token. + +## Proxies, heartbeats, and deployment + +Verify the real host and every proxy/CDN hop supports streaming without buffering. +Configure: + +- response buffering disabled; +- idle timeout longer than heartbeat interval; +- compression behavior tested (some stacks buffer compressed chunks); +- cache disabled; +- connection limits and HTTP protocol behavior; +- deploy/drain behavior for active streams; +- sticky routing only if the live source truly requires it. + +Use comment heartbeats such as `: keep-alive\n\n` when needed. Heartbeats prove +connection liveness, not workflow progress, and should not advance the durable +cursor. + +## Failure signatures + +| Signature | Likely defect | Correction | +|---|---|---| +| Reconnect misses one event | Replay/live subscription race | High-water handoff | +| Memory grows per client | Unbounded delivery queue | Capacity and slow-consumer policy | +| Duplicate progress after reconnect | IDs unstable or replay inclusive | Stable ID and strictly-after cursor | +| Another org's events appear | Cursor/scope not bound to authorization | Scope-aware cursor and base filter | +| Stream works locally, batches in production | Proxy buffering/compression | End-to-end host test | +| Workflow cancels when browser closes | Transport and domain cancellation conflated | Explicit workflow cancel command | +| DB connections never return | Subscription/cursor not scoped | Disconnect finalizer test | +| `Last-Event-ID` accepted from wrong run | Opaque cursor decoded without scope check | Reject safely | + +## Tests and verification + +- Parse every emitted event against its versioned schema. +- Connect without auth, with expired auth, and across organizations. +- Replay from every retained cursor boundary and from expired/ahead/wrong-scope + cursors. +- Inject an event at the replay/live handoff and prove it arrives once or is + safely deduplicable. +- Stall reads and verify memory remains bounded and policy activates. +- Disconnect during replay, live wait, heartbeat, and blocked write; assert all + resources close. +- Test terminal events and whether the stream closes as documented. +- Run through the production proxy/CDN with buffering and timeouts observed. +- Restart the API host and prove durable replay still works. + +## Sources and freshness + +- HTML Living Standard, server-sent events: https://html.spec.whatwg.org/multipage/server-sent-events.html + (primary protocol source, checked 2026-07-17). +- Hono streaming helper documentation: https://hono.dev/docs/helpers/streaming + (primary framework source; verify selected runtime adapter behavior). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/` + and `evidence/web/kaiju-site-scope/` (workflow/status and product authorization + context; the full SSE design is inferred and must be executable-tested). diff --git a/skills/build-clis/SKILL.md b/skills/build-clis/SKILL.md index e287c3b..53fd891 100644 --- a/skills/build-clis/SKILL.md +++ b/skills/build-clis/SKILL.md @@ -7,8 +7,9 @@ description: Design, implement, refactor, review, diagnose, test, package, or re `deliver-software` owns request authority and repository completion. `deno-software` owns Deno manifests, tasks, permissions, compilation, and -publication. `explore-ecosystems` owns dependency topology. This skill owns the -CLI's public language and operational contract. +publication. `explore-ecosystems` owns dependency topology. `build-libraries` +owns reusable programming models extracted from or consumed by the CLI. This +skill owns the CLI's public language and operational contract. ## Evidence preflight @@ -73,6 +74,45 @@ invent the connection. 10. Preserve authored Markdown layout. Never run a broad formatter over CLI guidebooks, tables, or manuals unless the user explicitly requests it. +## Production stack doctrine + +When a CLI uses Optique, c12, defu, LogTape, and Zod together, assign one owner +per boundary: + +| Boundary | Owner | Failure to reject | +|---|---|---| +| Token grammar, choices, aliases, suggestions, completion, and manuals | Optique | Handwritten help or post-parse boolean reconciliation | +| Help-only default visibility | Optique document metadata | Parser defaults that materialize sparse source values | +| Project config discovery, formats, `extends`, env branches, and factories | c12 | Strict runtime validation before loader metadata is consumed | +| Recursive merge mechanics | defu behind an app merger | Public `defu(cli, env, file)` with accidental array concatenation | +| Runtime defaults, transforms, and external data contracts | Zod or another schema adapter | Defaults on sparse authoring or CLI patch schemas | +| Results, diagnostics, bootstrap errors, redaction, and sink lifecycle | LogTape at the executable boundary | `console.*`, duplicate loggers, or library-owned sink setup | + +Use Optique 1.2 features when they clarify public behavior: +`negatableFlag()` for tri-state Boolean overrides, `choice()` for schema-backed +enumerations, `deferredValue()` only for handler-time fallback functions, and +`runProgram()` hooks for per-command resources such as a single resolved config +snapshot and logger. Do not use `deferredValue()` as a replacement for ordinary +schema defaults. + +The defaulting rule is strict: authored config, environment, and CLI patches +stay sparse; documented defaults can appear in help; executable defaults apply +once in the complete runtime schema. Zod `.default()` is for already-valid +output defaults, `.prefault()` is for fallbacks that must pass through +transforms, and `.catch()` is error recovery rather than normal configuration. +`@optique/zod` adapts one CLI value to Zod validation; it does not transfer +multi-source precedence or runtime-default ownership to Optique. + +Record provenance while resolving source layers. Do not reconstruct it from the +final value: an explicit CLI value can equal the inherited config value and must +still remain the winner. Version and redact the provenance envelope before +rendering it or passing it through LogTape. + +Load dynamic project configuration once per invocation. If Optique source +contexts need a c12 snapshot and the handler also needs final config, thread +that snapshot through `runProgram()` hooks or an equivalent composition-root +resource. Do not let handlers call the full resolver again. + ## Reference routing - [audit.md](references/audit.md): repository audit, observed-versus-promised @@ -81,22 +121,56 @@ invent the connection. capability boundaries, and composition choices. - [commands.md](references/commands.md): command grammar, Optique, schemas, help, completion, manuals, aliases, and deprecation. +- [optique.md](references/optique.md): complete Optique package map, typed + grammar, source contexts, schema, prompt, logging, time, Git, discovery, + completion, manual, and runner integration patterns. +- [five-library-stack.md](references/five-library-stack.md): load when Optique, + c12, defu, LogTape, and Zod interact; includes source-flow maps, + wrong/right examples, default taxonomy, one-snapshot execution, and test + matrix. +- [defaults-provenance.md](references/defaults-provenance.md): detailed + three-schema model, `@optique/zod`, `.default()` versus `.prefault()`, nested + defaults, source resolution, field decisions, provenance envelopes, and + implementation tests. - [config.md](references/config.md): c12/defu, authored and runtime shapes, precedence, arrays, atomic unions, operations, and provenance. +- [c12-defu.md](references/c12-defu.md): detailed c12, defu, and jiti loading + lifecycle, merge algebra, dynamic factories, extension layers, provenance, + mutation, version boundaries, tests, and failure diagnosis. - [output.md](references/output.md): LogTape results, diagnostics, artifacts, redaction, renderers, sinks, and lifecycle. +- [logtape.md](references/logtape.md): complete LogTape category, sink, filter, + formatter, context, redaction, testing, bootstrap, and disposal manual. - [interaction.md](references/interaction.md): no-argument behavior, prompts, streams, secrets, TTYs, paging, progress, risk, and recovery. - [lifecycle.md](references/lifecycle.md): cancellation, resources, checkpoints, reconciliation, exit status, and error ownership. - [testing.md](references/testing.md): layered, subprocess, generated-surface, cancellation, packaging, and clean-machine verification. +- [benchmarking.md](references/benchmarking.md): correctness-gated mitata and + installed-artifact benchmarks for parsing, config loading, merging, Zod, + provenance, LogTape, startup, memory, and regression reporting. - [distribution.md](references/distribution.md): manifests, compiled assets, package contents, installation, upgrade, and uninstall. - [ecosystems.md](references/ecosystems.md): capability map for Optique, LogTape, c12/defu, schema tools, prompts, Temporal, and UnJS companions. -- [casebook.md](references/casebook.md): worked patterns and failures grounded - in the attached Kaiju CLI repository. +- [unjs.md](references/unjs.md): focused UnJS capability map, ownership + boundaries, package combinations, exclusions, and integration sequences. +- [unjs-runtime-config.md](references/unjs-runtime-config.md): load for jiti, + c12, defu, destr, confbox, pkg-types, pathe, or ufo implementation. +- [unjs-fetch-state.md](references/unjs-fetch-state.md): load for ofetch, + unstorage, ohash, or Hookable implementation and version boundaries. +- [unjs-build-release.md](references/unjs-build-release.md): load for unbuild, + nypm, Magicast, giget, changelogen, automd, rc9, or std-env implementation. +- [integration.md](references/integration.md): worked end-to-end sequences that + join parsing, source resolution, logging, execution, recovery, and release. +- [casebook.md](references/casebook.md): load for codebase-grounded Kaiju + traces and portable CLI policy maps covering command-language governance, + human versus machine output, dangerous-operation plans, standard streams, + secrets, paging, consentful config edits, telemetry, root program resources, + sparse adapters, config/provenance, browser/Common Crawl/WARC/domains flows, + stage/artifact boundaries, lifecycle/concurrency, field-extension checklists, + failure signatures, and behavior-to-test matrices. ## Completion gate @@ -104,4 +178,7 @@ Do not call CLI work complete until the real invocation path has been exercised for success, invalid input, operational failure, and cancellation as applicable; stdout, stderr, exit status, and artifacts match their contracts; generated surfaces agree; and the packaged or compiled entrypoint runs in a clean context. -Report blocked checks separately from failed checks. +Report blocked checks separately from failed checks. If the change makes a +performance claim, run a named representative benchmark against a recorded +baseline and report absolute change, relative change, variability, and semantic +oracles. diff --git a/skills/build-clis/references/benchmarking.md b/skills/build-clis/references/benchmarking.md new file mode 100644 index 0000000..87d7c5e --- /dev/null +++ b/skills/build-clis/references/benchmarking.md @@ -0,0 +1,273 @@ +# CLI benchmarking protocol + +Benchmark only after behavioral tests pass. A faster resolver that changes +precedence, provenance, redaction, output bytes, or cancellation is a defect. + +## Contents + +1. [Questions worth measuring](#questions-worth-measuring) +2. [Workload model](#workload-model) +3. [Startup and steady state](#startup-and-steady-state) +4. [Mitata harnesses](#mitata-harnesses) +5. [Merge and provenance fixtures](#merge-and-provenance-fixtures) +6. [Logging benchmarks](#logging-benchmarks) +7. [End-to-end artifact benchmarks](#end-to-end-artifact-benchmarks) +8. [Controls and statistics](#controls-and-statistics) +9. [Regression policy](#regression-policy) +10. [Benchmark report](#benchmark-report) + +## Questions worth measuring + +Measure a boundary to answer a product question: + +| Boundary | Question | +|---|---| +| Optique grammar | Does command count or choice vocabulary make parse/help slow? | +| Source binding | What does env/config/derived binding add to an ordinary invocation? | +| c12 | What is discovery and TypeScript config evaluation latency? | +| defu-backed merger | How does layer count, depth, and array policy scale? | +| Zod | What do sparse validation, complete parsing, transforms, and refinements cost? | +| Provenance | What latency and allocation overhead does explanation add? | +| LogTape | What is disabled-log, pretty, JSON, redaction, and flush cost? | +| Executable | What do users experience for `--help`, success, and invalid input? | + +Do not collapse these into one microbenchmark and call it “CLI performance.” +Startup-heavy CLIs and long-running commands have different budgets. + +## Workload model + +Create named fixtures from observed usage rather than random objects: + +| Fixture | Suggested shape | +|---|---| +| `tiny` | 1 command, 1 layer, 10 scalar fields | +| `typical` | 12 commands, 4 layers, 60 fields, 3 arrays, 2 nested objects | +| `large` | 60 commands, 8 layers, 500 fields, deep nesting, 20 array operations | +| `adversarial` | Maximum supported depth/size, branch replacements, empty arrays, secrets | + +For each fixture, record: + +- command and option count; +- source layer count and precedence; +- scalar, object, array, and union-path counts; +- append/prepend/replace operation counts; +- Zod transforms and refinements; +- provenance enabled or disabled; +- logging level, formatter, sink, and redaction state. + +Keep fixtures deterministic and committed. A benchmark that regenerates +different data on every run cannot diagnose a regression. + +## Startup and steady state + +Separate these measurements: + +1. **Process startup:** spawn the installed/compiled entrypoint once per sample. +2. **Cold application path:** import modules, build grammar, discover config, + initialize logging, and execute once in a fresh process. +3. **Warm parse/resolution:** reuse constructed schemas and parser terms in one + process. +4. **Long-running throughput:** repeat only the operation relevant to a daemon, + watcher, or batch command. + +Never report a warm in-process parse as command startup. Never include fixture +generation or filesystem cleanup inside a merge-only timing. + +## Mitata harnesses + +Prefer mitata for Deno benchmarks when the repository has not standardized on +another runner. Construct fixtures outside the timed callback. + +```ts +import { bench, group, run } from "mitata"; +import { CliPatchSchema, RuntimeConfigSchema } from "#/config/schema.ts"; +import { mergeConfigInputs } from "#/config/merge.ts"; +import { typicalLayers } from "./fixtures/config-layers.ts"; + +group("config resolution: typical", () => { + bench("merge only", () => { + mergeConfigInputs(...typicalLayers); + }); + + bench("merge + complete Zod parse", () => { + RuntimeConfigSchema.parse(mergeConfigInputs(...typicalLayers)); + }); + + bench("sparse Zod parse, one CLI patch", () => { + CliPatchSchema.parse(typicalLayers[0]); + }); +}); + +await run(); +``` + +Compare provenance with the same algorithm and fixture: + +```ts +group("provenance overhead: typical", () => { + bench("value only", () => resolveConfig(typicalSources, { + provenance: false, + })); + + bench("value + decisions", () => resolveConfig(typicalSources, { + provenance: true, + })); +}); +``` + +If the production resolver always records provenance, do not create a fake +value-only production mode solely to win a benchmark. A test-only comparator is +acceptable when clearly labeled. + +## Merge and provenance fixtures + +Benchmark the algorithmic dimensions independently: + +```text +layers: 1, 2, 4, 8, 16 +object depth: 1, 4, 8, supported maximum +field count: 10, 100, 1_000 +arrays: replace, empty clear, append, prepend, replace operation +unions: no switch, repeated atomic branch switch +provenance: off/comparator, decisions only, decisions + redacted values +``` + +Every benchmark fixture must first pass the same semantic oracle used by unit +tests. Before timing, assert: + +- public highest-to-lowest precedence resolves correctly; +- internal lowest-to-highest operation evaluation is correct; +- `false`, `0`, empty string, and empty arrays retain their intended meaning; +- inputs are not mutated and resolved arrays are copied; +- atomic union branches do not hybridize; +- provenance winner, shadowed values, and operation order are correct. + +Watch for accidental quadratic behavior in repeated object cloning, path-string +construction, flattening, and shadowed-contribution arrays. Profile before +replacing clear code with mutation or custom data structures. + +## Logging benchmarks + +Measure LogTape configurations separately: + +| Case | Why | +|---|---| +| Below-threshold diagnostic | Hot-path cost when log is disabled | +| Structured record to null/recorder sink | Dispatch and filtering cost | +| Pretty stderr formatter | Human path | +| JSON formatter | Automation/collection path | +| Nested redaction | Security overhead on representative objects | +| Raw result sink | Exact stdout route without diagnostic formatting | +| Flush/dispose | Short-command shutdown cost | + +Do not benchmark real terminal rendering in a tight loop and interpret it as +formatter speed. Use an in-memory sink for formatter comparison; use subprocess +latency separately for actual stderr behavior. + +Never remove structured redaction because an unrepresentative benchmark says it +is expensive. First reduce logged object size, avoid logging full config trees, +or move verbose diagnostics behind an explicit level. + +## End-to-end artifact benchmarks + +Use the artifact users execute: + +```ts +Deno.bench("installed --help startup", async () => { + const command = new Deno.Command(installedExecutable, { + args: ["--help"], + stdin: "null", + stdout: "null", + stderr: "null", + }); + const output = await command.output(); + if (!output.success) throw new Error("--help failed"); +}); +``` + +Process-spawn benchmarks are noisy. Use enough samples, run them separately from +microbenchmarks, and report the operating system, architecture, runtime, +artifact type, and power state. Include at least: + +- `--version` or `--help` fast path with no project config requirement; +- invalid token path; +- typical successful command with config discovery; +- machine-output path; +- one config factory path if factories are supported. + +Use an isolated temporary project and deterministic local files. Network access +belongs in a separately labeled integration benchmark, never the default suite. + +## Controls and statistics + +Before comparing commits: + +- use the same Deno version, lockfile, permissions, machine, and power profile; +- close competing CPU-heavy work and record thermal throttling risk; +- pin deterministic fixture content and temporary-directory layout; +- warm caches only for warm cases; create fresh processes for cold cases; +- keep stdout/stderr away from an interactive terminal unless testing it; +- validate output and exit status outside or before the timed region; +- compare distributions, not one fastest sample; +- retain runner output as an artifact. + +Use median and a tail percentile for process startup. For stable +microbenchmarks, report the runner’s estimate and variability. A result inside +normal run-to-run noise is inconclusive. + +## Regression policy + +Define budgets from user impact and baseline variance. Example policy: + +| Metric | Review trigger | Failure trigger | +|---|---:|---:| +| Typical warm resolution | >10% with stable confidence | >20% | +| Installed `--help` median | >10% and >10 ms | Product-specific budget exceeded | +| Installed `--help` p95 | >15% | Product-specific budget exceeded | +| Typical peak memory | >15% | Supported-environment budget exceeded | +| Provenance overhead | Explain any material change | Explicit product budget exceeded | + +These are example thresholds, not universal standards. Calibrate them with at +least several baseline runs. Require both a relative and meaningful absolute +change for noisy, short operations. + +Do not accept a performance change that breaks a semantic oracle. If a change +trades memory for latency or startup for steady state, state both effects and +which command population benefits. + +## Benchmark report + +Record enough context to reproduce the claim: + +```ts +import * as z from "zod"; + +export const BenchmarkRecordSchema = z.object({ + schemaVersion: z.literal(1), + commit: z.string().min(7), + runtime: z.string(), + platform: z.string(), + architecture: z.string(), + artifact: z.enum(["source", "installed", "compiled"]), + fixture: z.string(), + metric: z.string(), + medianMs: z.number().nonnegative(), + p95Ms: z.number().nonnegative().optional(), + samples: z.number().int().positive(), + notes: z.array(z.string()).default([]), +}); +``` + +The handoff should state: + +1. Which semantic tests passed before benchmarking. +2. Exact benchmark command and fixture. +3. Baseline and candidate revisions. +4. Median, variability/tail, and sample count. +5. Absolute and relative change. +6. Whether the result is conclusive. +7. Any unmeasured claim, such as Windows startup or network behavior. + +See [testing.md](testing.md) for correctness gates and +[defaults-provenance.md](defaults-provenance.md) for the resolution invariants +that benchmarks must preserve. diff --git a/skills/build-clis/references/c12-defu.md b/skills/build-clis/references/c12-defu.md new file mode 100644 index 0000000..76155c1 --- /dev/null +++ b/skills/build-clis/references/c12-defu.md @@ -0,0 +1,642 @@ +# c12, defu, and jiti configuration manual + +## Contents + +- [Evidence and version boundaries](#evidence-and-version-boundaries) +- [Responsibility map](#responsibility-map) +- [Configuration shapes](#configuration-shapes) +- [Resolution stages](#resolution-stages) +- [c12 loading capabilities](#c12-loading-capabilities) +- [Two mergers](#two-mergers) +- [Dynamic factories and jiti](#dynamic-factories-and-jiti) +- [Precedence and evaluation order](#precedence-and-evaluation-order) +- [Merge algebra](#merge-algebra) +- [Declarative array operations](#declarative-array-operations) +- [Atomic unions and special fields](#atomic-unions-and-special-fields) +- [Provenance and inspection](#provenance-and-inspection) +- [Configuration mutation](#configuration-mutation) +- [Validation boundaries](#validation-boundaries) +- [Testing strategy](#testing-strategy) +- [Failure signatures](#failure-signatures) +- [Extension checklist](#extension-checklist) +- [Sources and freshness](#sources-and-freshness) + +## Evidence and version boundaries + +Treat the configuration handoff as the normative merge contract. Treat the +attached Kaiju `@kaiju/config` package as observed implementation that still +requires executable verification. + +The attached lockfile resolves both c12 3.3.4 and c12 4.0.0-beta.5, with jiti +2.7.0 and defu 6.1.7. The CLI package selects c12 4 beta while other consumers +may resolve c12 3. Do not assume layer fields, loader options, internal defaults, +or TypeScript-loading behavior are identical across these lines. + +Before changing config code: + +1. identify which manifest owns the active resolver; +2. trace the exact c12 instance imported by that resolver; +3. inspect its public types and the lockfile-resolved jiti/defu versions; +4. inspect actual `loadConfig()` options, not c12 defaults from memory; +5. create a temporary project and execute discovery, `extends`, environment, + factory, and malformed-export cases through the public resolver. + +## Responsibility map + +| Owner | Responsibilities | Exclusions | +|---|---|---| +| c12 | File discovery, supported formats, RC/package/global sources when enabled, `extends`, environment layers, config factories, watch/update features, layer metadata | Application merge semantics, domain defaults, final runtime validation | +| jiti | Runtime loading/evaluation of TypeScript and JavaScript config modules where c12 uses it | Configuration precedence or trust policy | +| defu | Default-style recursive pair merge and custom merger hook | Complete multi-layer operation language or final schema validation | +| application resolver | Enabled sources, trust policy, precedence, low-to-high evaluation, operations, atomic fields, sanitation, provenance | Parser token grammar | +| Zod/Valibot | Authored shape, sparse patch, resolved runtime config, transformations, defaults | Discovery or code evaluation | +| Optique | CLI/environment/config bindings and early config selection | Project inheritance and merge policy | + +Do not hide ownership inside a generic `loadConfig()` call. c12 defaults are +still product behavior if the application leaves them enabled. + +## Configuration shapes + +Keep at least three schemas: + +| Shape | Contains | Must not contain | +|---|---|---| +| Authoring patch | Partial fields, ergonomic strings, shorthands, array operations, dynamic factory result | Runtime-only derived state | +| Resolved sparse patch | Normalized ordinary data, plain arrays, no defaults unless explicitly authored | `$append`, `$prepend`, `$replace`, c12 metadata | +| Runtime configuration | Complete values, defaults, normalized aliases, semantic values | Missing required values, authoring operations | + +Example: + +```ts +const AuthoringSchema = z.object({ + routes: z.union([ + z.array(z.string()), + z.object({ $append: z.array(z.string()) }).strict(), + z.object({ $prepend: z.array(z.string()) }).strict(), + z.object({ $replace: z.array(z.string()) }).strict(), + ]).optional(), +}); + +const PatchSchema = z.object({ + routes: z.array(z.string()).optional(), +}); + +const RuntimeSchema = z.object({ + routes: z.array(z.url()).default([]), +}); +``` + +Infer types from schemas. Do not maintain handwritten parallel interfaces that +can drift from accepted input or output. + +## Resolution stages + +Use one explicit pipeline: + +```text +select cwd, explicit path, environment, and source policy + -> c12 discovers and evaluates authored file layers + -> validate every retained authored layer + -> normalize c12 exports and metadata + -> build sparse environment patch + -> build sparse CLI/programmatic patch + -> merge highest-priority inputs using low-to-high evaluation + -> remove authoring operations and loader-only metadata + -> validate resolved sparse patch + -> apply defaults and semantic transformations once + -> return runtime config plus provenance +``` + +The observed Kaiju resolver calls c12 with an intentionally narrow source policy: + +```ts +await loadConfig({ + cwd, + name: "project", + configFile: resolvedConfigFile, + configFileRequired: Boolean(resolvedConfigFile), + rcFile: false, + globalRc: false, + packageJson: false, + dotenv: false, + envName: false, + omit$Keys: true, + merger: mergeC12ConfigInputs, + context, +}); +``` + +This proves only that one revision disables RC, global RC, package metadata, +dotenv, and environment-specific layers. It does not mean c12 lacks them or the +product should always disable them. The important merger detail is that c12 gets +a loader-aware merger, not the strict public application merger. + +## c12 loading capabilities + +Decide each capability explicitly: + +- conventional named config discovery; +- explicit config path and missing-explicit-file failure; +- JavaScript/TypeScript module loading; +- JSON, JSONC, YAML, or TOML according to installed support and product policy; +- `extends` chains and preset packages; +- environment-specific config branches; +- RC, user/global RC, and package metadata sources; +- dotenv loading and mutation policy; +- dynamic whole-config factories with a typed context; +- layer metadata and config-file provenance; +- watching/reloading; +- configuration creation/update hooks. + +Do not enable remote `extends` casually. Remote presets expand the trust and +reproducibility boundary. Prefer installed, version-pinned presets. If remote +fetching is allowed, document protocol, cache, integrity, offline behavior, +credentials, redirects, and failure policy. + +Resolve relative paths against their owning layer when the field semantics +require it. Do not resolve all paths against the process cwd after layers from +several directories have merged. + +Define environment layer position. c12 discovery order is not automatically the +same as the product's public precedence. Prove it with conflicting values. + +## Two mergers + +c12 and the application resolver need different merge functions. + +The c12 merger runs while c12 is still discovering and composing file layers. It +must preserve loader metadata and control fields long enough for c12 to consume +them: `extends`, environment branches, config-file layer metadata, and dynamic +factory results. It may normalize enough structure for layer composition, but it +must not run the complete runtime schema or reject loader-only keys too early. + +The public application merger runs after c12 has loaded authored layers. It owns +product semantics: CLI/environment/config precedence, low-to-high operation +evaluation, plain-array replacement, operation-capable arrays, atomic +discriminated unions, strict unknown-key rejection, and final sparse patch +validation. + +Do not pass the strict application merger directly as c12's `merger`. That can +reject `extends` before c12 processes it, and it can evaluate `$append` or +`$prepend` before inherited arrays from extended layers are available. + +Use this shape: + +```text +c12DefuMerger + preserves loader metadata and c12 control fields + composes config-file layers + defers standalone array operations until inherited layers exist + +mergeConfigInputs + accepts only application-authored sparse patches + applies explicit field semantics + strips authoring operations + validates one resolved sparse patch +``` + +Add an integration fixture where a top-level config file both `extends` a base +file and appends to an array from that base. The expected result proves c12 +processed `extends` before the application merger validated the sparse output. + +## Implementation algorithm + +The observed Kaiju implementation uses this algorithm. Preserve the shape even +when local field names differ. + +```text +authoring operation schemas + -> authoring patch schema + -> c12 layer schema with loader control keys + -> c12 loader merger + -> loaded config normalization + -> CLI/env/file sparse patch merge + -> standalone operation lowering + -> sparse patch validation + -> c12 metadata and undefined stripping + -> complete runtime schema parse + -> provenance build or overlay +``` + +Definitions: + +| Function or schema | Job | +|---|---| +| `arrayOperationSchema(item)` | allow plain arrays, `$replace`, `$append`, and `$prepend` in authored files | +| `projectConfigAuthoringPatchSchema` | validate sparse config files before operations are lowered | +| `projectConfigLayerSchema` | allow root c12 controls such as `extends`, `$env`, and named env branches | +| `projectConfigExportValueSchema` | accept object export or array shorthand | +| `resolveStandaloneOperations()` | convert unresolved operation envelopes to arrays when no inherited value exists | +| `defuMerger` | apply application merge exceptions inside defu pair merging | +| `c12DefuMerger` | defer append/prepend without inherited arrays during c12 loading | +| `mergeC12ConfigInputs()` | c12-only custom merger | +| `mergeConfigInputs()` | public app merger, called highest-to-lowest but evaluated low-to-high | +| `normalizeLoadedConfig()` | convert c12 output into an app sparse patch | +| `sanitizeConfigInput()` | remove `extends`, `$meta`, and `undefined` before final sparse validation | +| `applyConfigPatch()` | merge one parsed Optique patch over an existing resolved config snapshot | + +The app merger callback handles only exceptions: + +```text +incoming $replace -> copy replacement array +incoming $append + inherited array -> inherited then appended +incoming $append alone -> appended only +incoming $prepend + inherited array -> prepended then inherited +incoming $prepend alone -> prepended only +incoming plain array over array -> copy incoming array +sources.commoncrawl.crawls object -> replace atomically +everything else -> return false to defu +``` + +The c12 merger changes one rule: + +```text +append/prepend without inherited array -> return false, do not lower yet +``` + +c12 may still be about to merge an extended base layer. Lowering too early loses +the chance to compose with that inherited array. + +The public app merge reverses caller order: + +```text +caller order: CLI, env, file +evaluation order: file, env, CLI +``` + +Then it parses `resolveStandaloneOperations(resolved)` with the sparse patch +schema. That last parse is the guard that operation envelopes and loader-only +fields did not leak toward runtime. + +## Dynamic factories and jiti + +Use a whole-config factory when configuration genuinely depends on typed load +context: + +```ts +export default defineConfig(({ cwd, environment }) => ({ + root: cwd, + output: environment === "production" ? "./dist" : "./tmp", +})); +``` + +Keep factories: + +- deterministic for a supplied context; +- side-effect-light; +- bounded and cancellable if asynchronous behavior is allowed; +- free of command execution; +- validated immediately after evaluation; +- evaluated once per CLI invocation. + +jiti is a code loader, not a sandbox. A TypeScript config module can execute +arbitrary code with the process's permissions. Treat config trust like code +trust. Do not load untrusted project configuration during `--help`, completion, +or other early surfaces unless the product explicitly requires it. + +Avoid arbitrary field-level callback functions. They cannot be represented in +JSON/JSONC, complicate provenance, and introduce hidden evaluation order. Use +serializable operations for merge behavior and reserve a whole-config factory +for actual computation. + +Two-pass parsing can accidentally evaluate a factory twice. Count executions in +an integration test: + +```ts +let loads = 0; +export default defineConfig(() => { + loads += 1; + return { output: `./run-${loads}` }; +}); +``` + +The public command should observe one load and a stable value. A test-only +counter can live in a fixture module or write to a temporary marker. + +## Precedence and evaluation order + +Document caller precedence from highest to lowest: + +```text +CLI/programmatic patch + > environment patch + > explicit/project config + > user/global/preset layers if enabled + > derived default + > runtime-schema default +``` + +Evaluate compositional operations from the foundation upward: + +```text +caller: mergeConfigInputs(cli, env, file) +execution: file -> env -> cli +``` + +This difference is essential. An environment prepend needs a resolved file +array, and a CLI append needs the result of both. + +Ignore missing layers, not meaningful falsy leaves. Preserve `false`, `0`, an +explicit empty string where schema-valid, and an empty array. Define `null` +semantics per field or reject it; do not let a generic merge library decide. + +Apply derived and schema defaults after authored precedence. A higher sparse +layer must not receive defaults that mask lower authored values. + +## Merge algebra + +Classify every field: + +| Category | Default rule | +|---|---| +| Scalar leaf | Higher defined value wins | +| Ordinary object | Recursively merge properties | +| Plain array | Higher array replaces complete lower array | +| Operation-capable array | Apply declared operation to resolved inherited array | +| Discriminated union | Higher branch replaces atomically | +| Default | Apply once after sparse merge | + +Do not inherit defu's array concatenation as accidental product behavior. A +route list, sink list, schema list, seed module list, or status filter may need +replacement, ordered composition, or deduplication. Decide path by path. + +Use a custom defu merger for pairwise exceptions: + +```ts +type Merger = Parameters[0]; + +const merger: Merger = (target, key, incoming, namespace) => { + const inherited = target[key]; + + if (isAppend(incoming)) { + setMergedValue( + target, + key, + Array.isArray(inherited) + ? [...inherited, ...incoming.$append] + : [...incoming.$append], + ); + return true; + } + + if (Array.isArray(incoming)) { + setMergedValue(target, key, [...incoming]); + return true; + } + + if (namespace === "sources.commoncrawl" && key === "crawls") { + setMergedValue(target, key, structuredClone(incoming)); + return true; + } + + return false; +}; +``` + +Inspect the installed defu callback orientation. In the observed contract, +`target[key]` is the inherited lower value and `incoming` is the higher layer. +A callback that returns `true` without assigning the intended value can keep the +wrong side. + +Isolate the required generic indexed-assignment assertion in one documented +helper. Do not scatter unsafe casts through the merger. + +Copy arrays and mutable nested operation values. The resolved config must not +alias caller inputs. + +## Declarative array operations + +Use serializable data: + +```ts +export function replace(values: readonly T[]) { + return { $replace: [...values] } as const; +} + +export function append(values: readonly T[]) { + return { $append: [...values] } as const; +} + +export function prepend(values: readonly T[]) { + return { $prepend: [...values] } as const; +} +``` + +Define exact semantics: + +| Authored value | Inherited `[A, B]` | No inherited value | +|---|---|---| +| `[C]` | `[C]` | `[C]` | +| `{ $replace: [C] }` | `[C]` | `[C]` | +| `{ $append: [C] }` | `[A, B, C]` | `[C]` | +| `{ $prepend: [C] }` | `[C, A, B]` | `[C]` | + +Raw pairwise defu reduction may never call the custom merger for a standalone +operation. Normalize operations that have no inherited value after traversal: + +```ts +function resolveStandalone(value: unknown): unknown { + if (isReplace(value)) return [...value.$replace]; + if (isAppend(value)) return [...value.$append]; + if (isPrepend(value)) return [...value.$prepend]; + if (Array.isArray(value)) return value.map(resolveStandalone); + if (!isPlainRecord(value)) return value; + return Object.fromEntries( + Object.entries(value).map(([key, child]) => [key, resolveStandalone(child)]), + ); +} +``` + +Resolve one layer at a time from low to high so an operation never receives an +unresolved operation object as its inherited value. + +Do not silently deduplicate. Duplicates may be meaningful. If a field needs +set-like behavior, specify equality, normalization, ordering, and provenance as +field semantics outside the generic merger. + +## Atomic unions and special fields + +Recursively merging discriminated union branches creates impossible hybrids: + +```text +lower: { kind: "range", from: "A", to: "B", limit: 3 } +higher: { kind: "named", crawls: ["C"] } +wrong: { kind: "named", crawls: ["C"], from: "A", to: "B", limit: 3 } +``` + +Replace the entire value when the higher layer selects a new branch. Key the +exception by a schema-known path, not a broad “objects with a `kind` key” guess. + +Other fields may need atomic replacement: credential providers, output +destinations, retry strategies, database connection modes, authentication +methods, and deployment targets. Inventory unions when schemas change. + +## Provenance and inspection + +Return normalized c12 layers alongside the runtime config. Preserve at least: + +- source kind; +- resolved path or package/preset identifier; +- environment/extends relationship; +- authored sparse contribution; +- evaluation order; +- load timestamp or revision where reproducibility matters. + +Maintain per-field winners if `config explain` is promised: + +```ts +const ValueProvenanceSchema = z.object({ + path: z.string(), + winner: z.object({ source: z.string(), detail: z.string().optional() }), + shadowed: z.array(z.object({ + source: z.string(), + detail: z.string().optional(), + })).default([]), + operations: z.array(z.string()).default([]), +}); +``` + +`config files` must report every contributing layer, not only the final +`_configFile`. `config explain` must report actual winners and shadowed values, +not print a static precedence list. + +Redact secrets before rendering resolved config or provenance. Avoid serializing +executable source or arbitrary error objects into support output. + +## Configuration mutation + +Use configuration mutation only with explicit target and consent: + +1. select the owned project or user file; +2. parse and validate its authored shape; +3. build the intended patch; +4. show a diff for `--dry-run` and interactive review; +5. write atomically; +6. preserve source style where practical; +7. reload through the public resolver; +8. report the exact changed path. + +Use c12 creation/update hooks where supported. Use `magicast` for +source-preserving TypeScript/JavaScript edits and `confbox` for structured +formats such as JSONC/YAML/TOML. Do not apply a JSON serializer to a TypeScript +config or discard comments in a user-authored file without authorization. + +`rc9` can own XDG-aware user RC reads/writes. Keep user config distinct from +project config and document their precedence and uninstall/preservation policy. + +## Validation boundaries + +Validate retained source layers, not only c12's final merged object. A malformed +scalar export can be hidden or collapsed during generic merging. + +Use this order: + +```text +each authored export -> authoring schema +merged operations -> resolved sparse patch schema +defaults/transforms -> complete runtime schema +``` + +Never pass authoring operation objects to source adapters. Use strict schemas to +reject unknown keys unless extension fields are deliberately supported. + +Keep external source errors distinct: + +- explicit config file missing; +- unsupported extension/format; +- evaluation/import failure; +- invalid factory return; +- cyclic or failed `extends`; +- malformed layer value; +- invalid merged patch; +- invalid complete runtime config. + +Render the layer path and schema issue path without leaking secrets. + +## Testing strategy + +Test three boundaries: + +1. merger unit tests: generic pair orientation, field exceptions, immutability, + operations, arrays, and atomic unions; +2. public resolver tests: valid public authoring shapes, source precedence, + defaults, provenance, and strict output; +3. c12 integration fixtures: real file discovery, formats, extends, environment + layers, factories, malformed exports, and exact version behavior. + +Required matrix: + +- missing versus `false`, `0`, empty string, and empty array; +- ordinary nested object contribution from three layers; +- plain array replacement and empty-array clearing; +- replace/append/prepend with and without inheritance; +- operation-on-operation across file, environment, and CLI; +- atomic union branch changes; +- no input aliasing; +- defaults applied once after merge; +- explicit file required/missing; +- nested extends and path resolution; +- dynamic factory evaluated once; +- malformed scalar, array, and unknown-key exports; +- layer list, winner, shadowed value, and operation provenance; +- help/version behavior when config is invalid; +- c12 3 versus c12 4 behavior if both are supported. + +Run type checking against the public types. defu's generic callback commonly +exposes `string | symbol` keys; use `String(key)` in diagnostics and isolate +indexed writes. + +## Failure signatures + +| Symptom | Likely cause | Corrective action | +|---|---|---| +| `$append` reaches runtime | Standalone operation was not normalized | Resolve operations after low-to-high traversal and validate patch | +| CLI append drops file values | Layers evaluated high-to-low | Reverse traversal and apply one higher layer at a time | +| Plain arrays concatenate | Inherited defu default was not overridden | Add explicit array replacement branch | +| Empty array fails to clear | Truthiness/missing logic treats it as absent | Distinguish `undefined` from valid empty values | +| Union contains fields from two variants | Recursive merge crossed atomic branch | Add precise path replacement | +| Factory side effect occurs twice | Source context and handler both reload c12 | Load once at composition root and reuse result | +| Defaults override an authored lower value | Patch schema injected defaults per layer | Keep sources sparse; default final runtime schema once | +| `config explain` lacks actual winners | Provenance discarded during merge | Track field contributions while resolving | +| `config files` shows one file | Only `_configFile` retained | Normalize all c12 layer metadata | +| Malformed scalar becomes empty config | Only final c12 result was validated | Validate retained source layers before normalization | +| Output mutates when input array changes | Merger retained caller reference | Clone arrays/operation payloads | +| TS error on `target[key]` | Generic defu indexed assignment | Use one documented mutation adapter | +| Config works on one package only | c12 major/prerelease lines diverge | Pin owner and run version-specific fixtures | +| `.env` winner differs by command | Multiple loaders own same environment setting | Create one source algebra and visible precedence | +| Remote preset changes without lock update | Mutable `extends` trust boundary | Pin installed preset or require integrity/cache policy | + +## Extension checklist + +For each new field: + +1. classify it as scalar, recursive object, plain array, operation array, atomic + union, or specialized value; +2. add it to the authored schema; +3. add the normalized field to the sparse patch without defaults; +4. add the runtime field and defaults/transforms; +5. define `undefined`, `null`, falsy, empty, and deletion semantics; +6. define source precedence and enabled sources; +7. add a path-specific merger rule only when necessary; +8. add provenance rendering and redaction policy; +9. test one, two, and three conflicting layers through real c12 files; +10. confirm no authoring syntax reaches a runtime consumer. + +For each new array operation, define behavior with no inherited array, order, +duplicates, immutability, serialization, provenance, and final-schema validity. +Reject operations whose semantics cannot be explained consistently in +TypeScript and JSONC. + +## Sources and freshness + +- Normative merge contract: `kaiju-config-resolution-handoff(2).md`, reviewed 2026-07-17. +- Normative CLI architecture: `productionized-cli-pattern-guidebook-v1.2.md`, reviewed 2026-07-22. +- Observed implementation: current Kaiju config c12 two-merger fix plus earlier `live-browser-cli(41).zip/packages/config` evidence. +- c12 official source: , discovery pointer for current loader APIs and version history. +- defu official source: , discovery pointer for current merger callback behavior. +- jiti official source: , discovery pointer for loader/runtime behavior. + +Freshness status: the attached repository evidence resolves c12 3.3.4 and +4.0.0-beta.5, defu 6.1.7, and jiti 2.7.0. The 2026-07-22 Kaiju fix proves the +two-merger pattern against c12 4 beta. Do not generalize observed layer metadata +or callback orientation to another version without integration tests. diff --git a/skills/build-clis/references/casebook.md b/skills/build-clis/references/casebook.md index 502bc10..65c2689 100644 --- a/skills/build-clis/references/casebook.md +++ b/skills/build-clis/references/casebook.md @@ -1,41 +1,1299 @@ # Kaiju CLI casebook -These cases are evidence patterns, not claims that the source repository is a -finished reference implementation. - -## Patterns to preserve - -- Static command registration keeps compiled and bundled commands visible. -- Shared parser groups plus final source schemas can separate token grammar from - domain validation, provided lost refinements are reapplied deliberately. -- Sparse CLI, environment, and file patches can merge before runtime defaults. -- Declarative array replace/append/prepend operations can compose from low to - high precedence while keeping inputs immutable. -- A LogTape result category with parent-sink override can preserve exact stdout. -- A typed `StageWriter` can own validated durable JSONL while optionally emitting - diagnostics through LogTape. - -## Failures to detect - -| Observed signature | Contract defect | -|---|---| -| README tasks point to missing files or permission sets | Docs/task/executable parity failure | -| README flags differ from parser values | Public-language drift | -| Root and package manifests use different c12/Optique/LogTape versions | Ownership and version boundary unresolved | -| Config resolves before parse and again inside handlers | Dynamic factories may execute twice; provenance and precedence can drift | -| `config explain` prints only static precedence | It does not explain field winners or shadowed values | -| `.env` participates through one parser path but not the public resolver | Split source ownership | -| One `--from`/`--to` pair becomes both crawl IDs and capture timestamps | One public name has two domain meanings | -| Positive and negative flags are independent options | Parser can accept a structurally invalid state | -| Result object serializes before field redaction | Nested secrets remain visible inside one string | -| Handler logs and rethrows; entrypoint logs again | Duplicate public diagnostics | -| Every failure exits 2 | Automation cannot distinguish usage, unavailable, conflict, cancellation, and internal defects | -| Active runner creates an unattached signal | Cancellation types exist without process ownership | -| Guidebook describes paging, telemetry, resume, or output modes | Normative aspiration, not executable proof | - -## Review procedure - -For a request against this codebase, first identify the exact installed revision, -then trace the active entrypoint. Do not treat unused helpers, README claims, or -guidebook policies as live behavior. Convert each accepted correction into a -subprocess or generated-surface oracle so the discrepancy cannot silently return. +## Contents + +- [Use this casebook](#use-this-casebook) +- [Repository trace index](#repository-trace-index) +- [End-to-end value trace template](#end-to-end-value-trace-template) +- [Portable CLI policy overlay](#portable-cli-policy-overlay) +- [Library ownership map](#library-ownership-map) +- [Optique integration details](#optique-integration-details) +- [Root program and one-snapshot config](#root-program-and-one-snapshot-config) +- [Config patch organization](#config-patch-organization) +- [Sparse adapter rules](#sparse-adapter-rules) +- [c12 loading details](#c12-loading-details) +- [Config merge and provenance mechanics](#config-merge-and-provenance-mechanics) +- [Field-extension checklist](#field-extension-checklist) +- [LogTape integration details](#logtape-integration-details) +- [Trace: browser detect](#trace-browser-detect) +- [Trace: Common Crawl detect](#trace-common-crawl-detect) +- [Trace: local WARC detect](#trace-local-warc-detect) +- [Trace: domains verify](#trace-domains-verify) +- [Stage, artifact, result, and diagnostic boundaries](#stage-artifact-result-and-diagnostic-boundaries) +- [Lifecycle, concurrency, cancellation, and cleanup](#lifecycle-concurrency-cancellation-and-cleanup) +- [Verification matrix](#verification-matrix) +- [Failure signatures](#failure-signatures) + +## Use this casebook + +Load this casebook when a CLI has multiple input sources, generated command +surfaces, durable artifacts, long-running source work, or the Optique + c12 + +defu + LogTape + Zod stack. It gives concrete trace shapes from the Kaiju CLI so +an agent can identify live behavior instead of relying on README claims, +guidebook aspirations, or package-name recall. + +This is not a promise that every ideal lifecycle feature is complete in Kaiju. +Keep three states separate while reviewing or editing: + +| State | Meaning | Required evidence | +|---|---|---| +| Intended architecture | The design the code appears to aim for | architecture docs plus code ownership | +| Observed implementation | What the current source actually does | traced entrypoints, adapters, schemas, tests | +| Executed behavior | What a subprocess or focused test proved | command/test output with exit status | + +## Repository trace index + +Start with this map before changing a public CLI surface: + +| Concern | Primary files | What to inspect | +|---|---|---| +| Root command graph | `clis/main/src/shared/program.ts`, `clis/main/src/commands/index.ts` | Static command imports, `runProgram()` metadata, help/completion policy, source contexts, hooks | +| Compatibility entrypoints | `clis/main/src/browser.ts`, `commoncrawl.ts`, `warc.ts`, `domains.ts`, `shared/entrypoint.ts` | Delegation to root program, bootstrap error logging, raw logging flag recovery | +| Source contexts | `clis/main/src/shared/source.ts` | `KAIJU_` env source, c12 config source, route-derived default source, fallback order | +| Shared parser fragments | `shared/output.ts`, `shared/routes.ts`, `shared/run-config.ts`, `shared/logging.ts`, `shared/time.ts` | aliases, env/config binding, derived defaults, early controls, Temporal values | +| Help-only defaults | `shared/defaults.ts`, `shared/defaults.test.ts` | `documentDefault()` preserves sparse parse output while feeding Optique docs | +| Config overlay | `shared/config.ts` | `resolveCommandConfig()`, `applyConfigPatch()`, explicit argv provenance aliases | +| Source adapters | `browser/adapter.ts`, `commoncrawl/adapter.ts`, `warc/adapter.ts`, `domains/adapter.ts` | sparse patch creation, profile expansion, alias normalization, source validation | +| Config package | `packages/config/src/index.ts`, `merge.ts`, `schemas.ts` | c12 loader policy, two mergers, operation syntax, final Zod defaults, provenance | +| Results/logs | `shared/logging.ts`, `packages/diagnostics/src/logging.ts`, `index.ts` | raw result sink, diagnostic routing, redaction, file buffering, reset/flush | +| Stage writer | `packages/stages/src/index.ts`, `schemas.ts`, `stage-payloads.ts` | envelope schema, per-stage queues, known payload schemas, binary rejection | +| Browser source | `clis/main/src/browser/run.ts`, `source.ts`, `route.ts`, `packages/browser/src/*`, `packages/discovery/src/*` | discovery, collection policy, archive/WARC/WACZ, resume, screenshot/storage/CDP/DNS/TLS knobs | +| WARC and Common Crawl runs | `shared/run.ts`, `packages/warc/src/*`, `packages/commoncrawl/src/*`, `packages/cdx/src/*` | planner, CDX decisions, range fetches, retry/rate limiting, worker pool, run summaries | +| Domain verification | `domains/run.ts`, `packages/domains/src/*` | input/output files, accepted/retry classification, audit stages, DNS/HTTP policy | +| Concurrency primitives | `packages/engine/src/queue.ts`, `shared/run.ts` | bounded queues, ordered async map, detector worker pool, closure paths | + +## End-to-end value trace template + +Trace every source-bearing public value through all arrows. Do not skip directly +from parser to handler. + +```text +public token/env/config field + -> Optique parser term and source-context wrappers + -> raw parsed value or absence + -> source adapter sparse ProjectConfigPatch + -> c12 loaded file patch, env patch, CLI patch + -> application merge semantics + -> complete Zod ProjectConfig default/transform validation + -> provenance leaf or conservative source detail + -> command run input + -> source adapter/planner/collector consumer + -> result, diagnostic, stage, artifact, or summary + -> focused test or subprocess oracle +``` + +Classify each public term first: + +| Term kind | Examples | Ownership | +|---|---|---| +| Early control | `--help`, `--version`, raw `--log-format` for bootstrap failure | executable/parser boundary before project config | +| Source-bearing config | `--out-dir`, `--run-id`, `--range-cache`, source network knobs | Optique parses, adapter emits sparse patch, resolver merges, Zod completes | +| Domain request input | route values, WARC file path, domains input file | parser/adapter validates shape; source run owns domain semantics | +| Result selector | `--json`, generated man/completion output | renderer/LogTape result route owns bytes | + +## Portable CLI policy overlay + +Kaiju is the worked example, but the reusable lesson is broader: a production +CLI has an execution architecture and a human-interface policy. Optique, +LogTape, c12, defu, and Zod can implement pieces of that policy; none of them +decides the product contract. + +```text +command language + -> parser grammar, aliases, completion, help, suggestions +source language + -> CLI/env/config/default provenance and precedence +result language + -> stdout machine contracts and human renderers +diagnostic language + -> stderr, files, support bundles, telemetry routes +safety language + -> dry-run plans, force, typed confirmations, checkpoints +state language + -> status, resume, explain, files, install/remove locations +``` + +When auditing or extending a CLI, inspect these policy layers explicitly: + +| Policy layer | Preferred owner | Review questions | Tests that prove it | +|---|---|---|---| +| Command names and flags | Optique parser plus app naming policy | Are long names stable? Are aliases deliberate? Are hidden aliases documented for compatibility? Are heterogeneous inputs flags rather than magic positionals? | help/man/completion snapshots; invalid sibling command suggestions; alias parse tests | +| No-argument and help behavior | root command policy | Does no-arg output orient a human without pretending to run? Does full help contain examples, config/env names, output modes, and issue/docs links? | subprocess `app`, `app --help`, `app command --help`; generated man smoke | +| Human versus machine output | result schemas and renderers, transported by LogTape | Is stdout exactly the requested result mode? Is stderr free to be human? Are JSON and JSONL stable schemas rather than stringified human output? | separate stdout/stderr/exit assertions for human, `--plain`, `--json`, `--jsonl` | +| Interaction | prompt adapter gated by app policy | Does every prompt have a flag/file/stdin equivalent? Does `--no-input` fail fast with missing fields? Does CI/non-TTY avoid hanging? | TTY and non-TTY subprocess tests; `--no-input`; CI env simulation | +| Dangerous operations | risk and operation-plan schemas | Is danger classified as none/mild/moderate/severe? Does dry-run emit the exact plan the executor consumes? Does severe risk require a typed target token? | dry-run plan schema test; `--force`; `--confirm=`; mismatch rejection | +| Standard streams | endpoint parser and IO adapter | Does `-` mean stdin/stdout only where file-like values are expected? Are empty strings rejected rather than overloaded? Does a command fail if required stdin is a TTY? | parser tests for `-`, `none`, path; subprocess with piped and TTY stdin | +| Secrets | `SecretSource` capability plus redaction | Are secrets read from file/stdin/prompt/provider/socket instead of argv or broad env? Are argv, config views, URLs, errors, and support bundles redacted? | redaction fixtures; process-argv snapshot excludes secret values; prompt gated on TTY | +| Pager | host `Pager` capability | Does paging activate only for long human output on TTY? Is `--no-pager` honored? Is JSON never paged? Is `PAGER` spawned as argv, not shell text? | redirected stdout, JSON, CI, no-pager, custom pager tests | +| Config locations and edits | c12/rc9/magicast/confbox behind adapters | Are project/user/system layers explicit? Do edits show target and diff, obtain consent, write atomically, and validate authored schema? | dry-run edit plan; temp XDG paths; format-preserving edit fixture; malformed edit rollback | +| Recovery and checkpoints | checkpoint schema and store capability | Can an interrupted run reconcile committed work and resume idempotent units? Is the request fingerprint checked? | kill/restart subprocess; incompatible resume rejection; checkpoint cleanup | +| Telemetry | consent schema and LogTape remote sinks | Is remote reporting disabled by default unless enterprise policy says otherwise? Are diagnostics separate from analytics? Is consent revocable? | default no-network test; consent config; sink redaction; telemetry failure does not fail command | +| Distribution and removal | packaging/release scripts | Does the installed or compiled shape include commands/assets/workers? Are uninstall and owned paths discoverable? | clean install/tarball/binary smoke; `doctor paths`; offline startup | + +Concrete schema anchors for portable policy: + +```ts +export const InteractionPolicySchema = z.object({ + mode: z.enum(["auto", "interactive", "non_interactive"]), + force: z.boolean().default(false), + dry_run: z.boolean().default(false), + confirm: z.string().optional(), +}); + +export const RiskLevelSchema = z.enum(["none", "mild", "moderate", "severe"]); + +export const OperationPlanSchema = z.object({ + operation: z.string(), + risk: RiskLevelSchema, + changes: z.array(z.object({ + kind: z.string(), + target: z.string(), + description: z.string(), + })), + confirmation_token: z.string().optional(), +}); + +export const FileEndpointSchema = z.discriminatedUnion("kind", [ + z.object({ kind: z.literal("path"), path: z.string().min(1) }), + z.object({ kind: z.literal("stdin") }), + z.object({ kind: z.literal("stdout") }), + z.object({ kind: z.literal("disabled") }), +]); + +export const TelemetryPolicySchema = z.object({ + mode: z.enum(["disabled", "local", "consented_remote"]), + consented_at: z.iso.datetime().optional(), + retention_days: z.int().positive().optional(), + endpoint: z.url().optional(), +}); +``` + +Use these schemas as contracts, not decorations. A `--dry-run` command should +emit an `OperationPlanSchema` object and the executor should consume the same +plan or a fingerprinted/persisted equivalent. A prompt should fill a normal +request schema, not bypass validation. A pager should consume already-rendered +human text; it should not be another logging sink. Telemetry routes should be +LogTape sinks selected by consent policy, not ad hoc network calls inside +domain handlers. + +Diagram/documentation policy for future casebooks: + +| Relationship | Preferred doc shape | Why | +|---|---|---| +| terminal flows and source precedence | fenced ASCII flow | stable in terminals, diffs, and Markdown previews | +| ownership boundaries and test matrices | tables | dense relationships stay readable | +| small regular graphs | Mermaid | useful only when auto-layout is predictable | +| nested architecture or page-constrained docs | prose plus tables or a designed visual | Mermaid auto-layout tends to obscure detail | + +Every diagram needs a text equivalent. Render documentation in the target +surface before claiming it is readable. + +## Library ownership map + +When all five libraries are present, the safe design is not “use everything +everywhere.” Give each library one crisp job. + +| Boundary | Optique | c12 | defu | Zod | LogTape | +|---|---|---|---|---|---| +| Public command grammar | Owns command tree, flags, aliases, choices, suggestions, completion, man/help metadata | none | none | value parser adapter only when useful | none | +| Environment/config/default source binding | May bind one parser term to env/config/derived contexts | Supplies loaded config object for config context | none | validates individual values if used through `@optique/zod` | none | +| Config file discovery | none | Owns file lookup, explicit config path, env branches, `extends`, TS/JS factory execution | Used as c12 merger helper | validates retained layer shapes after load | diagnostics only | +| App-level precedence | none, except parsed CLI value contribution | none after loaded file patch exists | Implements field-specific merge callback behind app resolver | validates sparse patch and final config | can report decisions | +| Defaults | Shows documented defaults; may provide handler-time `deferredValue()` | none | none | Owns executable runtime defaults after all sources merge | none | +| Provenance | Provides argv and parser/source-context evidence; does not provide full winner graph | file path and loader metadata evidence | merge operation source evidence | default-origin evidence | transports redacted explanation | +| Results and diagnostics | command metadata can influence output shape | none | none | validates result schemas | owns result and diagnostic routes | +| Durable artifacts | none | none | none | validates artifact/stage metadata | may mirror event; does not own artifact storage | + +Reject these cross-library anti-patterns: + +| Anti-pattern | Why it fails | +|---|---| +| `withDefault(option("--x", ...), value)` for a source-bearing field | The parser emits a CLI-shaped value even when the user omitted the flag, so config/env/default precedence is polluted. | +| `loadConfig({ merger: mergeConfigInputs })` | The strict app merger can reject c12 control keys before c12 resolves `extends` and env branches. | +| `defu(cli, env, file)` as the public merge policy | Default defu array concatenation and object recursion are not the CLI's product semantics. | +| One Zod schema for config authoring, sparse patches, and runtime config | Defaults materialize too early and authoring controls can leak into handlers. | +| `console.log(JSON.stringify(result))` beside LogTape | Stable stdout bypasses routing, redaction, sink isolation, and tests. | +| Durable JSONL stages written as ordinary log messages | Logs are observation transport; stage files are replay/test artifacts with schemas and counters. | + +## Optique integration details + +Optique should make the public command language rich and hard to misuse. It +should not become the application config resolver. + +### Parser construction + +Use Optique terms to model the public grammar: + +```ts +const parser = merge( + cliRouteInputOptionsParser, + cliOutputOptionsParser, + cliRunOptionsParser, + cliConfigOptionsParser, + cliLogOptionsParser, + object({ + mode: documentDefault(optional(option("--mode", choice(["detect", "collect"]))), "detect"), + range_cache_enabled: optional(negatableFlag("--range-cache")), + timeout: optional(option("--timeout", duration())), + }), +); +``` + +The parser owns: + +- flag spelling and aliases; +- choices and typo suggestions; +- conflict shapes such as positive/negative Boolean pairs; +- help and man-page metadata; +- shell completion metadata; +- typed value parsers such as durations, instants, URLs, integers, and enums. + +The parser does not own: + +- final runtime defaults for source-bearing fields; +- cross-source precedence; +- operation-aware array merging; +- field provenance winner graphs; +- durable artifacts or stage files; +- source execution policy. + +### Source contexts + +Kaiju uses three Optique source contexts: + +```ts +export const kaijuEnvContext = createEnvContext({ + prefix: "KAIJU_", + envFile: [".env", ".env.local"], +}); + +export const kaijuConfigContext = createConfigContext({ + schema: projectConfigPatchSchema, +}); + +export const kaijuDerivedDefaults = createDerivedDefaults({ + run_id: (parsed) => firstRouteValue(parsed)?.toLowerCase().replace(/[^a-z0-9._-]+/gu, "-"), +}); + +export const kaijuSourceContexts = [ + kaijuEnvContext, + kaijuConfigContext, + kaijuDerivedDefaults.context, +] as const; +``` + +Use source contexts for fields that are truly public command terms and can be +revalidated by the same parser. Do not assume this gives you complete +provenance. The command still needs explicit argv evidence to distinguish a +user-supplied CLI value from an env/config/derived fallback that Optique +re-emits. + +### Help-visible defaults versus executable defaults + +Use these patterns: + +| Need | Optique shape | Runtime effect | +|---|---|---| +| Show default in help, keep patch sparse | `documentDefault(optional(option(...)), value)` | omitted parse remains `undefined` | +| Early parser control default | `withDefault(option("--json"), false)` | safe because `json` is output mode, not config precedence | +| Handler-time secret/interactive/expensive fallback | `deferredValue(parser, fallback)` | handler receives resolver function; fallback is not persisted into config | +| Ordinary runtime fallback | no parser default | final Zod runtime schema applies `.default()` or `.prefault()` after merge | + +Wrong: + +```ts +// Pollutes the CLI patch: config/env can no longer win over omission. +concurrency: withDefault(option("--concurrency", integer()), 4) +``` + +Right: + +```ts +// Help shows the default, but omitted parser output remains absent. +concurrency: documentDefault(optional(option("--concurrency", integer())), 4) +``` + +### Command handler integration + +Command modules should do only four things: + +1. collect route/file/stdin request inputs; +2. create a sparse source patch; +3. resolve that patch against the hook-provided config snapshot; +4. run the source project and print the stable result. + +Kaiju-style command shape: + +```ts +export async function run(options, context) { + const routes = await resolveRouteUrls(options, normalizeBrowserRouteUrl); + const patch = createBrowserConfigPatch(options, routes); + const resolved = await resolveCommandConfig({ patch }, context); + + if (options.dry_run) { + printCliResult(formatPlan(resolved), options.json, humanPlan(resolved)); + return; + } + + const summary = await runBrowserProject({ config: resolved.config }); + printCliResult(summary, options.json, formatBrowserSummary(summary)); +} +``` + +Handlers should not call `Deno.exit`, read `Deno.env`, configure LogTape, or +call `resolveConfig()` when hook context is present. + +## Root program and one-snapshot config + +The root program should be the only place that joins parser, source contexts, +config snapshot, and logger resource. + +Observed Kaiju shape: + +```ts +await runProgram({ + commands, + args, + metadata, + help: "both", + completion: "both", + showDefault: true, + showChoices: true, + contexts: kaijuSourceContexts, + contextOptions: { + async load(parsed) { + const bootstrap = bootstrapConfigOptionsSchema.parse(parsed ?? {}); + const resolved = await resolveConfig({ + configFile: bootstrap.config_file, + noConfig: bootstrap.no_config, + }); + baseConfig = resolved; + return { config: resolved.config, meta: { configPath: resolved.configPath } }; + }, + }, + hooks: { + async beforeEach(invocation) { + if (!baseConfig) throw new Error("The command configuration context did not load."); + const logger = await configureCliLogging(invocation.value, ...invocation.path); + return { resource: { config: baseConfig, logger, args } }; + }, + }, +}); +``` + +Rules: + +- `commands` must be statically imported so bundling and `deno compile` can see + every command. +- Context loading may call c12 once. Handlers should use the hook resource or a + direct-test fallback, not reload full config. +- `beforeEach` configures command logging after final parsed command values are + available. +- Source-specific compatibility entrypoints delegate to the root program and use + `runCliEntrypoint()` only to report pre-handler failures. + +Failure oracle: + +```text +dynamic config factory increments marker + -> run one command + -> marker count must be 1 +``` + +If the count is 2, source context and handler both loaded config. + +## Config patch organization + +Organize patches by ownership. A source command can contribute both +project-level fields and source-specific fields, but the reason must be visible. + +```text +ProjectConfigPatch + runId? shared run identity override + outDir? shared default output root override + log? process-level logging preference + sources.browser? browser-specific source policy + sources.commoncrawl? Common Crawl source policy + sources.warc? local WARC source policy + sources.domains? domain-verification source policy +``` + +Keep these buckets distinct: + +| Bucket | Examples | Rule | +|---|---|---| +| Shared project fields | `runId`, `outDir`, `log.level`, `log.format` | set only if CLI/env/config source explicitly contributes or source input implies a run root | +| Source run identity | `sources.browser.runId`, `sources.commoncrawl.runId` | mirror when the source run needs a local override or manifest identity | +| Source input | browser routes, WARC files, Common Crawl route, domains input file | required by the source runner; do not hide in global config | +| Source network policy | Common Crawl delays/retries, domain DNS/HTTP timeout | sparse; final defaults live in runtime schema | +| Output artifacts | WARC path, WACZ path/dir/package, domain accepted/retry files | source-specific artifact writers consume them | +| Diagnostic preference | process `log` and optional source `logLevel` | process logging and source library logging may be related but are not the same field | + +Browser patch shape: + +```ts +const browserPatch: BrowserSourcePatch = {}; +assignIfDefined(browserPatch, "routes", routeUrls.length > 0 ? [...routeUrls] : undefined); +assignIfDefined(browserPatch, "runId", options.run_id); +assignIfDefined(browserPatch, "waitUntil", options.wait_until ?? maxCaptureDefault(options, "networkidle")); +assignIfDefined(browserPatch, "warc", createWarcPatch(options)); +assignIfDefined(browserPatch, "wacz", createWaczPatch(options)); +assignIfDefined(browserPatch, "screenshots", createScreenshotPatch(options)); + +const basePatch: ProjectConfigPatch = { + runId: options.run_id, + log: { level: logging.level, format: logging.format }, +}; +if (hasRouteInput || options.out_dir || options.out) basePatch.outDir = outDir; +if (Object.keys(browserPatch).length > 0) basePatch.sources = { browser: browserPatch }; +``` + +Common Crawl patch shape: + +```ts +const commoncrawlPatch = { + route: routeUrl, + logLevel: toSourceLogLevel(logLevel), + crawls: crawlSelector(options), + rangeCache: buildRangeCachePatch(options), + network: buildNetworkPatch(options), + analyzedAt: resolveAnalyzedAt(options), +}; +``` + +Do not normalize everything into a flat map before validation. Nested patch +shape mirrors the runtime owner graph and makes merge/provenance paths stable. + +## Sparse adapter rules + +Adapters turn command values into sparse patches. They are not runtime default +owners. + +Correct adapter skeleton: + +```ts +export function createSourceConfigPatch(input: SourceCliOptions): ProjectConfigPatch { + const options = sourceCliOptionsSchema.parse(input); + const source: NonNullable["source"] = {}; + + assignIfDefined(source, "route", options.route); + assignIfDefined(source, "timeoutMs", options.timeout?.total("milliseconds")); + assignIfDefined(source, "enabled", options.enabled === true ? true : undefined); + + const patch: ProjectConfigPatch = { + runId: options.run_id, + log: resolveCliLogging(options), + }; + if (Object.keys(source).length > 0) patch.sources = { source }; + return projectConfigPatchSchema.parse(patch); +} + +function assignIfDefined( + target: T, + key: K, + value: T[K] | undefined, +): void { + if (value !== undefined) target[key] = value; +} +``` + +Rules and examples: + +| Rule | Example | Failure if broken | +|---|---|---| +| Omitted means absent | no `--concurrency` -> no `source.concurrency` patch | config-owned concurrency is overwritten by parser default | +| Falsy can be explicit | `--no-range-cache` -> `{ enabled: false }`; `--retries 0` -> `{ retries: 0 }` | `if (value)` drops user intent | +| Empty arrays can be explicit | config operation can clear inherited routes | inherited values cannot be cleared | +| Aliases normalize once | `--out` wins over `--out-dir`; `--route`, `--route-url`, `--target` feed one route list | provenance and docs disagree about winner | +| Profile expansion is source policy | browser `capture_profile=max` synthesizes WARC/WACZ/discovery/screenshots settings | hidden defaults look like Zod or CLI defaults | +| Adapter output is revalidated | `projectConfigPatchSchema.parse(patch)` | invalid patch travels to merge/runtime code | + +Do not use parser defaults for source-bearing terms merely to show help text. +Use help-only metadata such as `documentDefault()` or parser documentation +support, then let the complete runtime schema apply executable defaults. + +## c12 loading details + +c12 owns project config discovery and authoring composition, not the final +runtime contract. + +Kaiju-style loader policy: + +```ts +await loadConfig({ + name: "project", + cwd, + configFile, + configFileRequired, + rcFile: false, + globalRc: false, + packageJson: false, + dotenv: false, + envName, + omit$Keys: true, + merger: mergeC12ConfigInputs, + context, +}); +``` + +Why each option matters: + +| Option | Reason | +|---|---| +| `configFile` / `configFileRequired` | explicit `--config` has clear missing-file behavior | +| `rcFile: false`, `globalRc: false`, `packageJson: false` | avoids hidden user/package config sources unless product policy chooses them | +| `dotenv: false` | dotenv participates through Optique env context in this design, not c12 mutation | +| `envName` | selected c12 env branch is explicit and testable | +| `omit$Keys: true` | c12 control metadata does not leak after loading | +| `merger: mergeC12ConfigInputs` | loader-aware merge keeps `extends` and env branches usable | + +Validate loaded exports with an authoring schema that accepts: + +- root object layers with c12 controls; +- root arrays as shorthand for several app patches; +- functions/factories when c12 supports them; +- array operation envelopes on known array-bearing fields. + +Reject: + +- scalar exports; +- array-shorthand elements that contain c12 control keys; +- complete runtime defaults during layer validation; +- source adapters receiving `$append`, `$prepend`, `$replace`, `extends`, or + `$development` objects. + +The loader merger must sometimes return `false` to let c12 continue its own +merge handling. The app merger may be stricter because c12 has already resolved +loader controls. + +## Config merge and provenance mechanics + +Kaiju uses different schema shapes and different merge moments: + +| Shape | Purpose | Defaults allowed? | +|---|---|---| +| authoring layer | config-file syntax, c12 root controls, array operations | no complete runtime defaults | +| sparse patch | CLI/env/file contribution after authoring normalization | no complete runtime defaults | +| complete runtime config | handler and source run contract | yes, Zod owns executable defaults | + +Observed two-merger algorithm: + +```text +c12 loadConfig(..., merger: mergeC12ConfigInputs) + -> c12 resolves files, extends, env branches, and factories + -> c12DefuMerger preserves loader controls and defers append/prepend without inherited arrays + -> normalizeLoadedConfig(loaded.config) + -> filePatch + +resolveConfig() + -> envPatch from process environment + -> cliPatch from Optique adapter + -> mergeConfigInputs(cliPatch, envPatch, filePatch) + -> public inputs are high-to-low, implementation loops low-to-high + -> resolveStandaloneOperations(resolved) + -> projectConfigPatchSchema.parse(...) + -> sanitize c12 controls and undefined values + -> projectConfigSchema.parse(...) applies final defaults + -> buildConfigProvenance(...) +``` + +Array and union merge rules: + +| Input shape | Merge rule | +|---|---| +| scalar | higher defined value wins; null/undefined layers are skipped according to patch policy | +| ordinary object | recursive property merge | +| plain array | higher array replaces inherited array; no default defu concat | +| `$replace` | operation value replaces inherited array | +| `$append` | inherited array then appended values | +| `$prepend` | prepended values then inherited array | +| standalone operation | lowered to its operation array after all pair merges | +| schema-known atomic union | higher union object replaces inherited union object as a whole | + +Standalone operation lowering is required because a lowest-layer operation has no +inherited array for defu's pair callback to inspect: + +```ts +function resolveStandaloneOperations(value: unknown): unknown { + if (isArrayReplaceOperation(value)) return value.$replace; + if (isArrayAppendOperation(value)) return value.$append; + if (isArrayPrependOperation(value)) return value.$prepend; + if (Array.isArray(value)) return value.map(resolveStandaloneOperations); + if (!isRecord(value)) return value; + return Object.fromEntries( + Object.entries(value).map(([key, child]) => [key, resolveStandaloneOperations(child)]), + ); +} +``` + +Provenance rules: + +- Build provenance from source-layer decisions, not final values alone. +- When `applyConfigPatch(base, patch, "optique", overrides)` overlays a command + patch, preserve base provenance for unchanged fallback values unless explicit + argv aliases map to that path. +- Attribute explicit CLI leaves through reviewed aliases such as `--route`, + `--route-url`, `--target`, `--range-cache`, `--no-range-cache`, `--out`, + `--out-dir`, `-v`, `--verbose`, and `--silent`. +- Arrays and atomic unions usually receive provenance at the owning field path, + not stale numeric indexes or inherited subfields. +- Redact provenance before `config show`, `config explain`, manifests, support + bundles, or LogTape routes. + +### Merge algorithm detail + +The public app merge receives inputs highest-to-lowest: + +```ts +mergeConfigInputs(cliPatch, envPatch, filePatch); +``` + +It evaluates them low-to-high so lower layers establish inherited values before +higher layers apply operations: + +```ts +let resolved: unknown = {}; +for (const layer of inputs.toReversed()) { + if (layer == null) continue; + resolved = defu(layer, resolved, defuMerger); +} +return projectConfigPatchSchema.parse(resolveStandaloneOperations(resolved)); +``` + +For a concrete route example: + +```ts +file = { sources: { browser: { routes: ["file"] } } }; +env = { sources: { browser: { routes: { $prepend: ["env"] } } } }; +cli = { sources: { browser: { routes: { $append: ["cli"] } } } }; + +mergeConfigInputs(cli, env, file) +// -> { sources: { browser: { routes: ["env", "file", "cli"] } } } +``` + +Do not reverse the public caller API to make the internal loop easier. The +public API should read like precedence: CLI, environment, file, defaults. The +implementation can reverse internally. + +### defu callback classification + +The merger needs field classification, not one universal deep-merge: + +| Classification | Callback behavior | +|---|---| +| plain array | assign the higher array; return true | +| `$replace` operation | assign replacement array; return true | +| `$append` operation with inherited array | assign `[...inherited, ...append]`; return true | +| `$prepend` operation with inherited array | assign `[...prepend, ...inherited]`; return true | +| `$append`/`$prepend` without inherited array during c12 load | return false to defer | +| `$append`/`$prepend` without inherited array during app merge finalization | leave for standalone lowering | +| atomic union path | assign higher object as a whole; return true | +| ordinary object | return false so defu recurses | + +Required invariants: + +- never mutate incoming layer objects; +- always copy resolved arrays; +- preserve duplicate array entries unless the product explicitly defines a set; +- empty arrays clear inherited arrays; +- only schema-known atomic union paths are atomic. + +### Provenance overlay detail + +Field provenance follows merge semantics. The implementation should know whether +a field was: + +1. explicitly supplied by CLI argv; +2. emitted by Optique from environment, config, or derived source context; +3. inherited from the loaded base snapshot; +4. supplied by the final Zod default. + +Kaiju's `inferExplicitCliProvenance()` collects supplied flags from argv and +maps aliases back to config paths: + +```ts +const PATH_FLAG_ALIASES = { + "analyzedAt": ["analyzed-at", "at"], + "log.level": ["log-level", "verbose", "v", "silent"], + "rangeCache.enabled": ["range-cache", "no-range-cache"], + "route": ["route", "route-url", "target"], + "routes": ["route", "route-url", "target", "route-urls", "routes-file", "stdin"], +}; +``` + +Do not mark every Optique-returned value as `cli`. A config fallback selected by +Optique can be equal to the resolved base value and should retain base +provenance unless argv proves explicit user intent. + +## Field-extension checklist + +Use this whenever adding a flag, env variable, config key, runtime field, output +field, plan field, checkpoint field, or provenance path. Most CLI bugs in this +stack happen because a new field is added in one layer and silently omitted from +another. + +### 1. Classify the field before writing code + +| Question | If yes | Owner | +|---|---|---| +| Does it change parser/bootstrap behavior before config is valid? | early control | root parser/entrypoint | +| Can it come from CLI, env, config, or default? | source-bearing config | Optique adapter + config resolver + Zod runtime schema | +| Is it a domain target/request input? | domain request | command adapter + domain schema | +| Does it alter output transport or shape? | result/diagnostic selector | renderer + LogTape routing | +| Does it mutate external state? | operation policy input | plan/risk/confirmation schema | +| Does it persist between runs? | state/checkpoint/config location field | storage/path/config adapter | +| Is it secret or sensitive? | secret reference | `SecretSource` + redaction policy | + +If the field is source-bearing, fill this map before implementation: + +```text +public spelling: + CLI flags: + env var: + config path: + default documentation: + +schemas: + authored layer: + sparse patch: + complete runtime: + +merge category: + scalar | object | plain array | operation array | atomic union + +provenance: + canonical path: + aliases: + redaction: + +consumers: + handler: + source package: + result/stage/artifact: + +tests: + parser: + sparse adapter: + merge: + defaults: + provenance: + subprocess: +``` + +### 2. Put defaults in the right layer + +| Default need | Correct placement | Test oracle | +|---|---|---| +| Help text says what will happen if omitted | Optique documentation metadata such as `documentDefault()` | help contains value; parse output omits field | +| Invocation-local Boolean/output default | Optique `withDefault()` on non-source-bearing control | omitted parse has safe control value; config cannot own it | +| CLI-only expensive/secret/interactive fallback | Optique `deferredValue()` or prompt integration gated by policy | fallback function runs only when handler needs it | +| Runtime config fallback | complete Zod schema `.default()` or `.prefault()` after merge | sparse patch omits field; runtime config contains fallback | +| Transformed default input | Zod `.prefault()` | fallback passes through transform/refinement | +| Malformed recovery | Zod `.catch()` only when recovery is explicit product policy | invalid value produces recovery record or expected fallback | + +Do not move a runtime default into Optique because help cannot display it. Fix +the help metadata instead. + +### 3. Add schema shapes deliberately + +For source-bearing config, three shapes should exist or be consciously rejected: + +| Shape | Field form | Defaults? | Validation job | +|---|---|---|---| +| authored layer | sparse values plus optional operation envelopes and c12 controls | no runtime defaults | accept config-file language | +| sparse patch | ordinary sparse values after operations/c12 controls are gone | no runtime defaults | accept CLI/env/file contribution | +| complete runtime | complete normalized values | yes | contract consumed by handlers/source packages | + +Example for an operation-capable array: + +```ts +const routeArraySchema = z.array(z.url()); +const routeOperationSchema = z.union([ + z.object({ $replace: routeArraySchema }), + z.object({ $append: routeArraySchema }), + z.object({ $prepend: routeArraySchema }), +]); + +const authoredBrowserSchema = z.object({ + routes: z.union([routeArraySchema, routeOperationSchema]).optional(), +}); + +const browserPatchSchema = z.object({ + routes: routeArraySchema.optional(), +}); + +const browserRuntimeSchema = z.object({ + routes: routeArraySchema.default([]), +}); +``` + +### 4. Register merge category and provenance at the same time + +Adding the schema without merge/provenance rules is incomplete. + +| Field category | Merge behavior | Provenance granularity | +|---|---|---| +| scalar | higher defined wins; preserve false/0/empty string if schema allows it | leaf path | +| ordinary object | recursive per-property merge | leaf paths | +| plain array | higher array replaces; empty clears | owning array path | +| operation array | operation transforms inherited array; lowered to plain array | owning array path plus operation source when explain supports it | +| atomic union | higher object replaces whole variant | union owner path | +| secret reference | merge reference, never secret value | reference path, redacted | + +If a field has aliases, add them to the explicit CLI-provenance map while the +parser is changed. Provenance cannot be reliably reconstructed later from the +final value because an explicit CLI value can equal a lower config value. + +### 5. Extend Optique as grammar, not config resolver + +Optique work for a new field should include: + +- canonical long flag and reviewed aliases; +- value parser or `@optique/zod` adapter for one-token validation when useful; +- `negatableFlag()` for tri-state Boolean overrides; +- source-context bindings only when the field truly belongs in env/config; +- help/man/completion metadata and examples; +- parser tests for omitted, explicit, invalid, alias, and source-context cases. + +Optique work should not include: + +- final runtime defaults for source-bearing config; +- operation-aware merging; +- secret values in argv; +- domain safety policy such as whether an action is severe. + +### 6. Extend c12/defu/Zod as a staged resolver + +For config-file fields: + +1. add authored syntax; +2. add retained c12 layer validation; +3. add sparse patch schema; +4. classify merge behavior; +5. add standalone operation cleanup if operations are supported; +6. apply complete Zod defaults once; +7. add `config show` and `config explain` rendering with redaction. + +Do not validate file layers with the complete runtime schema. That applies +defaults too early and makes absent config indistinguishable from authored +values. + +### 7. Add output/stage/artifact behavior if the field affects observability + +| If field changes... | Update | +|---|---| +| stable stdout result | result schema, renderer, JSON/plain fixtures | +| diagnostics | LogTape category/filter/formatter tests | +| persisted stage JSONL | stage payload schema, writer tests, migration/version note | +| artifact path/content | artifact manifest schema, digest/size/reference fields | +| support bundle/config explain | redaction and provenance rendering tests | + +Never prove a field by checking only TypeScript types. A CLI field is public +only when help, parser, source resolution, runtime behavior, and observable +output agree. + +### 8. Minimum behavior tests for every source-bearing field + +| Behavior | Example assertion | +|---|---| +| no source | runtime default applies after sparse merge | +| config source | config value appears in runtime config and provenance | +| env source | env beats config | +| explicit CLI | CLI beats env/config and provenance says CLI | +| explicit CLI equal to config | final value equals config, but provenance still says CLI | +| omitted CLI | config/env provenance is preserved | +| invalid CLI | parser/schema reports public flag name | +| invalid config | c12/file path and Zod issue path are visible | +| help/default | help shows documented default while parse output stays sparse | +| redaction | sensitive field is hidden in config show/explain/logs | + +## LogTape integration details + +LogTape gives the CLI one structured observation graph. It does not replace +schemas, config merge, stage writers, or artifact storage. + +### What LogTape does for the CLI + +| Need | LogTape role | +|---|---| +| Stable stdout | dedicated result category and raw sink | +| Human diagnostics | pretty/plain stderr sink with levels and categories | +| Machine diagnostics | JSON/JSONL diagnostic sink | +| Redaction | wrap every result and diagnostic sink before serialization | +| Bootstrap failures | minimal early configuration from raw logging flags | +| Library logging | packages receive loggers or structural logger contracts, not sink setup authority | +| Testability | recorder/test sinks assert category and structured properties | +| Lifecycle | `reset()`/flush/disposal prevents duplicate process-global sinks | + +### Result route + +Stable results must be pipe-safe: + +```ts +export function printCliResult(value: unknown, json: boolean, human: string): void { + const result = json ? `${JSON.stringify(value, null, 2)}\n` : `${human}\n`; + emitKaijuResult(result); +} +``` + +The underlying LogTape config should isolate result output: + +```ts +{ + category: ["kaiju", "result"], + sinks: ["result"], + parentSinks: "override", +} +``` + +If `parentSinks` is omitted or equivalent isolation is absent, JSON output can +inherit diagnostic prefixes and break automation. + +### Diagnostic route + +Diagnostics should carry structured properties: + +```ts +getKaijuLogger("cli", "entrypoint").error("{message}", { + error, + message: `Config failed: ${error.message}`, + command, +}); +``` + +Do not stringify the entire error/config/result before redaction. Field-based +redaction must see nested keys such as `authorization`, `password`, `cookie`, +`token`, and secret URLs before formatting. + +### Bootstrap failure route + +Some failures occur before the full parser or loaded config exists. Kaiju uses a +small raw-argv recovery path: + +```text +raw argv + -> readPreHandlerLoggingOptions() + -> configureKaijuLogging() + -> log one redacted entrypoint error + -> resetKaijuLogging() + -> exit usage/config class +``` + +This bootstrap parser should recover only logging controls such as +`--log-level`, `--log-format`, `--log-output`, and `--silent`. It must not +duplicate the whole domain parser. + +### What LogTape must not do + +- It must not be the durable stage writer. +- It must not store raw WARC/browser bytes. +- It must not decide config precedence. +- It must not apply runtime defaults. +- It must not be configured by reusable packages. +- It must not emit stable results through diagnostic formatters. + +## Trace: browser detect + +Representative command: + +```text +kaiju browser detect --route example.com --capture-profile max --wacz --out out/browser +``` + +Flow: + +```text +argv + -> browserOptionsParser + shared route/output/run/log parsers + -> resolveRouteUrls(..., normalizeBrowserRouteUrl) + -> createBrowserConfigPatch(options, routeUrls) + -> sparse project patch + -> resolveCommandConfig({ patch }, hookContext) + -> runBrowserProject({ config: resolved.config }) + -> optional discovery + -> browser adapter plan/collect + -> detection/facts/aggregates/derived/lead stages + -> run summary, archive manifest, optional WACZ + -> stable CLI result +``` + +Key implementation details: + +| Behavior | Implementation detail | Test idea | +|---|---|---| +| Host-only route convenience | browser route normalizer adds `https://` when no scheme exists | `example.com` becomes `https://example.com/` | +| Sparse route patch | routes assigned only when direct/file/stdin input exists | no route input preserves config routes | +| Max capture profile | `maxCaptureDefault()` fills discovery, WARC/WACZ, screenshots, CDP/storage/DNS/TLS/body policy | profile test asserts synthesized fields; normal omission test asserts absence | +| Headed flag | `--headed` maps to `headless: false` | absent headed does not patch headless | +| Output precedence | `resolveCliOutDir()` returns `out ?? out_dir ?? default` | `--out` beats env/config `out_dir` | +| Resume marker | existing `run.json` blocks unless resume/overwrite | subprocess or temp-dir test | +| WACZ dependency | WACZ requires WARC archive writer | WACZ enabled without archive throws | +| Archive profile | replay/analysis/forensic choose stage retention | WACZ package includes expected metadata set | + +## Trace: Common Crawl detect + +Representative command: + +```text +kaiju commoncrawl detect --route https://example.com --crawls latest:2 --range-cache --out out/cc +``` + +Flow: + +```text +argv + -> commonCrawlOptionsParser + -> createCommonCrawlConfigPatch(options, routeUrl, multipleRoutes) + -> crawlSelector() produces named/latest/range selector + -> rangeCache/network/source patches stay sparse + -> resolveCommandConfig() + -> runCommonCrawlProject() + -> adapter.plan() queries CDX and emits decision stages + -> range fetcher uses rate limiter, retry policy, cache policy + -> collected WARC units feed shared detection pipeline + -> summarizeCommonCrawlDecisions(stages) + -> run-summary.json and CLI result +``` + +Key implementation details: + +| Behavior | Implementation detail | Test idea | +|---|---|---| +| Multi-route output | explicit base plus route slug per route | two routes produce sibling subdirectories | +| Atomic crawl selector | `{ kind: "named" }`, `{ kind: "latest" }`, `{ kind: "range" }` replace each other | higher selector drops stale lower keys | +| Range cache tri-state | absent/true/false conflict handled by parser/schema | absent preserves config; negative flag sets false | +| Serialized range safety | range concurrency constrained until dedicated bounded parallel tests exist | invalid higher value errors with actionable message | +| Retry policy | 429/503/network retry with bounded delay and cap | retry-policy unit tests for Retry-After and max delay | +| Stage-derived summary | CDX decision summary is computed from stage lines | enqueue/reject/alias counts match stage fixtures | + +## Trace: local WARC detect + +Representative command: + +```text +kaiju warc detect --warc ./capture.warc.gz --route https://example.com --out out/warc +``` + +Flow: + +```text +argv + -> warcOptionsParser + route/output/run/log parsers + -> resolveSingleWarcRouteUrl() rejects several routes + -> createWarcConfigPatch(options, routeUrl) + -> resolveCommandConfig() + -> runWarcProject() + -> prepareRunOutputDir() + -> createWarcAdapter().plan() + -> source.warc records/resources/observations stages + -> shared detector/fact pipeline + -> finalization audit, run summary, CLI result +``` + +Key implementation details: + +| Behavior | Implementation detail | Test idea | +|---|---|---| +| Single route | WARC route helper rejects more than one route | route list with two URLs throws source-specific message | +| Marker conflict | existing `run-summary.json` blocks unless overwrite | temp output dir conflict test | +| Source terminology | WARC stages are written before detector observations | stage fixture includes source and detect layers | +| Worker lifecycle | detection worker pool closes in `finally` | failing adapter still closes workers | +| Resource records | browser-generated WARC resource records may be parsed separately from HTTP responses | readback test counts record types | + +## Trace: domains verify + +Representative command: + +```text +kaiju domains verify --input domains.txt --audit --output accepted.txt --retry-output retry.txt +``` + +Flow: + +```text +argv + -> domainVerifyOptionsParser + -> createDomainVerifyConfigPatch() + -> sparse domains source patch + -> resolveCommandConfig() + -> runDomainVerificationProject() + -> input records stream through bounded verification + -> DNS and HTTP classify accepted/retry/rejected + -> accepted, routes, retry, audit, and summary outputs + -> source.domains verification/summary stages + -> CLI summary result +``` + +Key implementation details: + +| Behavior | Implementation detail | Test idea | +|---|---|---| +| Direct input selects output root | input or explicit output files cause source/project outDir assignment | adapter test for input-only and output-only | +| `--no-www` | maps to `tryWww: false` | absence preserves config; flag disables www attempt | +| Window/concurrency relation | schema rejects window smaller than concurrency | domain config test | +| Retry classification | temporary DNS, timeout, invalid status, 5xx-like outcomes can be retryable | DNS/HTTP unit tests | +| Retry file semantics | retry output contains retryable inputs, not all rejects | run test with accepted/rejected/retry rows | +| Audit mode | writes per-domain stage records and summary | audit path and stage counts asserted | + +## Stage, artifact, result, and diagnostic boundaries + +Three channels stay separate: + +```text +stable result: requested stdout/destination bytes +operational logs: stderr or diagnostic sinks +artifacts/stages: typed durable files, archives, checkpoints, summaries +``` + +Stage writer contract: + +```ts +export interface StageWriter { + write(stage: StageName, data: StageRecordInput): Promise; + flush(): Promise; + snapshot(stage: StageName): readonly StageLine[] | undefined; + pathFor(stage: StageName): string; + counts(): Readonly>; +} +``` + +Envelope: + +```ts +{ + schema_version: 1, + run_id, + stage, + seq, + target, + data, + written_at, +} +``` + +Rules: + +- durable stage JSONL is written by the stage writer, not by arbitrary log calls; +- known payloads use stage-specific Zod schemas; +- unknown payloads must still be JSON-compatible; +- `Date` values serialize to ISO strings; +- `undefined` object fields are dropped; +- raw `ArrayBuffer` or typed-array payloads are rejected; +- binary evidence belongs in WARC/WACZ/artifact storage with digest, byte count, + and path/reference in the stage line; +- LogTape may mirror the validated stage line under a structured property, but + that mirror is not the durable source of truth. + +Result/diagnostic rules: + +- stable JSON output must not contain pretty diagnostics, colors, timestamps, or + category labels; +- diagnostics must not go to stdout in machine-result mode; +- redaction must happen before rendering; +- pre-handler failures recover only raw logging controls and must leave stdout + empty unless the command explicitly requested a result. + +## Lifecycle, concurrency, cancellation, and cleanup + +Target lifecycle: + +```text +composition root owns signal + -> one AbortController tree + -> parser/run resource gets signal + -> HTTP/browser/workers/queues receive signal + -> first interrupt requests cooperative stop + -> second interrupt forces termination + -> stages/diagnostics/artifacts flush and close + -> stable cancellation exit +``` + +Observed Kaiju behaviors to account for: + +| Area | Behavior | Guardrail | +|---|---|---| +| Source run signal | WARC/Common Crawl/browser runs create an `AbortController` internally | Do not claim process-level cancellation until a root signal handler is traced and subprocess-tested | +| Browser route queue | `runBoundedQueue()` limits active routes and stops taking new work when `shouldStop()` is true | active work must also observe the signal | +| Ordered map helper | `mapBoundedAsyncOrdered()` permits concurrent transforms but yields in input order with bounded reorder window | stalled early item can apply backpressure after window fills | +| Detection worker pool | workers compile registry once and process whole observation batches | close in `finally`; fail queued/pending tasks if worker fails | +| Stage writer | per-stage promise queue serializes append order | always `flush()` before summary trust or process exit | +| Diagnostics sink | JSONL writes are queued and reset flushes high-volume buffers | avoid reconfiguring LogTape while active work may still log | +| Browser archive | archive closes on success and in error catch; WACZ packaging reads closed WARC output | close before packaging and verify WACZ readback when claiming replay support | +| Output overwrite | browser uses `run.json`; WARC/Common Crawl use `run-summary.json` marker | destructive overwrite must be explicit and scoped | + +## Verification matrix + +| Behavior | Evidence target | +|---|---| +| Static command reachability | installed/compiled executable reaches every registered command and generated help | +| Help-only defaults | parser docs show default; parse output omits the value | +| Sparse adapters | adapter tests assert omitted options do not enter patch | +| Profile expansion | browser max profile tests assert only documented policy-implied fields | +| Alias precedence | route/output/log aliases map to one canonical field and provenance path | +| c12 loader semantics | real config fixtures cover `extends`, env branch, JS/TS factory, malformed export | +| App merge semantics | unit tests cover high-to-low caller order, low-to-high evaluation, arrays, operations, atomic unions, immutability | +| Zod final defaults | sparse patch parse has no complete defaults; complete schema parse does | +| Provenance | env/config/default/explicit CLI/equal-value overlay cases are all tested | +| Result isolation | stdout, stderr, diagnostic file, and result string are separately asserted | +| Redaction | nested secrets in config, provenance, diagnostics, errors, and result views are redacted before rendering | +| Stage writer | seq, queue ordering, known schemas, JSON fallback, binary rejection, noop snapshot, counts, flush | +| Browser source | discovery, resume, overwrite, WARC, WACZ, screenshots, failed route stages | +| Common Crawl source | CDX query/page/row/decision stages, retry stages, rate limiter, range cache, summary counts | +| WARC source | single route, record/resource classification, worker close, output marker | +| Domains source | accepted/routes/retry outputs, audit stages, retry classifications, summary file | +| Cancellation | subprocess sends interrupt and checks active work stops, resources flush, exit class is stable | +| Command-language compatibility | long names, aliases, hidden aliases, suggestions, help, completion, and man surfaces stay consistent | +| Interaction safety | `--no-input`, CI/non-TTY, prompts, and missing required values never hang | +| Dangerous operations | dry-run emits executor plan; `--force` and `--confirm=` gates match risk | +| Standard streams | `-` maps to stdin/stdout only for file-like fields; required piped input fails fast on TTY | +| Secrets | argv/env snapshots, config views, URLs, diagnostics, support bundles, and results redact references/values | +| Pager policy | pager activates only for long human TTY output and never for JSON/JSONL/redirected stdout/CI | +| Config edits | target, diff, dry-run, consent, atomic write, and authored-schema validation are all covered | +| Checkpoints/resume | interrupted runs reconcile state, reject incompatible fingerprints, and resume idempotent units | +| Telemetry consent | remote sinks are disabled by default, revocable, redacted, and non-blocking | +| Installed shape | compiled/packed artifact reaches commands, assets, workers, config loading, help, and uninstall/path inspection | + +## Failure signatures + +| Symptom | Likely boundary bug | +|---|---| +| Config value ignored when CLI flag omitted | parser default polluted sparse patch | +| Help default appears in dry-run patch | documented default was implemented as executable source value | +| Dynamic config factory executes twice | source context and handler both call full resolver | +| `extends` is rejected | strict runtime validation ran inside c12 loading | +| `$append` merges into `$append` object | c12/app merger distinction or standalone operation lowering is missing | +| Plain arrays concatenate | default defu behavior leaked past app merger policy | +| Common Crawl range selector keeps stale named selector fields | atomic union path was not handled | +| Equal CLI/config value loses config provenance | provenance inferred from final value only | +| `config explain` shows only static precedence | resolver did not retain field decisions and shadowed values | +| JSON stdout contains warning text | LogTape result and diagnostic routes are mixed | +| Redaction misses a result | object was stringified before field redaction | +| Stage JSONL contains bytes | artifact writer boundary was bypassed | +| WACZ exists but replay fails | package structure was validated without URL-targeted WARC/CDX readback | +| `Ctrl-C` does not stop work | AbortSignal type exists but is not connected to process and active operations | +| Retry file contains all rejects | domain retry classification collapsed retry and terminal rejection | +| Raising Common Crawl concurrency passes type-check | service-safety policy lacks behavior tests | +| Source command works from source but not binary | dynamic command discovery or assets are invisible to bundler/compiler | +| `--no-input` still prompts | prompt adapter bypasses interaction policy or TTY/CI gate | +| Severe destructive command accepts `--force` alone | risk policy collapsed moderate and severe operations | +| Dry-run summary differs from execution | dry-run renderer builds an approximate explanation instead of executor `OperationPlanSchema` | +| `-` creates a file named `-` unexpectedly | endpoint parser did not map standard-stream sentinel contextually | +| Secret appears in shell history or support bundle | CLI accepted secret value directly or stringified before redaction | +| JSON output opens in pager | pager attached to stdout/result route without output-mode and TTY checks | +| Config edit rewrites unrelated formatting | editor ignored format-preserving adapter or broad formatted authored config | +| Resume corrupts output after argument change | checkpoint lacks normalized request fingerprint/precondition check | +| Telemetry failure changes exit code | remote diagnostic/product sink is on critical execution path | +| Uninstall docs miss user state/cache paths | distribution contract lacks owned-path inventory command | + +## Sources and freshness + +- Observed implementation: attached Kaiju CLI source tree, including Optique 1.2 + migration, c12 two-merger config behavior, LogTape routing, stage writer, + browser/WARC/Common Crawl/domain source runs, and Zod default boundaries. +- Normative source: `productionized-cli-pattern-guidebook-v1.2.md`, reviewed + 2026-07-22. +- Expansion notes: `productionized-cli-pattern-guidebook-v1.2-notes.md`, + `cli-guidelines-audit-and-expansion.md`, and + `kaiju-config-resolution-handoff.md`, reviewed 2026-07-22. +- Re-check installed package exports, lockfile versions, and current tests before + copying APIs or behavior into another repository. diff --git a/skills/build-clis/references/commands.md b/skills/build-clis/references/commands.md index 0ca12a2..3423950 100644 --- a/skills/build-clis/references/commands.md +++ b/skills/build-clis/references/commands.md @@ -1,5 +1,8 @@ # Command language and Optique +For package-level Optique APIs, ecosystem selection, version boundaries, and +worked parser/source examples, load [optique.md](optique.md). + ## Design the language first Define nouns, verbs, nesting, defaults, aliases, destructive operations, output diff --git a/skills/build-clis/references/config.md b/skills/build-clis/references/config.md index 29f9e16..7b937dd 100644 --- a/skills/build-clis/references/config.md +++ b/skills/build-clis/references/config.md @@ -1,5 +1,8 @@ # Configuration resolution +For the complete c12/defu/jiti lifecycle, merge implementation, authored +operations, and diagnostic playbook, load [c12-defu.md](c12-defu.md). + ## Three different shapes Keep these contracts distinct: diff --git a/skills/build-clis/references/defaults-provenance.md b/skills/build-clis/references/defaults-provenance.md new file mode 100644 index 0000000..1b143b9 --- /dev/null +++ b/skills/build-clis/references/defaults-provenance.md @@ -0,0 +1,391 @@ +# Defaults and provenance across Optique, c12, defu, and Zod + +Load this reference when defaults feel duplicated, an omitted CLI option masks +configuration, help cannot describe runtime behavior, or `config explain` must +show why a value won. + +## Contents + +1. [The three representations](#the-three-representations) +2. [Default kinds and owners](#default-kinds-and-owners) +3. [`@optique/zod` is an adapter](#optiquezod-is-an-adapter) +4. [Zod v4 default semantics](#zod-v4-default-semantics) +5. [The complete resolution pipeline](#the-complete-resolution-pipeline) +6. [A field descriptor pattern](#a-field-descriptor-pattern) +7. [A provenance envelope](#a-provenance-envelope) +8. [Merge and provenance algorithm](#merge-and-provenance-algorithm) +9. [Arrays, objects, and union branches](#arrays-objects-and-union-branches) +10. [Evidence limits](#evidence-limits) +11. [Implementation sequence](#implementation-sequence) +12. [Required tests](#required-tests) + +## The three representations + +Do not use one schema as all three of these objects: + +| Representation | Meaning | May contain defaults? | +|---|---|---| +| Authored input | File syntax, shorthands, array operations, aliases | No executable defaults | +| Sparse patch | Facts contributed by one source | No | +| Complete runtime config | Final executable values after precedence | Yes | + +The key invariant is: + +```text +absence in a source patch means “this source made no decision” +``` + +Turning absence into a value before precedence resolution manufactures a +decision and can make a lower source unreachable. + +## Default kinds and owners + +The word “default” describes different behavior. Name the kind in code review. + +| Kind | Owner | Materializes during parse? | Participates in provenance? | +|---|---|---:|---:| +| Help/documentation default | Optique term metadata | No | No | +| Parser-local fallback | Optique `withDefault()` or equivalent | Yes | Only when parser is sole owner | +| Deferred handler fallback | Optique `deferredValue()` | When invoked | Yes, as a distinct deferred source | +| Derived source | Source adapter | Yes, as a low-priority patch | Yes | +| Runtime default | Complete Zod schema | After source merge | Yes, as `runtime-default` | +| Recovery value | Zod `.catch()` | After validation error | Yes, as recovery, never ordinary default | + +Use this decision table: + +| Question | Mechanism | +|---|---| +| Should help/man describe the value while omission stays absent? | Help-only metadata | +| Is Optique the only possible owner of the setting? | Parser-local default may be valid | +| Does evaluation require a prompt, secret store, or handler resource? | `deferredValue()` | +| Is it computed deterministically from already resolved context? | Derived source | +| Is it the final fallback for executable configuration? | Zod `.default()` or `.prefault()` | +| Should invalid user input silently become a fallback? | Usually reject; do not use `.catch()` casually | + +## `@optique/zod` is an adapter + +`@optique/zod` lets an Optique value parser use a Zod schema for one parsed +value. It can improve diagnostics, transformations, and metadata. It does not +make the Zod schema the owner of multi-source precedence. + +```ts +import { option } from "@optique/core/primitives"; +import { optional } from "@optique/core/modifiers"; +import { zod } from "@optique/zod"; +import * as z from "zod"; + +const endpointValueSchema = z.url(); + +const endpointTerm = optional(option( + "--endpoint", + zod(endpointValueSchema, { placeholder: "https://example.com" }), +)); +``` + +The placeholder documents or supports parsing machinery. It is not an entered +value, a config fallback, or provenance. Parsing no `--endpoint` must still +produce absence. + +Avoid putting a runtime default in the scalar adapter: + +```ts +// Wrong for a source-bearing option: omission becomes a CLI contribution. +zod(z.url().default("https://api.example.test")); +``` + +Instead, share a non-defaulted leaf schema and apply the default only in the +complete object: + +```ts +const endpointSchema = z.url(); + +const CliPatchSchema = z.object({ + endpoint: endpointSchema.optional(), +}); + +const RuntimeConfigSchema = z.object({ + endpoint: endpointSchema.default("https://api.example.test"), +}); +``` + +For a finite vocabulary, prefer an Optique `choice()` built from the schema’s +options when the installed version supports it. That lets Optique generate +choices and suggestions directly. Use the Zod adapter for scalar validation or +transformations that Optique should perform at token parse time. + +## Zod v4 default semantics + +Zod v4 `.default(value)` returns the default when input is `undefined`; the +default must already satisfy the schema’s output type. It can short-circuit +transforms. `.prefault(value)` supplies an input value and then runs the normal +parse pipeline. + +```ts +const normalizedDirectory = z.string() + .trim() + .transform((value) => value.replace(/\/$/, "")); + +const direct = normalizedDirectory.default("./dist"); +const parsed = normalizedDirectory.prefault(" ./dist/ "); + +direct.parse(undefined); // "./dist"; default is already output-shaped +parsed.parse(undefined); // "./dist"; fallback passes through trim + transform +``` + +Choose by shape, not preference: + +| Fallback is shaped like | Use | +|---|---| +| Final output after transforms | `.default()` | +| Raw input that must be normalized/refined | `.prefault()` | +| Invalid-input recovery policy | `.catch()`, with explicit provenance and tests | + +Nested defaults require deliberate object construction. If the outer object is +optional and absent, an inner default may never run unless the outer schema +receives an input object: + +```ts +const LoggingSchema = z.object({ + level: z.enum(["debug", "info", "warn", "error"]).default("info"), + format: z.enum(["pretty", "json"]).default("pretty"), +}); + +const RuntimeConfigSchema = z.object({ + // The empty input object activates the inner field defaults. + logging: LoggingSchema.prefault({}), +}); +``` + +Test the exact composed schema. Zod object composition can change where +refinements and transformations live; do not infer behavior from a leaf schema. + +## The complete resolution pipeline + +```text +tokens ──Optique──> sparse CLI patch ─┐ +environment adapter ─> sparse patch ─┼─> app merger ─> complete Zod parse +c12 resolved config ─> sparse patch ─┤ │ +derived source ───────> sparse patch ─┘ ├─> runtime config +runtime default metadata ────────────────────────────────└─> provenance +``` + +Recommended precedence: + +```text +explicit CLI > environment > resolved project config > derived > runtime default +``` + +The public merge API may list inputs highest-to-lowest for readability. A +pairwise merger that evaluates append/prepend operations must usually walk +lowest-to-highest internally: + +```ts +export function mergeConfigInputs( + ...highestToLowest: readonly (T | null | undefined)[] +): T { + let resolved = {} as T; + for (const layer of highestToLowest.toReversed()) { + if (layer == null) continue; + resolved = mergePair(layer, resolved); + } + return resolveStandaloneOperations(resolved); +} +``` + +The c12 `merger` and the application merger are separate. The c12 merger must +preserve loader control fields long enough for `extends`, environment branches, +and config factories to resolve. Strict patch validation belongs after c12 has +finished producing application data. + +## A field descriptor pattern + +Repeated literals drift. For important source-bearing fields, store shared +facts without pretending all libraries consume the same object: + +```ts +const timeoutField = { + cli: "--timeout", + env: "APP_TIMEOUT", + configPath: ["network", "timeoutSeconds"] as const, + description: "Abort an HTTP request after this many seconds.", + documentedDefault: 30, + valueSchema: z.number().int().positive(), +} as const; + +const CliPatchSchema = z.object({ + timeoutSeconds: timeoutField.valueSchema.optional(), +}); + +const RuntimeConfigSchema = z.object({ + timeoutSeconds: timeoutField.valueSchema.default( + timeoutField.documentedDefault, + ), +}); +``` + +Adapters may derive their parser term, env key, authoring field, help text, and +runtime default from this descriptor. Keep adapter construction explicit so a +library upgrade cannot silently change all boundaries. + +## A provenance envelope + +Return the usable config and its explanation together: + +```ts +type SourceKind = + | "cli" + | "environment" + | "config" + | "derived" + | "runtime-default" + | "recovery"; + +interface Contribution { + readonly source: SourceKind; + readonly sourceId: string; + readonly path: readonly (string | number)[]; + readonly operation: "set" | "append" | "prepend" | "replace" | "default"; + readonly explicit: boolean; + readonly redactedValue: unknown; +} + +interface FieldDecision { + readonly winner: Contribution; + readonly shadowed: readonly Contribution[]; +} + +interface ResolvedConfigEnvelope { + readonly schemaVersion: 1; + readonly value: T; + readonly decisions: Readonly>; + readonly sources: readonly { + readonly sourceId: string; + readonly kind: SourceKind; + readonly location?: string; + }[]; +} +``` + +The machine envelope is versioned. Human `config explain` output is a rendering +of it, not a second resolver. Redact structured contributions before formatting +or passing them to LogTape. + +## Merge and provenance algorithm + +Do not reconstruct provenance by comparing the final value with source values. +Equal values can have different authorship. Record decisions while resolving. + +For each normalized source layer: + +1. Preserve whether every field was absent or explicitly present. +2. Convert source-native names to canonical application paths. +3. Validate the sparse patch without defaults. +4. Attach `{ source, sourceId, explicit: true }` to present fields. +5. Merge lowest-to-highest so operation order is meaningful. +6. At each affected path, record the contribution and operation. +7. When a higher scalar or atomic branch replaces a value, mark the earlier + contribution as shadowed. +8. For append/prepend, retain every contributing source in execution order. +9. After authored sources resolve, apply derived values only to absent paths. +10. Parse with the complete Zod schema. +11. Compare pre-default and post-default structure by presence, not value. Add a + `runtime-default` decision for paths materialized by Zod. +12. Redact values and source locations before exposing the envelope. + +Conceptual implementation: + +```ts +for (const layer of lowestToHighest) { + for (const contribution of flattenPresentFields(layer.patch, layer.meta)) { + applyContribution(valueDraft, contribution, decisionDraft, mergePolicy); + } +} + +applyMissingDerivedContributions(valueDraft, decisionDraft, derivedPatch); + +const beforeDefaults = structuredClone(valueDraft); +const value = RuntimeConfigSchema.parse(beforeDefaults); +recordMaterializedDefaults(beforeDefaults, value, decisionDraft); +``` + +`recordMaterializedDefaults()` needs schema-owned default metadata for perfect +accuracy when transforms create or remove keys. Do not label every post-parse +difference a default: transforms are a separate operation. A robust +implementation either records defaults through field descriptors or separates +normalization from default materialization into observable passes. + +## Arrays, objects, and union branches + +| Shape | Default merge rule | Provenance rule | +|---|---|---| +| Scalar | Higher explicit value replaces lower | One winner, lower values shadowed | +| Ordinary object | Merge known properties recursively | Decision per leaf path | +| Plain array | Higher array replaces lower | Array-level winner unless element identity is specified | +| Empty array | Explicitly clears inherited values | Must remain an explicit winner | +| Append/prepend | Evaluate against inherited array | Record all ordered contributors | +| Discriminated union | Replace branch atomically at schema-known path | Branch-level winner; old branch fields shadowed | +| Secret | Same merge behavior | Value always redacted in public envelope | + +Do not invent per-element array provenance unless elements have stable identity. +If routes have IDs, define merge and explanation by ID. If they do not, explain +the array as one value plus ordered operations. + +## Evidence limits + +Optique normally yields the selected value, not necessarily a complete token +trace proving whether the same value came from an explicit token or a source +fallback. Provenance must be based on evidence the installed integration +actually exposes. + +Use this hierarchy: + +1. Parser/integration metadata that explicitly identifies the winning source. +2. A separately parsed minimal presence term for explicitly supplied aliases. +3. Raw-token inspection for exact registered aliases and `--name=value` forms. +4. Value comparison only as a last resort, and label it inferred. + +Never infer that CLI was absent because its value equals config. The user may +have explicitly supplied the same value. Test canonical names, short aliases, +hidden compatibility aliases, repeated options, and negated flags. + +## Implementation sequence + +1. Inventory every default and classify it with the owner table. +2. Define authored, sparse, and complete schemas separately. +3. Remove defaults from source-bearing Optique terms and patch schemas. +4. Keep scalar Zod adapters non-defaulted. +5. Add help-only default metadata and prove omission remains absent. +6. Select `.default()` versus `.prefault()` from input/output shape. +7. Load c12 once and preserve its loader metadata until loading completes. +8. Normalize each layer to one sparse patch vocabulary. +9. Implement and unit-test the pair merger before adding provenance. +10. Record provenance during merge, not after it. +11. Apply complete Zod validation and record runtime defaults. +12. Configure LogTape after bootstrap controls are known; redact the envelope. +13. Inject `{ config, provenance, logger }` into handlers. +14. Verify help, config explain, runtime behavior, and the installed artifact. + +## Required tests + +| Scenario | Assertion | +|---|---| +| CLI omitted, env present | Env wins; CLI has no contribution | +| CLI omitted, config present | Config wins; runtime default is not applied | +| All authored sources absent | Zod materializes one runtime default decision | +| Explicit CLI equals config | CLI still wins; config is shadowed | +| Help-only default | Help/man contains it; parsed patch omits the field | +| Zod adapter placeholder | Never appears in patch, config, or provenance | +| `.default()` after transform | Test documents short-circuit behavior | +| `.prefault()` after transform | Fallback passes through transform/refinement | +| Nested object absent | Intended nested defaults do or do not materialize explicitly | +| `false`, `0`, empty string | Preserved or rejected by schema policy, never treated as absence | +| Empty array | Explicitly clears inherited array | +| Append/prepend | Value and ordered contributors are both correct | +| Atomic union branch change | No fields leak from the lower branch | +| Dynamic config factory | Side-effect counter proves one evaluation | +| Secret contribution | Redacted in human output, JSON, logs, and errors | +| Recovery fallback | Marked `recovery`, never `runtime-default` | + +See [testing.md](testing.md) for executable test layers and +[benchmarking.md](benchmarking.md) for measuring the cost of resolution and +provenance without weakening these invariants. diff --git a/skills/build-clis/references/ecosystems.md b/skills/build-clis/references/ecosystems.md index 92ce7c3..d8d17a2 100644 --- a/skills/build-clis/references/ecosystems.md +++ b/skills/build-clis/references/ecosystems.md @@ -3,6 +3,11 @@ Use `explore-ecosystems` to verify versions and relationships. This map assigns capabilities; it is not an instruction to install every package. +Load [optique.md](optique.md), [logtape.md](logtape.md), +[c12-defu.md](c12-defu.md), or [unjs.md](unjs.md) when one of those ecosystems +materially owns the task. Load [integration.md](integration.md) when several +owners must be composed without duplicating responsibility. + ## Command language Optique can own typed grammar, source binding, help, discovery, completion, and diff --git a/skills/build-clis/references/five-library-stack.md b/skills/build-clis/references/five-library-stack.md new file mode 100644 index 0000000..605c8a3 --- /dev/null +++ b/skills/build-clis/references/five-library-stack.md @@ -0,0 +1,816 @@ +# Optique, c12, defu, LogTape, and Zod stack + +## Use this reference + +Load this reference whenever a CLI uses two or more of these at the same +boundary: Optique, c12, defu, LogTape, and Zod. The failure mode is rarely that +one library is bad. The failure mode is that two good libraries both become the +owner of the same decision. + +The target mental model: + +```text +public CLI language + Optique terms, help, suggestions, completion, manuals + | + v +sparse source contributions + CLI patch, env patch, config patch, derived patch + | + v +project configuration loading + c12 discovers, evaluates, extends, environments, and layer metadata + | + v +application merge policy + defu-powered internals behind explicit app semantics + | + v +runtime contract + Zod validates, transforms, and applies defaults once + | + v +execution and observation + handler receives injected capabilities; LogTape routes redacted results +``` + +If an implementation cannot draw this graph for one option, one config key, one +result, and one failure, it is not done. + +## Contents + +1. [The ownership map](#the-ownership-map) +2. [Data shapes](#data-shapes) +3. [Default taxonomy](#default-taxonomy) +4. [Optique patterns](#optique-patterns) +5. [c12 and defu patterns](#c12-and-defu-patterns) +6. [LogTape patterns](#logtape-patterns) +7. [Zod patterns](#zod-patterns) +8. [One-snapshot execution](#one-snapshot-execution-recipe) +9. [End-to-end trace and audit](#end-to-end-trace-example) +10. [Required tests and failure signatures](#required-test-matrix) + +For the precise `.default()`/`.prefault()` rules, `@optique/zod` boundary, +nested-object defaults, and field-level provenance algorithm, load +[defaults-provenance.md](defaults-provenance.md). For performance work, load +[benchmarking.md](benchmarking.md); a microbenchmark must not replace the +semantic matrix in this reference. + +## The ownership map + +| Decision | Owner | Good sign | Bad sign | +|---|---|---|---| +| What token shapes are accepted | Optique | Illegal forms are impossible or rejected before the handler | Handler receives `foo?: boolean`, `no_foo?: boolean`, and guesses intent | +| What users see in help/completion/man pages | Optique | Parser terms expose choices, aliases, metavars, hidden compatibility, documented defaults | Help text is handwritten and drifts from parsing | +| Which values were supplied by CLI | Optique parser output | Absent source-bearing options stay `undefined` | Parser default appears as a CLI value | +| Which files and layers exist | c12 | Explicit policy for discovery, `extends`, environment branches, dotenv, RC, package metadata | App code scans config files beside c12 or relies on c12 defaults accidentally | +| How loader control fields survive loading | c12-aware merger | `extends` and environment metadata survive until c12 consumes them | Strict app schema rejects `extends` before loading finishes | +| How app values merge | Application merger, using defu only internally | Field categories are explicit and tested | Public `defu(cli, env, file)` decides arrays, unions, and nulls | +| What complete runtime values mean | Zod complete schema | Defaults and transforms apply once after all sources merge | Sparse schemas contain `.default()` and mask lower layers | +| Where diagnostics and results go | LogTape composition root | Libraries receive loggers; executable configures sinks and redaction | Libraries call `console.*` or configure LogTape globally | +| What the domain may do | Portable handler contract | Handler receives `{ config, logger, fs, fetch, signal }` or equivalent | Handler reads argv/env/config or exits the process | + +## Data shapes + +Use names that make source state obvious. + +| Shape | Example fields | Defaults? | Owner | +|---|---|---|---| +| `CliOptions` | `out_dir?: string`, `range_cache_enabled?: boolean` | No for source-bearing fields | Optique parser | +| `EnvPatch` | `logging?: { level?: string }` | No | env adapter | +| `ConfigAuthoringPatch` | ergonomic strings, `$append`, `extends`-adjacent authored data | No runtime defaults | c12 layer validation | +| `ConfigPatch` | normalized sparse ordinary data | No | application resolver | +| `AppConfig` | complete values consumed by handlers | Yes | Zod complete schema | +| `CommandResult` | stable machine output | Usually explicit defaults for arrays | Zod result schema | +| `DiagnosticEvent` | level, category, message, properties | Schema defaults only for stable event fields | LogTape formatter/sink boundary | + +Bad: + +```ts +const CliOptionsSchema = z.object({ + out_dir: z.string().default("./out"), +}); + +const parser = object({ + out_dir: option("--out-dir", string()).withDefault("./out"), +}); +``` + +This creates two defaults before source precedence is known. The CLI default +can outrank config even when the user did not pass `--out-dir`. + +Good: + +```ts +const CliPatchSchema = z.object({ + out_dir: z.string().optional(), +}); + +const AppConfigSchema = z.object({ + out_dir: z.string().trim().min(1).default("./out"), +}); + +const parser = object({ + out_dir: optional(option("--out-dir", string({ metavar: "DIRECTORY" }))), +}); +``` + +Document `./out` in help without returning it from the parser when the option is +absent. + +## Default taxonomy + +| Default kind | Example | Mechanism | Enters sparse patch? | +|---|---|---|---| +| Runtime fallback | default output directory | Zod `.default()` on complete schema | No | +| Transforming fallback | default string that must trim/case/codec | Zod `.prefault()` on complete schema | No | +| Parser-local fallback | `--color` defaults to `auto` and has no config/env source | Optique `withDefault()` | Yes, intentionally | +| Help-only default | show `default: 4` for omitted concurrency | Optique document metadata helper | No | +| Handler-time fallback | prompt for a token only when command actually runs | Optique `deferredValue()` | No scalar until handler calls it | +| Error recovery fallback | tolerate malformed optional legacy input | Zod `.catch()` with explicit policy | Not for normal config | +| Derived fallback | infer cache dir from project root | derived source below authored values | No, unless recorded as derived provenance | + +Decision rule: + +1. If the value should lose to config or env, it is not an Optique parser + default. +2. If the fallback is an ordinary literal, it belongs in the complete schema. +3. If the fallback must run a transform, use `.prefault()`. +4. If the fallback requires a prompt, secret provider, expensive lookup, or + handler context, use `deferredValue()`. +5. If the fallback should appear in help, add documentation metadata and test + that parse output stays sparse. + +## Optique patterns + +### Choices from schemas + +Good: + +```ts +const LogLevelSchema = z.enum(["debug", "info", "warning", "error"]); + +const logLevel = option( + "--log-level", + choice(LogLevelSchema.options, { suggest: "nearest" }), +); +``` + +Why: + +- Zod owns the persisted/runtime enum contract. +- Optique owns spelling suggestions, choices, help, and completion. +- The option list has one literal source. + +Bad: + +```ts +const logLevel = option("--log-level", string()); +const LogLevelSchema = z.enum(["debug", "info", "warning", "error"]); +``` + +This forces Optique to treat `--log-level debgu` as any other string until a +later schema error. Users lose parser-aware suggestions. + +### Boolean overrides + +Good: + +```ts +const patch = object({ + range_cache_enabled: optional(negatableFlag({ + positive: "--range-cache", + negative: "--no-range-cache", + })), +}); +``` + +States: + +| argv | patch | +|---|---| +| none | `{}` | +| `--range-cache` | `{ range_cache_enabled: true }` | +| `--no-range-cache` | `{ range_cache_enabled: false }` | +| both | parse error | + +The complete schema can still default `range_cache_enabled` after config/env +precedence. + +### Deferred values + +Use `deferredValue()` for a secret prompt: + +```ts +const parser = object({ + service: option("--service", choice(["github", "gitlab"] as const)), + token: deferredValue( + optional(option("--token", string({ metavar: "TOKEN" }))), + async ({ service }: { readonly service: string }) => + await promptForToken(service), + { memoize: true }, + ), +}); + +async function handler(options: ParsedOptions) { + const token = await options.token({ service: options.service }); +} +``` + +Do not use it for: + +```ts +deferredValue(optional(option("--timeout", integer())), () => 30); +``` + +That is a schema default pretending to be a handler-time fallback. + +### Help-only documented defaults + +Use a local helper when the project needs default text in help but sparse parse +output: + +```ts +const parser = object({ + timeout_seconds: optional(documentDefault( + option("--timeout", integer({ min: 1, metavar: "SECONDS" })), + 30, + )), +}); +``` + +Tests: + +| Test | Assertion | +|---|---| +| `app --help` | contains `--timeout` and `default: 30` | +| `parse([])` | `timeout_seconds` is absent or `undefined` | +| config has timeout 10 and CLI omits it | final config is 10 | +| CLI passes `--timeout 5` | final config is 5 | + +## c12 and defu patterns + +### Two mergers + +Wrong: + +```ts +await loadConfig({ + name: "app", + merger: mergeConfigInputs, +}); +``` + +Why it fails: + +- c12 control keys such as `extends` may be rejected by strict app validation. +- `$append` can be evaluated before the inherited array from an extended file is + available. +- loader metadata and application semantics become one untestable function. + +Right: + +```ts +await loadConfig({ + name: "app", + merger: mergeC12ConfigInputs, +}); + +const patch = mergeConfigInputs(cliPatch, envPatch, loaded.config); +const config = AppConfigSchema.parse(patch); +``` + +Mental model: + +```text +base config file + -> c12 loader merger keeps loader fields alive +extended config file + -> c12 returns one authored sparse object +CLI and env patches + -> app merger applies product precedence +complete schema + -> Zod applies final defaults +``` + +### Kaiju implementation anatomy + +The Kaiju implementation has several small pieces. Keep them separate when +porting the pattern. + +| Piece | File-local role | Why it exists | +|---|---|---| +| `arrayReplaceOperationSchema()` | validates `{ $replace: [...] }` envelopes | author can state replacement explicitly | +| `arrayAppendOperationSchema()` | validates `{ $append: [...] }` envelopes | author can compose above inherited arrays | +| `arrayPrependOperationSchema()` | validates `{ $prepend: [...] }` envelopes | author can place higher-layer values first | +| `arrayOperationSchema(item)` | accepts plain array or one operation envelope | one authoring language for arrays | +| `projectConfigAuthoringPatchSchema` | sparse file shape with operation-capable fields | config files are not runtime configs | +| `projectConfigLayerSchema` | authoring patch plus c12 root control keys | `extends`, `$env`, `$development`, `$production`, `$test`, `$meta` are loader-stage only | +| `projectConfigExportValueSchema` | object layer or array shorthand | root arrays mean several Kaiju patches, not c12 control layers | +| `replace()`, `append()`, `prepend()` | TypeScript helpers returning plain operation data | TS config ergonomics without runtime callbacks | +| `isArray*Operation()` guards | runtime envelope detection | defu callback sees unknown values | +| `resolveStandaloneOperations()` | recursively lowers operations with no inherited array | lowest-layer operations cannot be resolved by defu pair callbacks | +| `defuMerger` | application pair merge rule | arrays, operations, atomic unions, and copies | +| `c12DefuMerger` | loader-stage pair merge rule | preserves c12 control behavior and defers some operations | +| `mergeC12ConfigInputs()` | c12 `loadConfig({ merger })` adapter | compose c12 layers without strict app parsing | +| `mergeConfigInputs()` | public application merge | high-to-low API, low-to-high evaluation, final sparse validation | +| `normalizeLoadedConfig()` | c12 result to app sparse patch | validates root export and array shorthand | +| `sanitizeConfigInput()` | removes c12-only and undefined fields | public sparse patch must not contain loader leftovers | +| `projectConfigSchema.parse()` | complete runtime parse | final defaults and semantic refinements happen once | +| `applyConfigPatch()` | patch over an existing resolved snapshot | Optique hook path avoids reloading dynamic config | + +The important shape is not “use defu.” It is this: + +```text +authoring schemas accept ergonomic source syntax + -> c12 loader merger composes loader layers without final validation + -> normalize loaded result into a sparse app patch + -> app merger evaluates source patches by product precedence + -> standalone operations are lowered + -> sparse patch schema rejects unresolved authoring syntax + -> complete Zod schema applies runtime defaults +``` + +### Export and layer validation + +Kaiju accepts two root export shapes: + +```ts +export default { + extends: "./base.config.ts", + $development: { log: { level: "debug" } }, + sources: { + browser: { + routes: { $append: ["https://main.example/"] }, + }, + }, +}; +``` + +or: + +```ts +export default [ + { outDir: "one" }, + { log: { level: "debug" } }, +]; +``` + +The root object may contain c12 control keys. Array elements are only Kaiju +patch shorthand and intentionally cannot contain `extends`, `$env`, or named +environment branches. That prevents a root array from becoming an ambiguous +mini-loader language. + +Validation occurs in two places: + +1. retained c12 file layers are checked with the export schema so malformed file + exports fail before source adapters receive them; +2. the normalized merged result is checked with the sparse patch schema so + operation objects and loader metadata cannot reach runtime consumers. + +### Pair-merger mechanics + +Kaiju's app merger is a defu callback with this orientation: + +```text +target[key] = inherited lower-precedence value +value = incoming higher-precedence value +namespace = dotted parent path +``` + +Returning `true` means “I completed the merge for this property.” The callback +must assign the intended value before returning: + +```ts +if (Array.isArray(target[key]) && Array.isArray(value)) { + target[key] = [...value]; + return true; +} +``` + +Returning `true` without assigning leaves the lower value in place. Returning +`false` delegates to defu's ordinary scalar/object behavior. + +App merger rules: + +| Condition | Assignment | Reason | +|---|---|---| +| incoming `$replace` | copy `$replace` array | explicit replacement | +| incoming `$append` and inherited is array | inherited then appended | compose upward | +| incoming `$append` without inherited array | appended array only | standalone operation | +| incoming `$prepend` and inherited is array | prepended then inherited | priority values first | +| incoming `$prepend` without inherited array | prepended array only | standalone operation | +| incoming plain array over inherited array | copy incoming array | arrays replace, no concatenation | +| `namespace === "sources.commoncrawl"` and `key === "crawls"` | copy incoming object | discriminated union is atomic | +| otherwise | return `false` | let defu handle scalar/object merge | + +c12 merger difference: + +```ts +if ((isAppend(value) || isPrepend(value)) && !Array.isArray(inherited)) { + return false; +} +return defuMerger(target, key, value, namespace); +``` + +That `return false` is deliberate. During c12 `extends`, an append in the child +may not yet see the base array. The loader-stage merger must not collapse it too +early. The final app merge lowers any standalone operation after c12 has +finished producing the loaded file patch. + +### Public merge evaluation + +The public API is highest-to-lowest: + +```ts +mergeConfigInputs(cliPatch, envPatch, filePatch); +``` + +The loop evaluates low-to-high: + +```text +resolved = {} +merge file over {} +merge env over resolved file +merge CLI over resolved env+file +parse resolveStandaloneOperations(resolved) with ConfigPatchSchema +``` + +The reversal is necessary because operations need inherited values. A caller +should not pass layers low-to-high; the function owns that reversal. + +Example: + +```ts +mergeConfigInputs( + { sources: { browser: { routes: append(["cli"]) } } }, + { sources: { browser: { routes: prepend(["env"]) } } }, + { sources: { browser: { routes: ["file"] } } }, +); +``` + +Resolution: + +```text +file: ["file"] +env: prepend(["env"]) over ["file"] -> ["env", "file"] +CLI: append(["cli"]) over ["env","file"] -> ["env", "file", "cli"] +``` + +### Loaded config normalization + +After c12 returns, Kaiju normalizes as follows: + +| Loaded value | Normalization | +|---|---| +| `null` or `undefined` | `{}` | +| root array | parse every element as an authoring patch, then `mergeConfigInputs(...items)` | +| root object | parse as one authoring patch, then `mergeConfigInputs(object)` | + +Then `resolveConfig()` builds: + +```text +filePatch = normalizeLoadedConfig(loaded.config) +envPatch = readEnvPatch(KAIJU_*) +cliPatch = projectConfigPatchSchema.parse(options.patch ?? {}) +merged = mergeConfigInputs(cliPatch, envPatch, filePatch) +sanitized = stripUnsupportedC12Keys(stripUndefinedValues(merged)) +config = projectConfigSchema.parse(sanitized) +``` + +Only the final `projectConfigSchema.parse()` applies runtime defaults. + +### Provenance overlay + +Kaiju records leaf-level provenance after final defaults are known: + +```text +if CLI/env/file patch has the leaf path -> that layer wins +else -> schema_default +``` + +`applyConfigPatch(base, patch, "optique", overrides)` overlays provenance on an +existing resolved snapshot. It preserves earlier provenance when an Optique +source re-emits the same fallback value. It marks leaves as `cli` only when +their option aliases appeared in `argv`; otherwise the origin remains +`optique`, meaning the value came through Optique source resolution rather than +an explicit token. + +This distinction matters for `config explain`: “Optique-resolved” is not always +“typed by the user on the command line.” + +### Array operations + +Given: + +```ts +// base config +{ routes: ["file"] } + +// env patch +{ routes: { $prepend: ["env"] } } + +// CLI patch +{ routes: { $append: ["cli"] } } +``` + +Public precedence is `CLI > env > file`, but operation evaluation is low to +high: + +```text +["file"] + -> prepend ["env"] = ["env", "file"] + -> append ["cli"] = ["env", "file", "cli"] +``` + +Required tests: + +- operation with inherited array; +- operation with no inherited array; +- operation after `extends`; +- operation on operation across three layers; +- empty array replacement; +- no input aliasing; +- provenance records operation source. + +### Atomic unions + +Wrong merge: + +```text +lower: { kind: "range", from: "2024-01", to: "2024-02" } +higher: { kind: "named", crawls: ["CC-MAIN-2024-10"] } +result: { kind: "named", crawls: [...], from: "...", to: "..." } +``` + +Right rule: replace the whole value at schema-known union paths. + +Do not make every object atomic. Ordinary config objects still merge by field. +Only union-like fields and explicitly atomic fields replace as a unit. + +## LogTape patterns + +Categories are contracts: + +| Category | Sink | Parent sinks? | Payload | +|---|---|---|---| +| `app.result` | stdout raw sink | Override | exact stable result text or bytes | +| `app.diagnostic` | stderr human/json/file | Inherit diagnostic graph | structured operational events | +| `app.config` | diagnostic graph | Inherit | loader, validation, provenance events | +| `app.bootstrap` | minimal early diagnostic graph | Inherit only bootstrap sinks | failures before final config | +| `app.deprecation` | diagnostic graph | Inherit | public interface compatibility warnings | + +Rules: + +- Redact structured properties before formatting. +- Do not serialize secrets into `result_text` before redaction. +- Use a raw result formatter for JSON, JSONL, completion scripts, and man pages. +- Flush LogTape before returning exit status. +- Libraries call `getLogger()` or receive a logger; they do not configure sinks. + +Bad: + +```ts +console.error("Loading config", config); +console.log(JSON.stringify(result)); +``` + +Good: + +```ts +logger.get(["app", "config"]).debug("Config layer loaded", { + path, + source, +}); + +resultLogger.info("Command result", { + result_text: renderResult(redactStructured(result)), +}); +``` + +## Zod patterns + +Use three schemas for configuration: + +```ts +const AuthoringPatchSchema = z.object({ + timeout: z.string().optional(), + routes: RouteOperationSchema.optional(), +}); + +const ConfigPatchSchema = z.object({ + timeout_seconds: z.number().int().positive().optional(), + routes: z.array(z.url()).optional(), +}); + +const AppConfigSchema = z.object({ + timeout_seconds: z.number().int().positive().default(30), + routes: z.array(z.url()).default([]), +}); +``` + +Use result schemas too: + +```ts +const CommandResultSchema = z.object({ + schema_version: z.literal("1"), + status: z.enum(["ok", "failed"]), + artifacts: z.array(z.object({ + path: z.string(), + kind: z.string(), + })).default([]), +}); +``` + +Do not trust TypeScript interfaces at boundaries where JSON, files, subprocess +output, logs, or persisted artifacts are involved. + +## One-snapshot execution recipe + +Implement this when a command needs config-backed parser sources and a handler +needs final config: + +1. Parse only early controls from raw argv: help/version/completion, config + path, logging format/output/silent, and no-config. +2. Configure a minimal LogTape bootstrap graph if a fallible operation comes + next. +3. Load c12 once with a c12-aware merger and explicit source policy. +4. Validate retained authored layers. +5. Build Optique source contexts from that loaded snapshot and injected env. +6. Parse the full Optique program into sparse patches. +7. In `runProgram().hooks.beforeEach`, apply the parsed patch over the loaded + snapshot using the strict app merger. +8. Parse the complete Zod runtime schema once. +9. Configure final LogTape sinks, filters, result route, and redaction. +10. Return `{ config, logger }` as the hook resource. +11. Handler receives parsed command value plus the resource; it does not reload + config, read env, configure logging, or call `Deno.exit`. +12. `afterEach`/`onError` flushes and disposes resources. + +Minimal shape: + +```ts +interface AppResource { + readonly config: AppConfig; + readonly logger: Logger; +} + +await runProgram({ + commands, + metadata, + hooks: { + async beforeEach(invocation) { + const cliPatch = CliPatchSchema.parse(invocation.value); + const config = applyConfigPatch(loaded.config, cliPatch); + const logger = await configureAppLogging(config.logging); + return { resource: { config, logger } }; + }, + async afterEach(context) { + await context.resource?.logger.flush(); + }, + async onError(context, error) { + context.resource?.logger.error("Command failed", { error }); + await context.resource?.logger.flush(); + }, + }, +}); +``` + +## End-to-end trace example + +Trace `--timeout`: + +| Stage | Value | Owner | +|---|---|---| +| CLI absent | no patch field | Optique | +| `APP_TIMEOUT=10` | env patch `timeout_seconds: 10` | env adapter | +| config `timeout: "PT20S"` | file patch `timeout_seconds: 20` | c12 + authoring schema | +| derived default | not used because env/config exist | derived source | +| runtime default `30` | not used because sparse patch has value | Zod complete schema | +| final config | `10` if env outranks config | app merger | +| HTTP timeout | ofetch receives 10 seconds as app policy | HTTP adapter | +| provenance | env winner, config shadowed, default shadowed | resolver | +| help | shows `--timeout SECONDS`, maybe `default: 30` | Optique docs | +| tests | subprocess proves stdout/stderr/exit and final request | release gate | + +If the CLI parser default returns `30`, env and config never get a fair chance. + +## Audit questions + +Ask these before editing: + +1. Which fields are source-bearing and must stay sparse? +2. Which fields are parser-local and may use `withDefault()`? +3. Which defaults must pass through Zod transforms and therefore need + `.prefault()`? +4. Which c12 sources are enabled, and is that product policy documented? +5. Does c12 receive a loader-aware merger rather than the strict app merger? +6. Which arrays replace, which compose, and which deduplicate? +7. Which object paths are atomic unions? +8. Does any dynamic config factory run more than once? +9. Can help/version/completion run when project config is broken? +10. Does LogTape have a separate raw result route and diagnostic route? +11. Are secrets redacted before every formatter and sink? +12. Does the handler receive all capabilities by injection? +13. Are generated completion and man surfaces produced from the parser? +14. Has the installed or compiled artifact been executed? + +## Required test matrix + +| Area | Cases | +|---|---| +| Parser absence | option omitted, config present; parser output stays sparse | +| Help defaults | help/man show documented default; parse omitted remains absent | +| Choices | valid choice, typo suggestion, completion list | +| Negatable flag | absent, true, false, conflict | +| Deferred value | specified branch, fallback branch, memoization, fallback error | +| c12 discovery | explicit path, missing explicit path, no-config, malformed config | +| c12 extends | base file, child override, child operation over base array | +| Environment branch | branch selected, branch disabled, branch precedence | +| Merge algebra | falsy values, plain arrays, operations, atomic unions, immutability | +| Defaults | config over default, env over config, CLI over env, final Zod default | +| Single snapshot | dynamic factory counter equals one | +| LogTape result | JSON stdout has no diagnostics; diagnostics go to stderr/file | +| Redaction | nested secret in config, URL, header, error object, result view | +| Bootstrap failure | invalid config honors raw logging options and leaves stdout empty | +| Generated surfaces | help, completion shells, man pages, hidden aliases | +| Package artifact | installed/compiled binary reaches each command | + +## Kaiju test evidence map + +Use this as the minimum shape of proof for another repository. Rename the tests +to the local project, but keep the behavioral coverage. + +| Invariant | Kaiju-style test evidence | +|---|---| +| Leftmost input is highest precedence | `mergeConfigInputs({ outDir: "cli" }, { outDir: "env" }, { outDir: "file" })` returns `"cli"` | +| Missing higher layers inherit lower values | empty CLI/env patches preserve file patch | +| Null/undefined layers are skipped | `undefined`, `null`, and real layers produce only real contributions | +| `false` and `0` are explicit values | `emitStages: false`, `retries: 0`, `cdxMinDelayMs: 0` survive | +| Invalid sparse output still fails | empty strings or invalid worker concurrency throw after merging | +| Ordinary objects merge by property | `log.level` from CLI plus `log.format` from file both survive | +| Plain arrays replace | higher `mime: ["text/plain"]` does not concatenate inherited MIME filters | +| Empty arrays clear inherited values | higher `affiliatedHosts: []` resolves to `[]` | +| `$replace` discards inherited array | replacement routes remove file routes | +| `$append` composes after inherited array | inherited route then appended route | +| `$prepend` composes before inherited array | prepended route then inherited route | +| Standalone operations lower to arrays | single-layer `append(["x"])` returns `["x"]`, not `{ $append: ... }` | +| Duplicates are preserved | append duplicate route keeps both entries | +| Higher plain array beats lower operation | CLI array replaces env append plus file array | +| Higher replace beats lower operation | CLI replace discards env append plus file array | +| Common Crawl selector is atomic | named selector replaces range selector and drops `from`, `to`, `limit` | +| Only schema-known union path is atomic | unrelated property named `crawls` still recursively merges | +| Inputs are not mutated | snapshots before/after merge remain equal | +| Resolved arrays are copied | output array is not the same reference as incoming or inherited array | +| c12 `extends` resolves before defaults | base `outDir` and route plus child log level and `$append` all survive | +| c12 environment branch applies when selected | `$development` can override log settings when `envName` is explicit | +| c12 control keys are forbidden in array shorthand | root array element with `extends` is rejected | +| malformed exports fail early | string export throws schema-valid object/array/function error | +| JSONC parser preserves `//` inside strings | route with `/a//b` survives JSONC loading | +| defaults happen after sparse resolution | `projectConfigSchema.parse()` supplies defaults only after merge | +| `applyConfigPatch()` preserves snapshot provenance | re-emitted Optique fallback does not steal env provenance | +| dynamic config factory runs once | temp config increments a marker file and command sees count `1` | +| help-only defaults do not parse | `documentDefault()` docs include default while `parse([])` returns `undefined` | +| Zod enum choices power suggestions | typos such as `standart`, `domian`, `reachabl` suggest nearest valid choice | +| negatable flag is tri-state | absent, `--range-cache`, `--no-range-cache`, and conflict are tested | +| LogTape category routing is structured | recorder sink sees category `["kaiju","test"]` and structured properties | +| diagnostic redaction wraps sinks | password field becomes `[REDACTED]` in diagnostic file | +| result route is raw stdout | `kaiju.result` has `parentSinks: "override"` and writes `result` string to stdout | + +When reporting verification, separate: + +- focused unit tests for parser and merger rules; +- c12 fixture tests for real loading behavior; +- runtime CLI tests with `--no-check` when the checked graph is resource-heavy; +- generated surface tests for help, completion, and man output; +- broad type-check or workspace gates that were skipped, blocked, or OOM-limited. + +## Failure signatures + +| Symptom | Likely boundary bug | +|---|---| +| Config value ignored when CLI flag omitted | Optique default became a sparse CLI value | +| `extends` is an unknown key | c12 received strict app validation too early | +| `$append` appends to an operation object | Operation evaluated before inherited c12 layers resolved | +| Zod default appears in provenance as user-authored | Complete schema parsed a sparse layer | +| Help lists choices but completion does not | Choices are handwritten in docs rather than parser terms | +| Handler sees both `cache` and `no_cache` | Boolean pair was not modelled with `negatableFlag()` | +| Prompt appears during `--help` or CI | Prompt adapter owns policy instead of executable boundary | +| Dynamic config increments twice | Source context and handler both call resolver | +| JSON output has warning text before it | Result and diagnostic LogTape routes are not isolated | +| Redaction misses `config show --json` | Secrets were stringified before structured redaction | +| Command exists in source but not binary | Dynamic discovery was invisible to bundler/compiler | + +## Sources and freshness + +- Normative source: `productionized-cli-pattern-guidebook-v1.2.md`, reviewed 2026-07-22. +- Observed implementation: current Kaiju CLI Optique 1.2.0, c12 two-merger, LogTape resource-threading, and Zod default audit. +- Official Optique 1.2 API docs: and . +- Official Zod v4 docs: . +- Official c12 source: . +- Official defu source: . +- Official LogTape docs: . + +Freshness status: verified against current Kaiju work and Optique 1.2 API docs +on 2026-07-22. Re-check exact package exports and installed lockfile versions +before copying code into a different repository. diff --git a/skills/build-clis/references/integration.md b/skills/build-clis/references/integration.md new file mode 100644 index 0000000..1cffef1 --- /dev/null +++ b/skills/build-clis/references/integration.md @@ -0,0 +1,364 @@ +# CLI integration sequences + +## Contents + +- [Composition invariants](#composition-invariants) +- [Five-library execution graph](#five-library-execution-graph) +- [Normal command execution](#normal-command-execution) +- [Bootstrap failure](#bootstrap-failure) +- [Configuration explanation](#configuration-explanation) +- [Machine-readable streaming result](#machine-readable-streaming-result) +- [Interactive configuration initialization](#interactive-configuration-initialization) +- [Long-running recoverable command](#long-running-recoverable-command) +- [Durable workflow client](#durable-workflow-client) +- [Project mutation command](#project-mutation-command) +- [Completion and manual generation](#completion-and-manual-generation) +- [Release sequence](#release-sequence) +- [Cross-cutting verification](#cross-cutting-verification) +- [Sources and freshness](#sources-and-freshness) + +## Composition invariants + +Keep these invariants across every sequence: + +1. parse early controls before loading fallible domain configuration; +2. evaluate each external source once; +3. preserve sparse values until precedence resolution finishes; +4. validate the complete request before harmful work; +5. inject capabilities and one abort signal into portable handlers; +6. transport stable results and diagnostics through separate LogTape routes when + LogTape is the selected owner; +7. persist durable artifacts through typed writers/stores, not log sinks; +8. render one public failure at one boundary; +9. flush/close owned resources before returning an exit status; +10. verify the exact installed invocation path. + +## Five-library execution graph + +When Optique, c12, defu, LogTape, and Zod are all present, the intended graph is: + +```text +Optique + owns token grammar, help, suggestions, completion, man pages, and sparse CLI values + | + v +c12 loader with c12-aware defu merger + owns config discovery, formats, extends, environment branches, and factories + | + v +application resolver with strict merge policy + owns precedence, arrays, operations, atomic unions, and sparse patch validation + | + v +Zod runtime schema + applies defaults and transformations once + | + v +Optique runProgram hook resource + threads the single resolved config snapshot and logger into the handler + | + v +LogTape + routes redacted results, diagnostics, bootstrap failures, and lifecycle flush +``` + +Do not collapse c12's merger and the application merger. Do not let Optique +materialize defaults into sparse source patches. Do not let the handler reload +configuration or configure LogTape. + +## Normal command execution + +```text +raw argv and process environment + -> parse help/version/completion and logging controls + -> configure bootstrap diagnostics + -> first-pass parse selects command and config path + -> c12 loads/evaluates file layers once with a loader-aware merger + -> Optique source contexts bind environment, config, and derived values + -> sparse CLI/environment/config patches resolve by explicit app precedence + -> runtime schema applies defaults and semantic checks + -> runProgram beforeEach creates `{ config, logger }` from that snapshot + -> composition root creates capabilities and AbortController + -> portable handler executes validated request + -> result schema validates outcome + -> renderer produces human/plain/JSON/JSONL representation + -> LogTape result category writes exact stdout bytes + -> diagnostics/resources flush and close + -> exit 0 +``` + +Reject a design in which the handler reloads config, reads globals, configures +logging, chooses output encoding, or exits the process. + +Prove this sequence with one command whose values conflict across CLI, +environment, and config. Assert the final request and provenance, not only the +human output. + +## Bootstrap failure + +Configuration and parser setup can fail before the main logger exists: + +```text +raw argv + -> recover only --log-level/--log-format/--log-output/--silent + -> configure minimal redacted LogTape graph + -> attempt c12 loading + -> validate retained config layer + -> one public configuration failure + -> flush selected diagnostic sink + -> leave stdout empty + -> exit configuration class +``` + +The bootstrap parser must not duplicate domain options. If an early control is +malformed, use a safe stderr fallback owned by the executable boundary. + +Required cases: + +- invalid TypeScript/JavaScript config; +- missing explicit config file; +- malformed scalar export; +- schema-invalid nested field; +- invalid `extends` layer; +- selected JSON diagnostic file; +- `--silent` behavior; +- help and version despite broken project config. + +## Configuration explanation + +`config explain` is a query over the resolver's recorded decision data: + +```text +same source load and merge used by execution + -> retain ordered layers and per-field contributions + -> retain append/prepend/replace operations + -> apply secret redaction to structured records + -> select one field or complete safe view + -> render winner, shadowed values, defaults, and source details + -> emit stable result through result category +``` + +Do not reconstruct provenance by reloading configuration or comparing only the +final object. Do not show a hardcoded precedence list as though it explains a +field. + +Machine output should use a versioned schema: + +```ts +const ExplainResultSchema = z.object({ + schema_version: z.literal("1"), + path: z.string(), + value: z.unknown(), + winner: z.object({ source: z.string(), detail: z.string().optional() }), + shadowed: z.array(z.object({ source: z.string(), detail: z.string().optional() })), + operations: z.array(z.object({ source: z.string(), operation: z.string() })), +}); +``` + +## Machine-readable streaming result + +For a command that streams records: + +```text +handler emits typed result records + -> validate/encode one record at a time + -> structured redaction sees the record before serialization + -> JSONL renderer emits exactly one value and newline + -> LogTape result sink writes raw bytes to stdout + -> operational progress goes only to stderr/file categories + -> backpressure is awaited at the result writer boundary +``` + +Do not accumulate an unbounded array for final `JSON.stringify()`. Do not log the +whole record as an opaque string before redaction. Do not let progress spinners +or warnings enter stdout. + +Define partial-failure policy: + +- fail-fast before any result; +- emit per-record success/error envelopes; +- write an external artifact and return a summary; +- or mark the stream incomplete in a final versioned record. + +The selected policy is part of the machine contract. Exit status alone cannot +undo records already consumed from stdout. + +## Interactive configuration initialization + +```text +parse explicit flags and --no-input + -> identify missing values + -> if non-interactive, fail with exact required flags + -> if interactive, Optique prompt adapter calls Clack/Inquirer renderer + -> build authored config patch + -> resolve owned target path through c12/path policy + -> magicast/confbox prepares source-preserving change + -> show plan/diff + -> require authorization appropriate to risk + -> atomic write + -> reload through public c12 resolver + -> validate authored, patch, and runtime schemas + -> stable result identifies changed file and next command +``` + +Never prompt merely because a value is absent. Honor TTY, CI, output mode, +redirection, cancellation, and no-input policy. Never print secret answers. + +## Long-running recoverable command + +A local long-running command needs more than retries: + +```text +validated request + -> canonical request/input fingerprint + -> acquire run identity and checkpoint owner + -> reconcile prior checkpoint and external artifacts + -> process idempotent unit + -> commit durable output + -> atomically record committed checkpoint + -> emit structured progress + -> repeat until complete or cancelled +``` + +On first SIGINT: + +```text +acknowledge immediately + -> root AbortController aborts new/active work + -> current unit resolves according to atomicity policy + -> committed checkpoint is persisted + -> subprocesses/workers/browser resources close within deadline + -> LogTape flushes + -> exit 130 where the host supports SIGINT status +``` + +On second SIGINT, follow the declared force policy. Do not claim recovery if a +crash between output commit and checkpoint write can duplicate or lose effects +without reconciliation. + +Compose with `build-workflows` when leases, durable timers, replay, cross-process +signals, or multi-worker recovery are material. + +## Durable workflow client + +Keep the CLI as a client over Temporal, `@effect/workflow`, or another durable +engine: + +```text +Optique parses start/status/signal/cancel/result command + -> c12 resolves endpoint/namespace/task-queue policy + -> credentials come from a safe SecretSource + -> client adapter sends a request with workflow/run ID + -> durable engine owns history, retries, timers, signals, and workers + -> CLI optionally follows status or returns immediately + -> stable result returns workflow/run identity and next commands +``` + +Parser packages such as `@optique/temporal` parse Temporal values. They do not +make an in-process operation durable. + +Define separate commands rather than a hidden conversational state: + +```text +app run start ... +app run inspect +app run follow +app run signal +app run cancel +app run result +``` + +Handle Ctrl-C while following as “detach” unless the public command explicitly +and safely maps it to workflow cancellation. Never cancel durable work merely +because the client terminal disconnected. + +## Project mutation command + +For dependency/config/code generation: + +```text +discover project and workspace ownership + -> classify package manager/runtime/framework + -> build typed operation plan and preconditions + -> render dry run + -> authorize apply + -> stage writes in owned temporary paths + -> apply files atomically where possible + -> invoke package manager/build generator with root signal + -> verify manifests, lockfiles, generated output, and clean consumer + -> emit changed-file result and rollback/recovery guidance +``` + +Use pkg-types/nypm/pathe or runtime-native equivalents behind adapters. Preserve +dirty user changes. Do not rewrite unrelated Markdown or configuration. + +## Completion and manual generation + +```text +statically registered command model + -> Optique program parser + -> help document + -> shell-specific completion generator + -> @optique/man roff generator + -> deterministic artifacts + -> shell/roff syntax checks + -> drift comparison in release gate +``` + +Do not require project config, network, credentials, or a repository to generate +basic completion/manual output. Dynamic completion providers must be bounded, +cached, and optional. + +Verify every claimed shell. Verify option aliases, choices, hidden terms, +deprecated terms, and subcommands. An artifact existing in `dist/` is not proof +it matches the active parser. + +## Release sequence + +```text +typecheck/lint/test source + -> generate static registry/completion/man/docs regions + -> fail on drift + -> build package or standalone executable + -> inspect package contents and embedded assets + -> install in clean environment + -> run version/help/config/representative success/failure/cancel cases + -> verify checksums/signing/provenance where claimed + -> verify install, upgrade, and uninstall instructions +``` + +Keep Deno compile, Node SEA, registry packages, and OS packages as different +artifact contracts. Test every target architecture claimed or narrow the claim. + +## Cross-cutting verification + +For each integration sequence, record: + +| Dimension | Evidence | +|---|---| +| Reachability | Active entrypoint imports/registers the command | +| Source ownership | One resolver returns values and provenance | +| Validation | Sparse and complete schemas exercise real invalid cases | +| Output | Exact stdout/stderr/file records and redaction | +| Failure | One diagnostic and stable exit class | +| Cancellation | Concrete process signal reaches active dependency | +| Cleanup | Files, workers, subprocesses, clients, and sinks close | +| Recovery | Restart/reconcile behavior executes rather than being described | +| Packaging | Installed artifact runs without source-tree fallbacks | +| Documentation | Help, manual, completion, and examples agree | + +Report each surface as intended, documented, implemented, or executable-verified. +Keep blocked checks distinct from failed checks. + +## Sources and freshness + +- Normative architecture: `productionized-cli-pattern-guidebook-v1.1(1).md`, reviewed 2026-07-17. +- Normative CLI audit: `cli-guidelines-audit-and-expansion(1).md`, reviewed 2026-07-17. +- Normative config merge contract: `kaiju-config-resolution-handoff(2).md`, reviewed 2026-07-17. +- Observed implementation and counterexamples: `live-browser-cli(41).zip`, reviewed 2026-07-17. +- Official project pointers: , , and . + +Freshness status: sequences describe ownership and verification invariants, not +proof that the attached CLI implements every stage. Re-run the source trace and +installed-artifact checks for the current repository revision and dependency +versions before making a completion claim. diff --git a/skills/build-clis/references/logtape.md b/skills/build-clis/references/logtape.md new file mode 100644 index 0000000..1728e82 --- /dev/null +++ b/skills/build-clis/references/logtape.md @@ -0,0 +1,427 @@ +# LogTape output and diagnostics manual + +## Contents + +- [Evidence and ownership](#evidence-and-ownership) +- [Package capability map](#package-capability-map) +- [Category architecture](#category-architecture) +- [Stable result isolation](#stable-result-isolation) +- [Diagnostic routing](#diagnostic-routing) +- [Structured events and context](#structured-events-and-context) +- [Filters, levels, and output policy](#filters-levels-and-output-policy) +- [Redaction before rendering](#redaction-before-rendering) +- [Bootstrap and reconfiguration](#bootstrap-and-reconfiguration) +- [Lifecycle and disposal](#lifecycle-and-disposal) +- [Library and application boundaries](#library-and-application-boundaries) +- [Testing](#testing) +- [Exclusions](#exclusions) +- [Failure signatures](#failure-signatures) +- [Sources and freshness](#sources-and-freshness) + +## Evidence and ownership + +Treat the productionized CLI guidebook as the normative architecture. Treat the +attached Kaiju diagnostics package as an observed LogTape 2.2.4 implementation, +not a complete or universally correct reference. + +Before adopting this design, establish the existing output owner. If the +repository does not use LogTape and the request does not authorize a migration, +preserve its verified transport and apply only compatible channel principles. + +If LogTape is the owner, route every observable process result and diagnostic +through it. Do not retain `console.log` for results or add a second reporter for +friendly progress. Durable application artifacts remain outside the log graph. + +## Package capability map + +| Capability | Package | Use | +|---|---|---| +| Categories, records, filters, formatters, sinks, context | `@logtape/logtape` | Required transport core | +| Human terminal diagnostics | `@logtape/pretty` | Pretty formatter after stream/color policy | +| File and rotating diagnostics | `@logtape/file` | Durable support logs and configured files | +| Structured secret protection | `@logtape/redaction` | Wrap every result and diagnostic route before formatting | +| Recorder-backed assertions | `@logtape/testing` | Categories, levels, messages, context, and properties | +| Parser-owned verbosity and destinations | `@optique/logtape` | CLI terms only; does not configure the graph | +| Static rules | `@logtape/lint` | Adopt only when runtime/linter integration maturity fits | +| OpenTelemetry | `@logtape/otel` | Explicitly selected remote telemetry route | +| Other remote/system sinks | Sentry, syslog, CloudWatch, Windows Event Log integrations | Deployment-specific and consent/policy gated | + +Inspect the installed version and package exports. Do not add: + +- `@logtape/config` when c12 and the application schema already own config; +- a database adapter that does not support the actual dialect or custom adapter; +- every remote sink merely because the ecosystem provides one; +- LogTape packages to manifests that do not import them directly. + +## Category architecture + +Model categories as ownership and routing namespaces: + +```text +app.result stable general result +app.clickhouse.result stable ClickHouse-specific result, if separately owned +app.cli command lifecycle diagnostics +app.config config discovery and resolution +app.http request/retry diagnostics +app.workflow durable-work client diagnostics +app.deprecation compatibility warnings +app.progress structured progress events +logtape.meta transport configuration failures +``` + +Use category hierarchy to share policy. Do not encode the entire event type in a +free-form message string. Bind stable dimensions such as command, run ID, target, +and request ID as context or structured properties. + +Define category constants so result routing cannot drift: + +```ts +export const APP_CATEGORY = ["app"] as const; +export const RESULT_CATEGORY = [...APP_CATEGORY, "result"] as const; +``` + +## Stable result isolation + +A stable result is user-requested output. It may be JSON, JSONL, a completion +script, a man page, a generated config, or plain text. It must remain pipe-safe. + +Configure a dedicated raw sink and block inherited diagnostic sinks: + +```ts +await configure({ + sinks: { + result: redactByField(createResultSink(), redactionOptions), + diagnostic: redactByField(createDiagnosticSink(options), redactionOptions), + }, + loggers: [ + { + category: [...RESULT_CATEGORY], + lowestLevel: "info", + sinks: ["result"], + parentSinks: "override", + }, + { + category: [...APP_CATEGORY], + lowestLevel: options.level, + sinks: ["diagnostic"], + }, + ], +}); +``` + +`parentSinks: "override"` is the important isolation contract in the observed +LogTape version. Verify the option name and behavior in the installed version. + +Carry the raw rendered text as a structured property: + +```ts +export function emitResult(text: string): void { + getLogger([...RESULT_CATEGORY]).info("{result}", { + result: text.endsWith("\n") ? text : `${text}\n`, + }); +} +``` + +The raw sink reads only the expected property: + +```ts +function createResultSink(): Sink { + const encoder = new TextEncoder(); + return (record): void => { + const result = record.properties.result; + if (typeof result !== "string") { + throw new TypeError("Result records require a string result property."); + } + Deno.stdout.writeSync(encoder.encode(result)); + }; +} +``` + +Do not let the result sink add timestamps, levels, category names, colors, or a +second newline. Define exact newline and empty-result policy. For JSONL, emit one +complete JSON value per record and reject embedded raw newlines where the schema +forbids them. + +## Diagnostic routing + +Diagnostics describe execution rather than return the requested value. Route +them to stderr, a selected file, or explicitly configured remote sinks. + +Select formatters by mode: + +```ts +function diagnosticFormatter(options: LogOptions): TextFormatter { + if (options.format === "json") return jsonLinesFormatter; + if (options.format === "plain") return defaultTextFormatter; + return getPrettyFormatter({ + timestamp: "time", + colors: options.colors, + properties: true, + wordWrap: options.width, + }); +} +``` + +Resolve terminal width and color for stderr independently from stdout. Never put +ANSI styling in machine results. `NO_COLOR`, forced color, explicit flags, CI, +TTY status, and stream capability need one precedence policy. + +Use a file sink when diagnostics must survive the process or support a bug +bundle. Define rotation, retention, permissions, path ownership, flushing, and +failure policy. A failed optional diagnostic file should not silently destroy a +successful domain result; the product must decide whether it degrades, warns, or +fails before starting work. + +## Structured events and context + +Keep message templates stable and properties queryable: + +```ts +logger.info("migration generated", { + migration_id: migration.id, + statement_count: migration.statements.length, + output_path: migration.path, +}); +``` + +Avoid pre-rendered interpolation when the values matter: + +```ts +logger.info(`Generated ${migration.id} with ${migration.statements.length}`); +``` + +The structured form preserves filtering, JSON encoding, redaction, recorder +tests, support-bundle extraction, and future telemetry. + +Bind stable context once per operation: + +```text +run_id +command +target +request_id +plan_id +attempt +``` + +Use lazy properties for expensive debug-only calculations. Do not hash files, +serialize large graphs, or inspect the filesystem before knowing a sink will +receive the debug record. + +Log a cause as structured, redacted data. Public error rendering remains owned +by the executable boundary. Do not emit the same failure in a handler and again +at the entrypoint. + +## Filters, levels, and output policy + +Keep result mode separate from diagnostic verbosity: + +```ts +const OutputPolicySchema = z.object({ + mode: z.enum(["human", "plain", "json", "jsonl"]), + quiet: z.boolean().default(false), + silent: z.boolean().default(false), + color: z.enum(["auto", "always", "never"]).default("auto"), +}); +``` + +Suggested semantics: + +- `--quiet` suppresses routine human diagnostics while preserving warnings, + errors, and requested results; +- `--silent` suppresses diagnostics but preserves requested results unless the + public contract explicitly says otherwise; +- repeated `-v` raises diagnostic detail but does not change result encoding; +- a configured level and repeated verbosity need an explicit algebra; +- subsystem filters may raise or lower a category without changing siblings. + +The observed Kaiju adapter treats repeated verbosity as an override only when it +raises detail above its baseline. That is one product policy, not a LogTape rule. +Document and test the chosen behavior. + +Progress is a structured event stream. A TTY renderer may turn it into a spinner +or live region; a JSON diagnostic sink may retain state changes; normal stderr +may show only acknowledgement and milestones. Do not add Clack or Consola as an +untracked second event transport. + +## Redaction before rendering + +Wrap every sink before any formatter serializes the record: + +```ts +const sink = redactByField(createDiagnosticSink(options), { + fieldPatterns: [ + ...DEFAULT_REDACT_FIELDS, + /^authorization$/iu, + /^cookie$/iu, + /^set-cookie$/iu, + ], + action: () => "[REDACTED]", +}); +``` + +Redaction after `JSON.stringify()` cannot see nested field names. Never convert +a config, headers object, result, or error cause into one opaque string before +redaction. + +Cover more than field names: + +- passwords, API keys, authorization, cookies, and tokens; +- secrets embedded in URLs and connection strings; +- arrays and nested records; +- errors and causes; +- argument/config snapshots; +- result output such as `config show`; +- bootstrap failures; +- support bundles and remote routes. + +Use value-pattern redaction or keyed pseudonymization when field names are +insufficient. Pseudonyms must use protected key material and must not make the +original recoverable. + +Redaction is defense in depth. Continue to reject secrets in argv and general +diagnostic environment dumps. + +## Bootstrap and reconfiguration + +Configuration can fail before the full CLI is parsed. Implement a deliberately +small early-control pass: + +```text +raw args and environment + -> logging-only controls + -> bootstrap LogTape configuration + -> c12/source loading + -> full Optique parse + -> validated final logging policy + -> reconfigure once + -> command execution +``` + +The early pass may recover log format, destination, level, and silent policy. It +must not duplicate domain grammar. Test malformed configuration with each early +logging option and verify stdout remains empty. + +If reconfiguration calls `reset()`, ensure no active command logs between reset +and final configuration. Cache an exact normalized signature only if repeated +configuration is semantically idempotent. Tests must reset process-global state +between cases. + +## Lifecycle and disposal + +Configure LogTape at the composition root. Before exit: + +1. stop new work; +2. abort or complete active operations; +3. close owned domain resources; +4. flush asynchronous sinks; +5. dispose file and remote transport resources; +6. reset process-global logging state for embedded/test contexts; +7. return the stable exit status. + +Do not close the host's stdout or stderr stream. The adapter owns wrappers, not +process-global streams. + +Exercise success, domain failure, parse failure, config failure, cancellation, +and second-interrupt paths. A direct `Deno.exit()` or `process.exit()` before +awaited cleanup can lose file or remote records. + +## Library and application boundaries + +Expose a small structural logger contract to reusable packages: + +```ts +export interface DiagnosticLogger { + debug(message: string, properties?: Readonly>): void; + info(message: string, properties?: Readonly>): void; + warn(message: string, properties?: Readonly>): void; + error(message: string, properties?: Readonly>): void; +} +``` + +Libraries record events but never choose sinks, files, remote endpoints, global +levels, or redaction policy. Without application configuration, library logging +must remain safe and silent according to LogTape's library-first model. + +Keep durable JSONL stage writers, databases, checkpoints, and exports as their +own typed capabilities. They may also emit diagnostic records, but LogTape must +not become their persistence protocol. + +## Testing + +Use `@logtape/testing` recorder sinks for structured assertions and subprocess +tests for byte-level process contracts. + +Test: + +- category, level, message template, properties, and bound context; +- result category does not reach parent diagnostic sinks; +- diagnostic category never reaches stdout; +- exact result bytes and newline behavior; +- pretty/plain/JSON diagnostic shape; +- quiet, silent, repeated verbosity, and subsystem levels; +- nested redaction in results and every diagnostic sink; +- bootstrap config failure routing; +- selected file destination and file flush; +- one public failure record, not duplicates; +- process cancellation and sink cleanup; +- reset isolation between tests; +- remote sink absence until consent/policy is present. + +Example route assertions: + +```text +command success with --json + stdout: exactly one JSON document + stderr: empty at normal level + file: no result record + +invalid config with --log-format json --log-output errors.jsonl + stdout: empty + stderr: empty when file exclusively owns diagnostics + file: one redacted JSONL failure record + exit: configuration class +``` + +## Exclusions + +- Do not add Consola beside LogTape for friendly output. It is an alternative + terminal logger/reporter and would split routing and testing ownership. +- Do not send pager control sequences through a sink. A pager consumes a + completed human result document after result rendering policy selects it. +- Do not use remote telemetry without consent or explicit organization policy. +- Do not let telemetry failure block the command unless the product explicitly + requires audit delivery. +- Do not put `console.*` fallbacks inside handlers. If bootstrap transport can + fail, define one minimal emergency boundary at the executable root. +- Do not treat LogTape as a workflow history, queue, database, or artifact store. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| JSON contains timestamps/category prefixes | Result inherited diagnostic formatter | Check result category and `parentSinks` | +| Result appears twice | Result reaches result and parent sinks | Check category hierarchy and override | +| Secret survives inside a JSON string | Serialization happened before redaction | Keep object structured until sink wrapper | +| `--silent` removes requested JSON | Result and diagnostics share one filter | Separate result category from level policy | +| Invalid config ignores `--log-output` | Logger configured only after config resolution | Add logging-only bootstrap pass | +| Same error appears twice | Handler and boundary both render/log | Give public error output one owner | +| File log misses final records | Process exits before flush/disposal | Trace awaited lifecycle boundary | +| Test records leak between cases | Process-global LogTape state not reset | Reset in test teardown | +| Debug logging is expensive when hidden | Properties computed eagerly | Use lazy evaluation and category filters | +| Pretty output corrupts a pipe | TTY/color resolved globally, not per stream | Inspect stdout/stderr policy separately | +| Durable stage data appears as log messages | Artifact persistence was collapsed into diagnostics | Restore typed writer/store ownership | +| Remote data leaves process unexpectedly | Sink enabled without consent/policy | Trace telemetry configuration and defaults | + +## Sources and freshness + +- Normative source: `productionized-cli-pattern-guidebook-v1.1(1).md`, reviewed 2026-07-17. +- Normative ecosystem audit: `cli-guidelines-audit-and-expansion(1).md`, reviewed 2026-07-17. +- Observed implementation: `live-browser-cli(41).zip/packages/diagnostics/src/logging.ts` and its tests/manifests, reviewed 2026-07-17. +- Official documentation: , discovery pointer for current categories, sinks, formatters, filters, redaction, and integrations. +- Official source: , discovery pointer for package/version history. + +Freshness status: the concrete configuration example is grounded in LogTape +2.2.4 from the attached lockfile. Verify `parentSinks`, sink wrapper signatures, +formatter APIs, lint maturity, and optional package names against the installed +version before copying code. diff --git a/skills/build-clis/references/optique.md b/skills/build-clis/references/optique.md new file mode 100644 index 0000000..766ea41 --- /dev/null +++ b/skills/build-clis/references/optique.md @@ -0,0 +1,509 @@ +# Optique command-system manual + +## Contents + +- [Evidence and version discipline](#evidence-and-version-discipline) +- [Package capability map](#package-capability-map) +- [Parser architecture](#parser-architecture) +- [Typed grammar](#typed-grammar) +- [Sparse sources and precedence](#sparse-sources-and-precedence) +- [Schemas and validation](#schemas-and-validation) +- [Optique 1.2 features](#optique-12-features) +- [Discovery, running, and packaging](#discovery-running-and-packaging) +- [Help, completion, and manuals](#help-completion-and-manuals) +- [Prompts and interaction adapters](#prompts-and-interaction-adapters) +- [Logging, time, and Git integrations](#logging-time-and-git-integrations) +- [Error ownership](#error-ownership) +- [Testing and verification](#testing-and-verification) +- [Failure signatures](#failure-signatures) +- [Sources and freshness](#sources-and-freshness) + +## Evidence and version discipline + +Treat the productionized CLI guidebook as normative architecture and the +attached Kaiju CLI as observed implementation. Do not claim an Optique feature +works merely because the package appears in a manifest or guide. + +Older attached implementations mixed stable Optique 1.1.1 packages with +`1.2.0-dev.2329+7836254a` packages. Current Kaiju work has moved the active CLI +line to Optique 1.2.0. Before editing any repository, still verify the installed +line rather than assuming a guidebook or previous lockfile is current: + +1. inventory the root manifest, CLI manifest, import map, and lockfile; +2. group every `@optique/*` package by the exact resolved version; +3. inspect the package export map or type declarations for the installed build; +4. keep core, runner, discovery, and integration packages on compatible lines; +5. run the packaged executable because a type-compatible dev build can still + fail during bundling, discovery, or completion generation. + +Do not normalize a prerelease version to a caret range or stable line without a +documented migration. Do not copy a current documentation example into an older +installed version without checking its exports. When 1.2.0 is installed, prefer +its native parser features over local compatibility shims. + +## Package capability map + +Select packages by owned capability. Do not install the entire ecosystem. + +| Capability | Package or integration | Ownership | +|---|---|---| +| Grammar algebra and value parsing | `@optique/core` | Options, arguments, commands, constructs, modifiers, values, usage metadata | +| Executable runner | `@optique/run` | Argument execution, help/version/completion behavior where used directly | +| Command modules and program runner | `@optique/discover` | Command definition, program metadata, registration/discovery, dispatch | +| Environment values | `@optique/env` | Typed environment and dotenv-backed source context | +| Configuration values | `@optique/config` | Binding parser terms to a loaded configuration context | +| Derived fallbacks | `@optique/derived-defaults` | Values computed after a first parse without becoming CLI values | +| Zod value parsing | `@optique/zod` | Zod diagnostics, transformations, and schema-derived metadata | +| Valibot value parsing | `@optique/valibot` | Modular validation and picklist-derived metadata where available | +| Validator-neutral boundary | Standard Schema integration | Interoperable validation; not rich completion metadata by itself | +| Prompts | `@optique/prompt` | Missing-value prompt binding and prompt contract | +| Clack presentation | `@optique/clack` | Clack-backed implementation of Optique prompt behavior | +| Inquirer presentation | Optique Inquirer integration | Alternative prompt renderer; verify exact installed package/export | +| Diagnostics options | `@optique/logtape` | Verbosity and diagnostic destination parser terms, not LogTape configuration | +| Temporal values | `@optique/temporal` | Duration, instant, and plain-date value parsers | +| Manuals | `@optique/man` | Roff manual generation from the parser/program document | +| Git values | Optique Git integration | Git-aware parsers/completion; verify exact package and repository requirement | + +The audit guidebook also names completion, configuration, environment, derived +defaults, prompt, Git, Clack, Inquirer, LogTape, Temporal, Zod, Valibot, and +Standard Schema integrations. Treat that as an ecosystem-discovery requirement, +not proof of the precise import path in an arbitrary installed revision. + +## Parser architecture + +Keep four models separate: + +```text +tokens and structural grammar + -> sparse source patch + -> resolved request schema + -> portable command handler +``` + +Optique owns the first arrow and may own typed adapters for other sources. It +does not own domain orchestration, persistence, retry policy, risk policy, or +process-global logging configuration. + +Define a portable command independently when commands must be exercised outside +the CLI: + +```ts +import type { StandardSchemaV1 } from "@standard-schema/spec"; + +export interface CommandDefinition { + readonly path: readonly string[]; + readonly patchSchema: StandardSchemaV1; + readonly requestSchema: StandardSchemaV1; + readonly resultSchema: StandardSchemaV1; + run(request: TRequest, context: CommandContext): Promise; +} +``` + +Let the executable adapter: + +1. parse raw tokens; +2. load source contexts once; +3. construct a sparse patch; +4. resolve precedence and defaults; +5. validate the complete request; +6. call the portable command; +7. route its typed outcome. + +Do not read `Deno.args`, `process.argv`, environment variables, or config files +inside the domain handler. + +## Typed grammar + +Use the smallest grammar that prevents invalid structures before execution. +The attached implementation uses these public subpaths: + +```ts +import { merge, object } from "@optique/core/constructs"; +import { optional } from "@optique/core/modifiers"; +import { message } from "@optique/core/message"; +import { negatableFlag, option } from "@optique/core/primitives"; +import { choice, integer, string } from "@optique/core/valueparser"; +``` + +Preserve absence for source-bearing values: + +```ts +import { object } from "@optique/core/constructs"; +import { optional } from "@optique/core/modifiers"; +import { option } from "@optique/core/primitives"; +import { string } from "@optique/core/valueparser"; + +export const syncPatchParser = object({ + endpoint: optional(option("--endpoint", string({ metavar: "URL" }))), + output: optional(option("--output", string({ metavar: "DIRECTORY" }))), +}); +``` + +Do not add `withDefault()` to a value that should fall through to environment or +configuration. Use `withDefault()` only for parser-local behavior or when the +parser is intentionally the sole source owner. For source-bearing values, +document defaults in help/man output without materializing them into the parsed +patch. + +Represent alternatives structurally. Prefer a parser result that already +selects one branch: + +```ts +type OutputSelection = + | { readonly mode: "stdout"; readonly format: "json" | "jsonl" } + | { readonly mode: "directory"; readonly path: string } + | { readonly mode: "remote"; readonly endpoint: URL }; +``` + +Do not independently parse `--stdout`, `--output`, and `--endpoint` into three +optional properties and defer impossible combinations to the handler when the +grammar can exclude them. + +Keep long names canonical. Attach a short alias to the same term. Model visible, +hidden compatibility, and deprecated aliases deliberately. Never accept +arbitrary command prefixes or reinterpret an unknown command silently. + +## Sparse sources and precedence + +Use one visible fallback chain: + +```text +explicit CLI + > environment + > resolved configuration + > derived default + > final request-schema default +``` + +Use an Optique source context for each non-token source. In the observed Kaiju +implementation: + +```ts +import { createConfigContext } from "@optique/config"; +import { createDerivedDefaults } from "@optique/derived-defaults"; +import { createEnvContext } from "@optique/env"; + +export const envContext = createEnvContext({ + prefix: "APP_", + envFile: [".env", ".env.local"], +}); + +export const configContext = createConfigContext({ schema: configPatchSchema }); + +export const derived = createDerivedDefaults({ + run_id: (parsed: unknown) => deriveRunId(parsed), +}); + +export const contexts = [envContext, configContext, derived.context] as const; +``` + +Bind one parser term to a source rather than parsing the same setting in +uncoordinated loaders: + +```ts +const output = bindEnv( + bindConfig(option("--output", outputParser), { + context: configContext, + key: (config) => config.output, + }), + { + context: envContext, + key: "OUTPUT", + parser: outputParser, + }, +); +``` + +Check actual wrapper nesting and source-context precedence in the installed +version. Encode the intended winner order in tests rather than inferring it from +visual nesting. + +Two-pass parsing is legitimate when an early token such as `--config` determines +which source to load. Keep the first pass minimal. Load dynamic configuration +once and reuse its result. Do not execute an async config factory before parsing +and then execute it again inside a command. When using `@optique/discover`, +thread the loaded snapshot through `runProgram()` hooks or an equivalent +composition-root resource. + +Derived defaults consume first-pass values but remain lower precedence than +authored sources. They must be deterministic for the same input and must not +perform unbounded network or filesystem work. + +## Schemas and validation + +Use parser structure for token-language validity and schemas for value/domain +validity. + +Choose adapters intentionally: + +- use Zod v4 when schemas own transformations, codecs, JSON Schema, complex + refinements, or broad server-side integration; +- use Valibot when modular imports and client bundle size materially matter; +- accept Standard Schema at portable library/plugin boundaries; +- use the Optique-specific Zod or Valibot adapter when it provides richer + diagnostics or completion metadata than the generic boundary. + +Example Zod parser: + +```ts +import { zod } from "@optique/zod"; +import * as z from "zod"; + +const endpoint = option( + "--endpoint", + zod(z.url(), { placeholder: "https://example.com" }), +); +``` + +Do not assume Standard Schema exposes enum choices, labels, JSON Schema, or +completion metadata. Store explicit completion hints beside a generic schema or +use the richer adapter. + +Keep schema stages distinct: + +- authored schema: ergonomic config syntax, operations, and shorthands; +- sparse patch schema: normalized values without defaults; +- resolved request schema: defaults, cross-field refinements, brands, and + executable domain values. + +When composing Zod objects from `.shape`, `pick`, `extend`, or manual fields, +audit refinements and transformations. Reapply cross-field rules to the final +schema and test the exact composed schema. + +Remote/asynchronous validation used for completion must be bounded, cached, and +optional. Completion should degrade to no dynamic suggestions rather than block +ordinary parsing or hang the shell. + +## Optique 1.2 features + +Use `negatableFlag()` for paired Boolean options: + +```ts +export const cachePatch = object({ + range_cache_enabled: optional(negatableFlag({ + positive: "--range-cache", + negative: "--no-range-cache", + }, { + description: message`Enable or disable range-cache reuse.`, + })), +}); +``` + +This produces one tri-state patch field: absent, explicitly `true`, or +explicitly `false`. Do not parse `--cache` and `--no-cache` as two independent +flags and reconcile them later. + +Use `choice(schema.options, { suggest: "nearest" })` for enum-like CLI +languages when the Zod schema exposes the option list. Let Optique see the +choices natively so help, completion, and spelling suggestions are generated +from the parser. Use `@optique/zod` for richer scalar schemas such as URLs, +branded values, transformations, or diagnostics that are not just an enum. + +Use `deferredValue()` only when the fallback genuinely belongs at handler time: +interactive prompt fallback, secret lookup, expensive project discovery, +handler-scoped services, or values that need a runtime context. The parsed field +is a `DeferredValue` function; calling it returns the specified value or runs +the fallback. This is not an ordinary defaulting mechanism and should not be +used for static config defaults. + +Optique 1.2 value parsers carry type-appropriate placeholders used during +deferred prompt resolution. The placeholder exists to keep first-pass parsing +and `map()` transforms structurally valid. It is not user intent and must not +be serialized into sparse patches, provenance, or final command requests. + +Use `runProgram()` lifecycle hooks for per-command resources: + +```ts +await runProgram({ + commands, + metadata, + hooks: { + async beforeEach(invocation) { + const config = await resolveOnce(invocation.value); + const logger = await configureLogger(config.logging); + return { resource: { config, logger } }; + }, + async afterEach(context) { + await context.resource?.logger.flush(); + }, + }, +}); +``` + +This is the right place to thread one resolved config snapshot, logger scope, +tracing span, or lazy service into handlers. It avoids loading c12 for source +contexts and then loading the same dynamic project config again in the handler. + +## Discovery, running, and packaging + +Use explicit command imports for bundlers and standalone executables: + +```ts +import browserDetect from "./commands/browser/detect.ts"; +import configShow from "./commands/config/show.ts"; +import configValidate from "./commands/config/validate.ts"; + +export const commands = [browserDetect, configShow, configValidate] as const; +``` + +The observed Kaiju CLI passes a static registry to `runProgram()` even though it +uses `@optique/discover` command definitions. This preserves visibility to Deno +compile and bundlers. + +```ts +await runProgram({ + commands, + args, + metadata, + help: "both", + completion: "both", + contexts, + contextOptions: { + async load(parsed) { + return loadSourcesOnce(parsed); + }, + }, +}); +``` + +Runtime directory scanning is acceptable only when the installed environment +retains the directory and supports dynamic imports. Prove it in the packed or +compiled artifact. A generator may create the static registry, but commit or +generate it before compilation and add drift detection. + +Use `@optique/run` for a direct parser runner where a command-module program is +unnecessary. Do not add both runners without an explicit composition boundary. + +## Help, completion, and manuals + +Generate these surfaces from the same parser document: + +- root and subcommand help; +- help command and/or `--help` according to product policy; +- shell completion for every shell claimed by the release; +- roff man pages through `@optique/man`; +- command examples, aliases, choices, defaults, and deprecations. + +The observed implementation uses `createProgramParser()` and +`generateManPageAsync()`. Verify their signatures against the installed +prerelease before copying that pattern. + +Help, version, completion, and manual generation must run without a valid +project config unless their content truly depends on it. A malformed config +should not prevent the user from discovering how to repair configuration. + +Test Bash, zsh, fish, PowerShell, and Nushell only when claimed. Parse or source +the generated artifact with the real shell where available. Snapshot tests alone +do not prove scripts are syntactically valid. + +## Prompts and interaction adapters + +Optique prompt integrations bind a missing value to an interaction adapter. +They do not own the product's automation policy. + +Gate every prompt on: + +- stdin is an appropriate TTY; +- CI and non-interactive policy; +- `--no-input` or equivalent; +- output mode and redirection; +- cancellation; +- secret-input requirements. + +The observed `config init` command imports `prompt` from `@optique/clack`. +Treat this as evidence of an adapter, not proof that every command is safe in +CI. Test the actual command with closed stdin and `--no-input`. + +Choose Clack or Inquirer as presentation alternatives. Do not install both +without a concrete interaction requirement. Keep parser and handler independent +from the renderer so tests can inject answers without terminal automation. + +## Logging, time, and Git integrations + +Use `@optique/logtape` for parser terms such as repeated verbosity and log +destination. Configure LogTape itself at the executable composition root. +`verbosity()` and `logOutput()` do not establish categories, sinks, redaction, +or flushing. + +Use `@optique/temporal` to parse authored durations, instants, and dates into +semantic Temporal values. Do not immediately turn a duration back into a loose +string. Validate supported units, zero/negative behavior, and serialization. + +Use Git-aware integration only when Git concepts are public command inputs or +completion targets. Inject repository discovery and subprocess capabilities; +do not make ordinary help depend on a repository. Verify exact package exports +because the guidebook records the integration family, not a stable universal +import path. + +## Error ownership + +Optique may render syntax and value-parser failures. The application boundary +owns merged-source validation, domain failures, public diagnostics, and exit +classes. + +Do not: + +- catch Optique stderr text and wrap it repeatedly; +- log a parse error and rethrow it to a second logger; +- collapse all failures to exit 2; +- silently reinterpret typos; +- expose raw schema internals without a user-facing path and correction. + +Normalize Deno task separators only at the executable boundary if tasks inject +an extra `--`. Keep that compatibility adapter outside the domain parser and +test direct binary and task invocation separately. + +## Testing and verification + +Add tests at five levels: + +1. parser algebra: option order, aliases, exclusions, missing values, and sparse + output; +2. source contexts: CLI/environment/config/derived/default precedence and one + source load; +3. resolved request: refinements, transformations, and default timing; +4. generated surfaces: help, completion, man, and registry drift; +5. subprocess/package: stdout, stderr, exit status, signals, and static command + reachability in the artifact users install. + +Use table-driven tests for every public term: + +| Case | Expected proof | +|---|---| +| flag absent, config present | config survives sparse CLI patch | +| flag and environment present | flag wins and provenance records both | +| first-pass config selector | factory executes once | +| mutually exclusive terms | parser rejects before handler | +| malformed config | help/version still work | +| hidden alias | parses but does not clutter normal help | +| completion network unavailable | completion returns promptly | +| compiled binary | every statically registered command is reachable | + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Config value disappears when no flag is passed | Parser default polluted sparse patch | Search `withDefault`, `??`, and defaulted patch schemas | +| Dynamic config runs twice | Two parser/resolver passes reload sources | Trace source-context loading, `runProgram()` hooks, and handler config reads | +| Command works from source but not binary | Dynamic discovery or asset graph is invisible | Inspect registration and compiled contents | +| Final schema accepts an invalid combination | Object composition lost a refinement | Inspect `.shape`, `pick`, `extend`, and final `superRefine` | +| Completion hangs | Remote validation/provider is unbounded | Add deadline, cache, and empty fallback | +| Help fails on malformed project config | Bootstrap surfaces load domain config too early | Split early controls from source-bearing execution | +| Prompt hangs in CI | TTY/no-input policy is outside prompt adapter | Trace interaction policy before prompt binding | +| `-v` and configured level disagree | Verbosity and source level have no explicit algebra | Define winner/raising semantics and test repeats | +| Alias appears as a new canonical command | Compatibility alias lacks visibility/deprecation metadata | Inspect command metadata and generated docs | +| Stable and dev packages resolve together | Optique package lines are incompatible | Compare import map and lockfile exact versions | + +## Sources and freshness + +- Normative source: `productionized-cli-pattern-guidebook-v1.2.md`, reviewed 2026-07-22. +- Normative ecosystem audit: `cli-guidelines-audit-and-expansion(1).md`, reviewed 2026-07-17. +- Observed implementation: current Kaiju CLI Optique 1.2.0 migration and earlier `live-browser-cli(41).zip/clis/main` evidence. +- Official project documentation: , discovery pointer for current APIs; re-verify against the installed package exports. +- Official source: , discovery pointer for package/version history and implementation details. +- Official JSR API documentation: and , verified for `deferredValue()`, `negatableFlag()`, placeholders, and hooks on 2026-07-22. + +Freshness status: package names and architectural capabilities are grounded in +the attached sources plus the 2026-07-22 Optique 1.2.0 verification. Exact +examples remain version-bound evidence. Inspect the current official docs, +installed types, and lockfile before implementation. diff --git a/skills/build-clis/references/output.md b/skills/build-clis/references/output.md index 14f4078..fbf64de 100644 --- a/skills/build-clis/references/output.md +++ b/skills/build-clis/references/output.md @@ -1,5 +1,8 @@ # Results, diagnostics, and durable artifacts +For complete LogTape category, sink, filter, formatter, context, redaction, +testing, bootstrap, and disposal patterns, load [logtape.md](logtape.md). + ## Three channels 1. Stable command results are user-requested output. They go to stdout or an diff --git a/skills/build-clis/references/testing.md b/skills/build-clis/references/testing.md index 4ad85e8..2927c91 100644 --- a/skills/build-clis/references/testing.md +++ b/skills/build-clis/references/testing.md @@ -1,51 +1,342 @@ -# CLI verification - -## Layered checks - -1. Schema tests cover normalization, refinements, defaults, aliases, and invalid - combinations. -2. Parser/source tests cover grammar, sparse patches, and adapter mapping. -3. Resolver tests cover real files, extension layers, factories, precedence, - operations, provenance, and defaults. -4. In-process program tests cover handler composition and typed results. -5. Subprocess tests cover exact stdout, stderr, exit codes, signals, TTY modes, - prompts, and bootstrap failures. -6. Generated-surface tests cover help, completion, man, docs, and drift. -7. Packaging tests cover compiled or published artifacts in a clean consumer. - -## Behavioral matrix - -At minimum verify: - -- successful human and machine-readable result; -- invalid token grammar; -- semantically invalid merged input; -- missing and malformed configuration; -- unavailable network/service; -- permission failure; -- one public diagnostic per failure; -- quiet and silent semantics; -- nested secret redaction; -- redirection and pipe behavior; -- non-interactive prompt behavior; -- first and second interrupt where implemented; -- resource cleanup and logger flush; -- resume/retry behavior where claimed. - -## Configuration oracles - -Use real temporary c12 projects for extension and factory behavior. Unit tests of -the custom merge function cannot prove loader integration. Use a counter to prove -one factory evaluation and assert resolved provenance, not only the final value. - -## Installation oracles - -Run the artifact through the same entrypoint users receive. Check version, help, -one real command, failure output, completion/man generation or installation, -required assets/workers, and uninstall/cleanup. Test every OS/architecture that -the release claims; otherwise narrow the claim. - -## Reviewability oracle - -For Markdown changes, inspect the diff and fail if unrelated paragraphs, tables, -or code blocks were reformatted. Code formatter checks must exclude Markdown. +# CLI verification playbook + +Verification must prove the public contract through the artifact users receive. +Static schema inspection, parser unit tests, and a successful type-check are +necessary evidence, but none proves the whole CLI. + +## Contents + +1. [Test layers](#test-layers) +2. [Schema and default tests](#schema-and-default-tests) +3. [Optique grammar and source tests](#optique-grammar-and-source-tests) +4. [c12 and defu tests](#c12-and-defu-tests) +5. [Provenance tests](#provenance-tests) +6. [LogTape and stream tests](#logtape-and-stream-tests) +7. [Lifecycle and cancellation tests](#lifecycle-and-cancellation-tests) +8. [Generated and installed surfaces](#generated-and-installed-surfaces) +9. [Benchmark gates](#benchmark-gates) +10. [Evidence reporting](#evidence-reporting) + +## Test layers + +| Layer | Proves | Cannot prove alone | +|---|---|---| +| Schema unit | Defaults, transforms, refinements, invalid combinations | CLI grammar or source precedence | +| Parser unit | Token language and sparse output | Real c12 loading or stream bytes | +| Merger unit | Precedence and merge algebra | c12 integration or factory count | +| Real config fixture | Discovery, `extends`, env branches, factories | Packaged executable behavior | +| In-process program | Handler/resource composition | OS signals and exact stdio | +| Subprocess | Exit, stdout, stderr, signals, TTY/pipe behavior | Published package contents | +| Generated surface | Help/completion/man agreement | Installed asset availability | +| Clean consumer | Install/compile/package contract | Every supported platform unless run there | +| Benchmark | Cost under a named workload | Correctness or general performance | + +Use the repository’s task graph when present. A Deno-oriented split often looks +like: + +```text +deno test path/to/schema_test.ts +deno test path/to/merge_test.ts +deno test --allow-read --allow-write path/to/config_fixture_test.ts +deno test --allow-run --allow-read path/to/cli_subprocess_test.ts +deno check path/to/entrypoint.ts +deno task build +deno test path/to/installed_artifact_test.ts +deno bench path/to/cli_bench.ts +``` + +Do not invent permissions or task names. Inspect the manifest and report the +actual commands and exit statuses. + +## Schema and default tests + +Test authored, sparse, and complete schemas independently. + +```ts +import { expect } from "@std/expect"; +import { describe, it } from "@std/testing/bdd"; + +describe("configuration schema stages", () => { + it("keeps source patches sparse", () => { + expect(CliPatchSchema.parse({})).toEqual({}); + }); + + it("materializes defaults only in the complete schema", () => { + expect(RuntimeConfigSchema.parse({}).timeoutSeconds).toBe(30); + }); + + it("runs prefault input through normalization", () => { + expect(RuntimeConfigSchema.parse({}).outputDirectory).toBe("./dist"); + }); +}); +``` + +Required default matrix: + +| Case | Required assertion | +|---|---| +| Help-only default | Visible in help/man; absent in parsed patch | +| Scalar `.default()` | Output-shaped fallback materializes | +| Scalar `.prefault()` | Input-shaped fallback passes through transforms/refinements | +| Nested object omitted | Inner defaults activate only when intended | +| Explicit `false` | Does not fall through | +| Explicit `0` | Does not fall through | +| Explicit empty string | Rejected or retained by documented schema policy | +| Invalid user value | Fails; does not silently become `.catch()` fallback | +| Recovery value | Labeled recovery in result/provenance | +| Composed object | Cross-field refinements still execute | + +Test the exact schema used at runtime, not a similar leaf schema. If schema +composition uses `.shape`, `.pick()`, `.extend()`, or codecs, include a +regression test for every refinement or transformation that must survive. + +## Optique grammar and source tests + +Verify successful parses and grammar failures: + +- canonical long option and short alias; +- hidden/deprecated compatibility alias; +- unknown option and nearest suggestion; +- required value and invalid scalar; +- repeated option policy; +- mutually exclusive structural branches; +- `negatableFlag()` absent, positive, negative, and conflict; +- enum `choice()` help, completion, and typo suggestions; +- help/version/completion before project config loading; +- `deferredValue()` specified branch, fallback branch, memoization policy, and + fallback error; +- placeholder never appears in patch, provenance, or request. + +For every source-bearing term, assert parser omission: + +```ts +const patch = parseCli([]); +expect(Object.hasOwn(patch, "timeoutSeconds")).toBe(false); +``` + +Then test precedence through real adapters: + +| CLI | Env | Config | Expected winner | +|---|---|---|---| +| absent | absent | absent | runtime default | +| absent | absent | present | config | +| absent | present | present | environment | +| present | present | present | CLI | +| explicit value equal to config | absent | same value | CLI, not config | + +Do not infer precedence from wrapper nesting. Encode the intended result against +the installed Optique integration packages. + +## c12 and defu tests + +### Unit-test the application merger + +Cover: + +- public highest-to-lowest input order; +- internal lowest-to-highest pair evaluation; +- null and undefined layers skipped; +- ordinary objects merged by property; +- plain arrays replace; +- empty arrays clear; +- `$append`, `$prepend`, and `$replace` evaluate against inherited arrays; +- standalone operations lower to concrete arrays; +- duplicate behavior is explicit; +- higher plain array or replace defeats lower operations; +- schema-known discriminated unions replace atomically; +- unrelated same-named properties do not become atomic accidentally; +- inputs are not mutated; +- output arrays/objects do not retain mutable input references. + +Table-driven example: + +```ts +for (const testCase of mergeCases) { + Deno.test(testCase.name, () => { + const before = structuredClone(testCase.layers); + const actual = mergeConfigInputs(...testCase.layers); + expect(actual).toEqual(testCase.expected); + expect(testCase.layers).toEqual(before); + }); +} +``` + +### Use real temporary c12 projects + +Merger unit tests cannot prove loader behavior. Create temporary files that +exercise: + +- default config discovery; +- explicit path and missing explicit path; +- supported JSON/JSONC/TypeScript formats; +- malformed or unsupported exports; +- base `extends` plus child overrides; +- base array plus child append/prepend/replace; +- selected and disabled environment branches; +- sync and async config factories; +- loader control keys consumed before strict application validation; +- environment-independent JSONC parsing, including `//` inside strings. + +Prove one dynamic factory evaluation with an external side effect: + +```ts +// The temporary config increments a marker file when evaluated. +const result = await runCliInFixture(projectDirectory, ["config", "show"]); +expect(result.code).toBe(0); +expect(await Deno.readTextFile(markerPath)).toBe("1"); +``` + +Do not reset the marker between a hidden first load and handler load; that would +mask the bug. Handlers must receive the already loaded snapshot. + +## Provenance tests + +Test value and explanation together: + +| Scenario | Value oracle | Provenance oracle | +|---|---|---| +| Config only | Config value | Config winner with file/source ID | +| Env over config | Env value | Env winner; config shadowed | +| CLI over env | CLI value | CLI winner; env and config shadowed | +| CLI equals config | Same value | CLI still explicit winner | +| Runtime default | Default value | `runtime-default`, no fabricated CLI source | +| Derived value | Derived value | Derived source, not runtime default | +| Append/prepend | Ordered array | Every contributing operation in order | +| Empty clear | Empty array | Explicit higher winner | +| Union switch | New branch only | Branch-level replacement and shadowed old branch | +| Recovery | Recovery value | `recovery`, not ordinary default | + +Additional invariants: + +- every decision path exists in the resolved config or is explicitly marked as + a transformation/removal; +- no secret appears in human explain output, JSON output, LogTape records, or + error causes; +- machine envelope has a schema version and parses with its published schema; +- human output is a rendering of the same decisions; +- aliases and `--name=value` forms retain explicit CLI evidence; +- inference is labeled when the parser integration cannot expose source + identity directly. + +Snapshot tests are useful for rendering, not for resolver correctness. Assert +winner, shadowed sources, operation, path, and redaction structurally. + +## LogTape and stream tests + +Use a recorder or in-memory sink for structured diagnostics, then subprocesses +for byte-level stream behavior. + +Verify: + +- diagnostic category and structured properties; +- level filtering and category inheritance; +- result category blocks inherited diagnostic sinks with + `parentSinks: "override"` when supported by the installed version; +- JSON/JSONL/completion/man results reach stdout as exact raw bytes; +- warnings, debug records, and bootstrap failures never contaminate stdout; +- diagnostics reach stderr or the configured file; +- nested secrets, URLs, headers, config envelopes, and error objects are + redacted before formatting; +- quiet/silent semantics do not hide required machine results or failures; +- buffers flush on success, failure, and cancellation; +- sink and resource disposal occurs once. + +Do not compare only visible strings. Assert raw byte sequences, trailing newline +policy, empty stderr/stdout where required, exit code, and generated file +contents. + +Bootstrap tests must use logging controls available before full config is +valid. A malformed config should still honor raw `--log-level`, `--log-format`, +or equivalent early controls without attempting the failing full parse twice. + +## Lifecycle and cancellation tests + +Subprocess tests should cover: + +- normal success and operational failure; +- invalid token grammar and invalid merged semantics; +- missing permissions or inaccessible files; +- closed stdin and redirected stdin; +- non-interactive prompt fallback or deterministic failure; +- first interrupt initiating graceful cancellation; +- second interrupt forcing termination when that contract exists; +- child process/worker/browser receiving cancellation; +- bounded cleanup and stable signal exit mapping; +- checkpoint/resume semantics where claimed; +- one public diagnostic per failure. + +Use synchronization instead of fixed sleeps. Have the child emit a readiness +marker or create a file before the test sends a signal. Bound every wait so a +regression fails rather than hanging the suite. + +## Generated and installed surfaces + +Generate help, shell completion, man pages, and docs from the same static command +model. Tests should detect: + +- a command present in source but missing from the standalone binary; +- visible versus hidden/deprecated aliases; +- choices or documented defaults missing from completion/help/man; +- stale generated output after grammar changes; +- missing compiled assets, workers, templates, certificates, or dynamic + imports; +- version metadata disagreement. + +Run the packaged/compiled artifact in a clean temporary consumer. Check: + +1. `--version` and `--help` without project config. +2. Every top-level command is reachable. +3. One typical real command succeeds. +4. Invalid input has exact output and exit class. +5. Completion/man generation or installation works. +6. Required runtime assets exist. +7. Uninstall/cleanup behavior where distributed by an installer. + +Run every operating system and architecture the release claims. If the matrix +was not run, narrow the claim in the handoff. + +## Benchmark gates + +Benchmarks run after semantic tests. Each fixture must pass its merge, +provenance, and output oracle before timing. Separate: + +- installed cold startup; +- warm parser/resolver throughput; +- c12 discovery and factory evaluation; +- merge and provenance overhead by layer/field/depth count; +- Zod sparse and complete parsing; +- LogTape disabled, formatted, redacted, and flush paths. + +Require the same runtime, lockfile, fixture, artifact type, and environment for +baseline and candidate. Report distributions and absolute changes. See +[benchmarking.md](benchmarking.md) for the full protocol. + +## Evidence reporting + +Separate facts by check and outcome: + +| Status | Meaning | +|---|---| +| Passed | Command completed successfully; record command and exit status | +| Failed | Command completed and found a defect | +| Blocked | Environment/resource limitation prevented a result | +| Skipped | Intentionally not run; state why | +| Unverified | Claim inferred from source/docs only | + +Do not report streamed output as a pass until the final process exit is known. +For resource-heavy Deno graphs, it can be reasonable to report focused checked +tests separately from `deno test --no-check` runtime evidence and from a broad +workspace check that OOMs or hangs. Do not turn that separation into a claim +that type-checking passed. + +For Markdown skill changes, inspect the diff and fail if unrelated paragraphs, +tables, or code blocks were reformatted. Code formatters must exclude authored +Markdown unless the user explicitly asks for reformatting. + +The final handoff should list: + +- files or public behavior changed; +- exact commands run and exit status; +- focused behaviors proved; +- broad, platform, live-service, and benchmark checks not run; +- any observed pre-existing failure kept outside the change scope. diff --git a/skills/build-clis/references/unjs-build-release.md b/skills/build-clis/references/unjs-build-release.md new file mode 100644 index 0000000..bd553e0 --- /dev/null +++ b/skills/build-clis/references/unjs-build-release.md @@ -0,0 +1,468 @@ +# UnJS build, release, content, and project-operation packages + +## Contents + +- Selection and ownership +- Versioned capability map +- unbuild package builds +- nypm package-manager operations +- Magicast source-preserving edits +- giget template acquisition +- changelogen release planning +- automd bounded generated regions +- rc9 user configuration +- std-env environment signals +- Integration sequences +- Failure signatures +- Verification +- Sources and freshness + +## Selection and ownership + +These packages solve different parts of a project operation. Do not collapse them +into one generic “UnJS tooling” abstraction. + +| Need | Package owner | Keep outside the package | +|---|---|---| +| Build a publishable JS/TS library | unbuild | export policy, runtime support promise, release acceptance | +| Detect and invoke the repository package manager | nypm | authorization, operation plan, cancellation, manifest ownership | +| Edit static-ish JS/TS configuration while preserving nearby style | Magicast | semantic validation, consent, atomic write and rollback | +| Acquire a template or repository archive | giget | source trust, pinning, destination authority, post-extract verification | +| Derive changelog and semver intent from commits | changelogen | release approval, version authority, publication credentials | +| Regenerate owned Markdown regions | automd | human-authored prose and broad formatting | +| Read or write XDG user preferences | rc9 | project config discovery, secrets, application schema | +| Detect runtime/CI/TTY/agent hints | std-env | security policy, per-stream terminal state, repository ownership | + +The exact APIs below were checked against published package declarations for the +versions named in each section. Inspect the target lockfile before copying them +into a repository with another version. + +## Versioned capability map + +| Package | Verified version | Public surface used here | Important version boundary | +|---|---:|---|---| +| unbuild | 3.6.1 | `defineBuildConfig`, Rollup/mkdist entries, declarations, stubs | README identifies obuild as an experimental successor; do not migrate by name alone | +| nypm | 0.6.8 | detection, install/add/remove/dedupe/run/dlx, command builders, `dry` | Current manager union includes npm, yarn, pnpm, bun, Deno, Aube, and nub | +| changelogen | 0.6.2 | config, git diff parsing, Markdown generation, semver decision, GitHub release helpers | Release/publish CLI paths mutate Git, manifests, tags, and registries | +| automd | 0.4.3 | `transform`, `automd`, `defineGenerator`, config resolution | Generated blocks are marker-owned; this is not a Markdown formatter | +| giget | 3.3.0 | `downloadTemplate`, providers, registry provider, offline/cache modes | Glob-string `ignore` depends on Node `path.matchesGlob` support | +| magicast | 0.5.3 | `loadFile`, `writeFile`, `parseModule`, `generateCode`, `builders` | Static-ish syntax only; helpers are experimental and may move | +| rc9 | 3.0.1 | parse/read/write/update plus XDG user-config variants | `readUser`/`writeUser`/`updateUser` are deprecated | +| std-env | 4.2.0 | runtime/provider/agent detection and environment flags | Many exported constants are evaluated once at module initialization | + +## unbuild package builds + +unbuild owns transformation into a package distribution. It does not decide +which entrypoints are public or whether the packed package works in every stated +runtime. + +```ts +// build.config.ts -- unbuild 3.6.1 +import { defineBuildConfig } from "unbuild" + +export default defineBuildConfig({ + entries: [ + "./src/index", + { + builder: "mkdist", + input: "./src/runtime/", + outDir: "./dist/runtime", + }, + ], + outDir: "dist", + declaration: "compatible", + sourcemap: true, + rollup: { + emitCJS: false, + }, +}) +``` + +`declaration: "compatible"` emits the compatibility declaration set described +by 3.6.1; `"node16"`, `true`, `false`, and auto-detection have different output +contracts. Match `package.json` `exports`, `main`, `module`, and `types` to files +that the build actually emits. + +Use builder entries deliberately: + +- Rollup bundles entry graphs and can emit declarations. +- mkdist preserves a file-oriented source structure while transpiling. +- copy moves reviewed assets. +- untyped generates schema/default/documentation artifacts. + +`unbuild --stub` is a development convenience powered by jiti. Never publish a +stubbed `dist`; run a clean non-stub build before packing. Watch mode is marked +experimental in 3.6.1. + +Verification must cross the package boundary: + +```sh +rm -rf dist +npx unbuild +npm pack --dry-run +``` + +Then install the real tarball in clean ESM and, if promised, CommonJS consumers. +Test every export condition, declaration path, subpath, asset, side effect, and +runtime. A successful source test does not prove the packed graph. + +## nypm package-manager operations + +nypm translates one operation into the selected package manager. Keep detection, +planning, authorization, execution, and verification separate. + +```ts +// nypm 0.6.8 +import { + addDependency, + detectPackageManager, + type PackageManager, +} from "nypm" + +export async function planAdd( + cwd: string, + names: readonly string[], +): Promise<{ manager: PackageManager; command: string; args: string[] }> { + const manager = await detectPackageManager(cwd, { + includeParentDirs: false, + }) + if (!manager) throw new Error("Package manager ownership is unresolved") + + const result = await addDependency([...names], { + cwd, + packageManager: manager, + dry: true, + }) + if (!result.exec) throw new Error("nypm did not produce an executable plan") + return { manager, ...result.exec } +} +``` + +Display the exact plan through the CLI result channel, obtain the required +authority, then call the same operation without `dry`. Re-read the manifest and +lockfile afterward. Use `workspace` only after resolving which workspace owns the +dependency. + +The 0.6.8 API also exports `installDependencies`, `addDevDependency`, +`removeDependency`, `dedupeDependencies`, `runScript`, `dlx`, and command-string +builders. Important constraints: + +- detection checks `packageManager`, `devEngines.packageManager`, then known + files/lockfiles; it does not decide which manifest should own a dependency; +- `includeParentDirs` can cross a package boundary, so default it deliberately; +- `dedupeDependencies({ recreateLockfile: true })` may replace a lockfile; +- the published README states Bun and Deno dedupe may remove the lockfile and + reinstall all dependencies; +- `dlx` downloads and executes code; +- the public operation options do not expose an `AbortSignal` in 0.6.8. If hard + cancellation is required, own the subprocess boundary rather than claiming + nypm propagates the root signal. + +## Magicast source-preserving edits + +Use Magicast when the owned input is static-ish JavaScript or TypeScript and a +targeted AST edit preserves more human structure than serializing a new file. + +```ts +// magicast 0.5.3 +import { generateCode, loadFile } from "magicast" + +const module = await loadFile<{ default: { integrations?: string[] } }>( + "project.config.ts", +) +module.exports.default.integrations ??= [] +module.exports.default.integrations.push("telemetry") + +const preview = generateCode(module).code +// Validate preview, show a diff, and write atomically only after authorization. +``` + +For an authorized direct write, 0.5.3 exports `writeFile(module, filename)`. The +core/browser-safe subpath exposes parsing/generation without filesystem helpers: + +```ts +import { generateCode, parseModule } from "magicast/core" + +const module = parseModule("export default defineConfig({})") +const options = module.exports.default.$type === "function-call" + ? module.exports.default.$args[0] + : module.exports.default +options.features = ["audit"] +const output = generateCode(module).code +``` + +Do not invent `createNode`: the published 0.5.3 README mentions it in one import +example, but the 0.5.3 declaration export list does not contain it. Use verified +`builders`, or stop and inspect the installed declarations. Treat +`magicast/helpers` as experimental; its README says helpers may move. + +Magicast cannot safely represent every dynamic program. Computed exports, +conditional mutation, spread-heavy structures, generated code, or unfamiliar +callee shapes require a refusal/manual path. Catch parse/shape failures, preserve +the original, and provide a concrete manual edit. Semantic validation after +generation is still mandatory. + +## giget template acquisition + +Download into a fresh staging directory, not the final project directory. + +```ts +// giget 3.3.0 +import { downloadTemplate } from "giget" + +const result = await downloadTemplate( + "gh:acme/service-template#6f1c2f5", + { + cwd: stagingRoot, + dir: "candidate", + registry: false, + install: false, + force: false, + preferOffline: true, + ignore: ["pnpm-lock.yaml", "package-lock.json"], + }, +) +``` + +Pin a tag or commit for reproducible scaffolding. Registry slugs and default +branches are discovery conveniences, not immutable identities. Record the +normalized `source`, resolved version/ref, archive checksum when available, and +template license. + +Security and authority rules: + +- `forceClean` recursively removes the destination and is destructive; +- `force` permits extraction into an existing directory and can overwrite; +- `install` delegates dependency installation to nypm and executes package + manager behavior after extraction; +- `auth`/`GIGET_AUTH` is a secret; redact it and do not place it in a result; +- a custom `TemplateProvider` or registry controls tar URL and headers; allowlist + protocols/hosts and bound size/time; +- inspect extracted paths, symlinks, scripts, hooks, binary files, and manifests + before merging into the target; +- string-glob `ignore` requires the Node versions documented by 3.3.0; use the + callback form or verify runtime support elsewhere. + +Offline mode means “use cached content only”, not “prove cached content is the +expected release”. Bind cache entries to the pinned source identity and checksum. + +## changelogen release planning + +Use the programmatic read/generate APIs to produce a reviewable release plan +before any version, commit, tag, push, GitHub release, or registry mutation. + +```ts +// changelogen 0.6.2 +import { + determineSemverChange, + generateMarkDown, + getGitDiff, + loadChangelogConfig, + parseCommits, +} from "changelogen" + +const config = await loadChangelogConfig(cwd, { + from: "v1.4.0", + to: "HEAD", + output: false, +}) +const raw = await getGitDiff(config.from, config.to, cwd) +const commits = parseCommits(raw, config) +const markdown = await generateMarkDown(commits, config) +const bump = determineSemverChange(commits, config) +``` + +Review commit classification, breaking changes, scope mapping, excluded authors, +repository links, prerelease policy, and zero-major semantics. Generated release +notes are evidence to review, not release truth. + +CLI boundaries are materially different: + +- plain `changelogen` can generate/output notes; +- `--bump` updates version and changelog state; +- `--release` can create a commit and tag; +- `--push` publishes Git changes; +- `--publish` publishes to npm; +- GitHub release sync can create or update remote releases. + +Do not combine these under one implicit “release” confirmation. Require clean +worktree/preflight evidence, explicit targets and versions, credential checks, +package verification, then separate authorization for irreversible external +steps. Never log provider tokens from config or environment. + +## automd bounded generated regions + +automd owns only explicit marker regions: + +```md + +generated content + +``` + +Preview one document without writing it: + +```ts +// automd 0.4.3 +import { transform } from "automd" + +const result = await transform(source, { + dir: repositoryRoot, +}) +if (result.hasIssues) throw new Error("automd reported generator issues") +const preview = result.contents +``` + +For repository-owned generation, `automd({ input, output, ignore, generators })` +returns per-file results and an optional `unwatch`. `defineGenerator` registers a +named generator with a `generate(context)` function. Keep custom generator input +schemas explicit and deterministic. + +Never run automd as permission to rewrap, reorder, or reformat surrounding human +prose. Verify that bytes outside marker ranges are unchanged. Remote fetch +generators introduce network, pinning, trust, and reproducibility requirements. +Watch mode needs an explicit shutdown owner. + +## rc9 user configuration + +Use rc9 for small user-scoped RC preferences, not as a replacement for c12 +project configuration. + +```ts +// rc9 3.0.1 +import { readUserConfig, updateUserConfig } from "rc9" + +type UserPreferences = { + color?: "auto" | "always" | "never" +} + +const current = readUserConfig({ name: ".kaijurc" }) +const updated = updateUserConfig( + { ...current, color: "never" }, + { name: ".kaijurc" }, +) +``` + +`readUserConfig` and related methods use `$XDG_CONFIG_HOME` or +`$HOME/.config`. The older `readUser`/`writeUser`/`updateUser` names are +deprecated in 3.0.1. + +rc9 uses dotted-key flatten/unflatten behavior by default and destr-like native +value conversion. `count=123` becomes a number; quote a value when the domain +requires a string. Conflicting `x=` and `x.y=` keys require deliberate handling; +`flat: true` disables unflattening. Validate the result with the application +schema. Do not store long-lived secrets merely because the path is user-scoped, +and do not assume rc9 writes are atomic or permission-hardened without verifying +the implementation and filesystem result. + +## std-env environment signals + +std-env provides portable hints: + +```ts +// std-env 4.2.0 +import { + hasTTY, + isCI, + isMinimal, + providerInfo, + runtime, +} from "std-env" + +const environment = { runtime, isCI, isMinimal, hasTTY, providerInfo } +``` + +Use `runtime === "node"` for strict Node detection. In Node-compatible Deno or +Bun, `isNode` can also be true. `hasTTY` describes stdout; prompts still need +stdin and stderr checks. `isMinimal` is a composite hint affected by CI, test, +TTY, and `MINIMAL`; it is not user consent. + +Most exported flags are snapshots evaluated during module initialization. +`detectProvider()` and `detectAgent()` rerun those specific detections, but do +not mutate all exported constants. Never use runtime/provider/agent detection as +an authorization or security boundary. + +## Integration sequences + +### Reviewable dependency installation + +```text +Optique parses semantic dependency/workspace terms + -> repository evidence selects the owning manifest + -> nypm detects the package manager + -> nypm dry mode produces exact command and args + -> LogTape result channel presents the plan + -> authorization permits apply + -> manifest and lockfile are re-read and checked +``` + +### Safe scaffold + +```text +pinned giget source -> fresh staging directory -> archive/source record + -> inspect paths, manifests, scripts, licenses, and generated files + -> Magicast previews bounded config edits if needed + -> nypm dry plan -> authorized dependency install + -> repository checks and packed/build artifact verification + -> merge reviewed files into target +``` + +### Release + +```text +clean source -> unbuild clean build -> pack/install consumer tests + -> changelogen read-only plan -> reviewed version and notes + -> authorized manifest/changelog update -> repeat build and tests + -> separate commit/tag/push/publish authorization + -> registry/install verification and release record +``` + +## Failure signatures + +| Signature | Likely cause | Required correction | +|---|---|---| +| package works from source but import fails after publish | unbuild output and export map disagree | inspect tarball and clean consumer resolution | +| linked development works but package contains jiti stubs | `unbuild --stub` was packed | clean non-stub build before pack | +| dependency added to wrong workspace | nypm manager detection was mistaken for manifest ownership | resolve workspace owner before apply | +| cancellation leaves package manager running | nypm API has no signal in the pinned surface | own a cancellable subprocess boundary | +| config edit drops comments or throws on access | Magicast input is outside supported static-ish shape | preserve original and use manual/specialized AST path | +| scaffold deletes existing project | `forceClean` used without destination authority | stage in new directory and prohibit implicit deletion | +| cached template is stale or malicious | offline cache not bound to source digest | pin and verify source/checksum | +| generated changelog chooses wrong bump | commit convention/scope/zero-major policy mismatch | review parsed commits and explicit version | +| Markdown diff rewrites prose | automd run escaped marker ownership | assert bytes outside markers unchanged | +| RC token changes type | rc9 native parsing converted an unquoted value | quote and validate domain string | +| prompt runs in CI | `hasTTY` or `isMinimal` treated as complete interaction policy | inspect stdin/stdout/stderr and explicit noninteractive mode | + +## Verification + +1. Pin package versions and inspect their declarations in the target lockfile. +2. Run nypm mutation APIs in `dry` mode and snapshot exact manager/command/args. +3. Run Magicast and automd against adversarial fixtures; assert unowned bytes are + unchanged and invalid shapes fail without writes. +4. Download giget templates into isolated temporary directories; test pinned, + offline, private, invalid archive, symlink, traversal, overwrite, and size + cases. +5. Build with unbuild, inspect the tarball, install it in clean consumers, and + execute every public export/runtime promise. +6. Generate changelog/version plans from fixed Git histories before testing any + mutation command in an isolated remote/registry. +7. Test rc9 with XDG overrides, dotted-key collisions, quoted/native values, + permissions, interrupted writes, and corrupted files. +8. Test std-env under Node, Deno compatibility, CI, no-TTY, and per-stream + redirection; do not snapshot only one developer shell. + +## Sources and freshness + +Primary sources inspected 2026-07-17: + +- published npm package declarations and READMEs for unbuild 3.6.1, nypm 0.6.8, + changelogen 0.6.2, automd 0.4.3, giget 3.3.0, magicast 0.5.3, rc9 3.0.1, + and std-env 4.2.0; +- official repositories under ; +- `live-browser-cli(41).zip` package manifests and lockfile as repository + evidence, not proof that every package is installed or used. + +The npm tarball URLs and integrity values are recorded in `evals/sources.json`. +Recheck installed declarations before using a different version. In particular, +unbuild internals, Magicast helpers, changelogen release behavior, giget provider +and runtime options, and std-env provider lists are version-sensitive. diff --git a/skills/build-clis/references/unjs-fetch-state.md b/skills/build-clis/references/unjs-fetch-state.md new file mode 100644 index 0000000..0a16dd1 --- /dev/null +++ b/skills/build-clis/references/unjs-fetch-state.md @@ -0,0 +1,1047 @@ +# UnJS fetch, state, fingerprints, and hooks + +## Contents + +- [When to load this reference](#when-to-load-this-reference) +- [Version and evidence boundary](#version-and-evidence-boundary) +- [Capability ownership](#capability-ownership) +- [ofetch](#ofetch) +- [unstorage](#unstorage) +- [ohash](#ohash) +- [hookable](#hookable) +- [Integration patterns](#integration-patterns) +- [Exclusions and non-capabilities](#exclusions-and-non-capabilities) +- [Failure signatures](#failure-signatures) +- [Testing and verification](#testing-and-verification) +- [Sources and freshness](#sources-and-freshness) + +## When to load this reference + +Load this reference when a CLI uses or is considering one or more of: + +- `ofetch` for HTTP transport; +- `unstorage` for cache, local state, or checkpoint persistence; +- `ohash` for cache keys, fingerprints, or structural comparison; +- `hookable` for application-owned extension points. + +Read only the package sections relevant to the change. Read all four sections +when they form one fetch-cache, resumable-import, or plugin lifecycle. + +This reference defines application ownership around the packages. It does not +make a package direct merely because it appears in a lockfile, and it does not +upgrade a cache into a durable workflow, a hash into an idempotency protocol, or +a hook collection into a trusted plugin system. + +## Version and evidence boundary + +The exact stable package artifacts verified on 2026-07-17 are: + +| Package | npm `latest` | Exact surface used here | Important alternate line | +|---|---:|---|---| +| `ofetch` | `1.5.1` | npm 1.5.1 package README, exports, declarations, and built source | npm `alpha` is `2.0.0-alpha.3` | +| `unstorage` | `1.17.5` | npm 1.17.5 package, declarations, drivers, core, and official guide | npm `alpha` is `2.0.0-alpha.7` | +| `ohash` | `2.0.11` | npm 2.0.11 package README, exports, and declarations | npm `1x` is `1.1.6` | +| `hookable` | `6.1.1` | npm 6.1.1 package README, exports, declarations, and built source | attached projects also pin `5.5.3` | + +Treat the version in the repository manifest and lockfile as authoritative for +that repository. The attached evidence contains several distinct states: + +- the attached CLI lock resolves `ohash@2.0.11`; +- the attached finance, Better Auth, Kaiju website, and motion locks resolve + stable `ofetch@1.5.1`, `unstorage@1.17.5`, and `ohash@2.0.11` transitively; +- attached projects resolve both `hookable@5.5.3` and `hookable@6.1.1`; +- the attached Kaiju site scope also resolves `ofetch@2.0.0-alpha.3` and + `unstorage@2.0.0-alpha.7` through another dependency graph. + +A lockfile occurrence proves resolution, not direct application use or an +approved API. Do not copy the stable examples below into an alpha dependency. +Inspect that exact alpha package's exports, declarations, changelog, and source. +Likewise, do not use a v6-only Hookable surface in a package pinned to v5. + +## Capability ownership + +| Concern | Package role | Application-owned contract | +|---|---|---| +| HTTP request and response mechanics | `ofetch` | endpoint policy, authentication, schema validation, retry safety, deadline, cancellation, redaction, error mapping | +| Key-value access and backend adaptation | `unstorage` | key namespace, value schema, migration, consistency, atomicity, retention, lease, recovery, driver selection | +| Structural serialization and digest | `ohash` | canonical input envelope, schema version, identity meaning, collision policy, secret handling | +| Awaitable in-process callbacks | `hookable` | hook vocabulary, payload schema, ordering, timeout, cancellation, trust, failure policy, cleanup | + +Prefer small application adapters over exporting package instances throughout a +codebase. This keeps package-version details at one boundary and prevents +interceptors, driver options, fingerprints, and hooks from becoming invisible +global policy. + +## ofetch + +### Exact stable surface + +The verified 1.5.1 root export includes: + +```ts +import { + FetchError, + ofetch, + type FetchOptions, + type FetchResponse, +} from "ofetch"; +``` + +The callable `ofetch` has three additional properties: + +- `ofetch.raw(request, options)` returns the `Response` extended with parsed + data in the package-specific `_data` field; +- `ofetch.native` exposes the underlying native-compatible `fetch` function; +- `ofetch.create(defaults)` creates another callable instance. + +There is no default export. The package has conditional exports for Node and +browser, Deno, worker, and other web-compatible conditions. In Node, the 1.5.1 +package uses its Node entry and `node-fetch-native`; where `globalThis.fetch` is +available it uses that implementation. + +### Response and request behavior + +`ofetch()` supplies a compile-time result type only. It does not validate the +wire response. Fetch as `unknown`, then parse with the repository's runtime +schema owner: + +```ts +const raw: unknown = await ofetch("/api/jobs/42", { + baseURL: config.apiBaseUrl, + retry: false, + signal, +}); + +const result = JobResponseSchema.safeParse(raw); +if (!result.success) { + throw new InvalidRemoteResponse({ cause: result.error }); +} + +return result.data; +``` + +`JobResponseSchema` and `InvalidRemoteResponse` are application-owned symbols in +this example, not `ofetch` APIs. + +In 1.5.1: + +- JSON-compatible object bodies are stringified for payload methods; +- `content-type: application/json` and `accept: application/json` are added for + JSON-compatible bodies on the packaged implementation's payload methods + (`POST`, `PUT`, `PATCH`, and `DELETE`) when absent; +- response parsing is selected from `responseType`, `parseResponse`, or the + response content type; +- supported explicit response types are `json`, `text`, `blob`, `arrayBuffer`, + and `stream`; +- the default JSON path uses `destr`, not `JSON.parse`; +- `query` is the current option and `params` is a deprecated alias; +- `baseURL` and query composition use `ufo` semantics; +- HTTP 4xx/5xx responses throw `FetchError` unless `ignoreResponseError` is + true. + +Use `parseResponse: JSON.parse` only when strict JSON parsing is intentionally +required. A generic type argument does not change parsing or validate shape. + +### Client defaults and interceptors + +Create one transport adapter per endpoint policy rather than one mutable global +client: + +```ts +const api = ofetch.create({ + baseURL: config.apiBaseUrl, + headers: { + accept: "application/json", + }, + retry: false, + onRequest({ options }) { + options.headers.set("x-request-id", requestContext.id); + }, + onResponse({ response }) { + diagnostics.httpResponse({ + requestId: requestContext.id, + status: response.status, + }); + }, + onRequestError({ error }) { + diagnostics.httpTransportFailure({ + requestId: requestContext.id, + error, + }); + }, + onResponseError({ response }) { + diagnostics.httpStatusFailure({ + requestId: requestContext.id, + status: response.status, + }); + }, +}); +``` + +`requestContext` and `diagnostics` are application-owned dependencies. Do not +log complete request options, headers, URLs with secret query parameters, or +parsed response bodies by default. + +Interceptor arrays are awaited sequentially. In the verified implementation, +`onResponse` runs after body parsing, including for an error status, and then +`onResponseError` runs before retry/error mapping. With +`ignoreResponseError: true`, the status-error branch and `onResponseError` are +skipped. Do not use `ignoreResponseError` merely to inspect a failure; map the +thrown `FetchError` or use an explicitly owned raw-response policy. + +`ofetch.create()` defaults are cloned and inherited only one level deep. Treat +nested defaults such as mutable header objects as construction-time values. +Do not mutate a shared object after creating clients and expect isolated state. + +### Errors + +Map package errors once at the HTTP adapter boundary: + +```ts +try { + return await api(path, options); +} catch (error) { + if (error instanceof FetchError) { + throw new RemoteRequestFailed({ + cause: error, + status: error.status, + data: error.data, + }); + } + throw error; +} +``` + +`RemoteRequestFailed` is application-owned. A `FetchError` exposes optional +`request`, `options`, `response`, `data`, `status`, `statusText`, and compatible +status aliases. Do not serialize the whole error object into user output: it can +retain a request and options containing credentials. + +### Retry policy + +The verified 1.5.1 default is one retry for non-payload methods and zero retries +for `POST`, `PUT`, `PATCH`, and `DELETE`. The default status set is `408`, `409`, +`425`, `429`, `500`, `502`, `503`, and `504`; the default delay is zero. A +numeric `retry` explicitly supplied by the caller also applies to payload +methods. + +Therefore: + +- set `retry: false` when the domain owner performs retries; +- do not add payload retries merely because an idempotency header exists; +- define attempt count, total deadline, delay/backoff, retryable transport + errors, retryable statuses, and server `Retry-After` behavior explicitly; +- prove that the remote operation recognizes the chosen idempotency identity; +- emit one logical-operation correlation ID plus attempt numbers, not four + unrelated success/failure stories. + +The package's `retryDelay` option accepts either milliseconds or a function of +the fetch context. It is not a full retry budget or backoff protocol. + +### Cancellation and deadline + +`timeout` is milliseconds and is disabled by default. A critical 1.5.1 source +detail is that the package creates its timeout controller only when no `signal` +was supplied. Passing both a root `signal` and `timeout` does not compose them; +the supplied signal wins and the internal timeout is not installed. + +Compose the root cancellation and deadline outside `ofetch`, then pass the one +resulting signal. This helper is application code, not an `ofetch` API: + +```ts +function withDeadline( + parent: AbortSignal, + timeoutMs: number, +): { signal: AbortSignal; dispose(): void } { + const controller = new AbortController(); + const onAbort = () => controller.abort(parent.reason); + + if (parent.aborted) { + onAbort(); + } else { + parent.addEventListener("abort", onAbort, { once: true }); + } + + const timer = setTimeout(() => { + controller.abort(new DOMException("HTTP deadline exceeded", "TimeoutError")); + }, timeoutMs); + + return { + signal: controller.signal, + dispose() { + clearTimeout(timer); + parent.removeEventListener("abort", onAbort); + }, + }; +} + +const deadline = withDeadline(rootSignal, config.httpTimeoutMs); +try { + return await api("/jobs", { + retry: false, + signal: deadline.signal, + }); +} finally { + deadline.dispose(); +} +``` + +If the target runtime has a verified `AbortSignal.any` and +`AbortSignal.timeout`, those primitives can replace the helper. Still dispose +streams, readers, agents, and application resources; aborting fetch is not +process cleanup. + +### Raw responses and streams + +Use `ofetch.raw` when status, headers, or both parsed data and transport metadata +belong to the adapter result: + +```ts +const response: FetchResponse = await api.raw("/jobs/42", { + retry: false, + signal, +}); + +const etag = response.headers.get("etag"); +const raw = response._data; +``` + +`_data` is package-specific, not a standard `Response` property. Do not return +the extended response across the domain boundary when a smaller owned result +will do. + +For a stream, request `responseType: "stream"`, own the reader, propagate +cancellation, bound idle time, parse framing incrementally, and cancel/release +the reader in `finally`. `ofetch` selects `stream` automatically for +`text/event-stream` in 1.5.1, but protocol-level reconnection, cursor replay, +duplicate suppression, backpressure, and terminal event semantics remain the +application's responsibility. + +### Node dispatcher boundary + +The 1.5.1 declarations expose `dispatcher` for Node 18+ Undici-compatible +dispatchers and `agent` for the older Node polyfill path. This is runtime- +specific configuration. Do not place an Undici dispatcher in a browser/Deno +adapter or disable certificate verification as a compatibility workaround. +Proxy, connection pool, certificate, and disposal policy belong to the Node +composition root and must be tested under the packaged runtime. + +## unstorage + +### Exact stable surface + +The verified 1.17.5 root exports include: + +```ts +import { + createStorage, + defineDriver, + prefixStorage, + restoreSnapshot, + snapshot, + type Driver, + type Storage, + type StorageMeta, + type StorageValue, +} from "unstorage"; + +import fsDriver from "unstorage/drivers/fs"; +import fsLiteDriver from "unstorage/drivers/fs-lite"; +import denoKvDriver from "unstorage/drivers/deno-kv"; +import httpDriver from "unstorage/drivers/http"; +import memoryDriver from "unstorage/drivers/memory"; +import redisDriver from "unstorage/drivers/redis"; +``` + +Drivers are default exports from `unstorage/drivers/`. Some drivers need +optional peer dependencies even though the driver module is listed in the +package export map. Verify and install the selected driver's peers in the owning +workspace package. + +### Core value and key semantics + +`createStorage()` uses an in-memory driver when none is supplied. Normal item +writes serialize non-string values; normal reads parse through `destr`. Missing +items resolve to `null`, not `undefined`. Writing `undefined` removes the item. +Raw reads and writes bypass the normal value path where the selected driver +supports them or use the core fallback serialization. + +Keys normalize into colon-delimited segments. The docs accept slash-like input, +but use one canonical key builder in the application and reserve namespaces: + +```text +cache:http:v2: +checkpoint:import:v3: +lease:import:v1: +``` + +Include the value-schema version in the namespace or envelope. Do not store +tokens, passwords, or secret response bodies merely because a driver is local. + +A TypeScript generic constrains callers; it does not validate data loaded from +disk, a remote store, an older application version, or another process: + +```ts +const storage = createStorage({ + driver: fsDriver({ + base: config.stateDirectory, + noClear: true, + }), +}); + +const checkpoints = prefixStorage(storage, "checkpoint:import:v3"); +const raw: unknown = await checkpoints.getItem(runId); + +if (raw !== null) { + const parsed = CheckpointSchema.safeParse(raw); + if (!parsed.success) { + throw new InvalidCheckpoint({ cause: parsed.error, runId }); + } +} +``` + +`Checkpoint`, `CheckpointSchema`, and `InvalidCheckpoint` are application-owned. +The schema must reject incompatible request identity, not silently coerce it. + +### Mounts and prefixes + +`storage.mount(base, driver)` routes keys with the matching normalized prefix to +that driver; more-specific mountpoints take precedence. An unmounted key uses +the default driver. `storage.unmount(base, dispose)` defaults to disposing the +removed driver. The root mount cannot be unmounted. + +```ts +const storage = createStorage(); +storage.mount( + "checkpoint:import", + fsDriver({ base: config.checkpointDirectory, noClear: true }), +); +storage.mount( + "cache:http", + memoryDriver(), +); + +await storage.setItem("checkpoint:import:run-42", checkpoint); +await storage.setItem("cache:http:job-42", cachedResponse); +``` + +This example deliberately gives checkpoints and cache different owners. The +memory mount does not become durable because another mount uses the filesystem. +Use `getMount()` and `getMounts()` in diagnostics/tests when a key could route to +the wrong backend. + +`prefixStorage(storage, base)` provides a smaller typed view; it is namespacing, +not isolation. The underlying driver, lifecycle, consistency, and permissions +are unchanged. + +### Stable and experimental operations + +The stable 1.17.5 storage surface includes `hasItem`, `getItem`, `setItem`, +`removeItem`, `getMeta`, `setMeta`, `removeMeta`, `getKeys`, `clear`, `dispose`, +`mount`, `unmount`, `getMount`, `getMounts`, `watch`, and `unwatch`, plus short +aliases such as `get`, `set`, `has`, `del`, and `remove`. + +In the 1.17.5 declarations, these are explicitly experimental: + +- `getItems` and `setItems`; +- `getItemRaw` and `setItemRaw`. + +Do not make an experimental batch/raw API part of a stable public library +contract without pinning the version and owning a compatibility adapter. + +The internal type is named `TransactionOptions`, but it is an open option bag +passed to drivers. That name does not establish begin/commit/rollback, +compare-and-set, multi-key atomicity, serializability, or exactly-once behavior. +The core `setItems` fallback performs writes in parallel, and a driver-specific +batch method still has only that driver's guarantees. + +### Driver capability contract + +The core `Driver` requires `hasItem`, `getItem`, and `getKeys`. Mutation, +metadata, raw, batch, watch, clear, and dispose methods are optional. The driver +may expose `flags.maxDepth` and `flags.ttl`, its original `options`, and a native +instance through `getInstance`. + +Capability-check the actual driver at construction time when the application +requires a feature. In the verified 1.17.5 core, a normal `setItem` returns +without writing if the driver has no `setItem`; some read-only options also make +driver mutations no-ops. A resolved promise is not proof that state changed. +For critical state, write, read back, and verify identity/version at the claimed +consistency boundary. + +Representative verified drivers: + +| Driver | Exact import/options | Guarantees to avoid inventing | +|---|---|---| +| memory | default or `unstorage/drivers/memory` with no options | process-local `Map`; `dispose()` clears it; no restart durability | +| fs | `fsDriver({ base, ignore?, readOnly?, noClear?, watchOptions? })` | colon keys map to paths; rejects `..` key segments; `readOnly` mutations and `noClear` clearing are no-ops; watch uses Chokidar | +| fs-lite | `fsLiteDriver({ base?, ignore?, readOnly?, noClear? })` | smaller filesystem surface; do not assume watch parity with `fs` | +| redis | `redisDriver({ url?, host?, cluster?, clusterOptions?, base?, ttl?, scanCount?, preConnect? })` plus `ioredis` peer | per-item/default TTL in seconds; key enumeration uses `SCAN`; no multi-key transaction is exposed by the storage API | +| Deno KV | `denoKvDriver({ base?, path?, openKv?, ttl? })` plus `@deno/kv` peer as required by this package line | backend is Deno KV, but the adapter surface does not expose Deno atomic operations | +| HTTP | `httpDriver({ base, headers? })` | maps storage operations to its HTTP protocol; not a generic REST contract or offline store | + +`preConnect` in the verified Redis driver initializes inside a `try/catch` that +writes a failure with `console.error`. In a LogTape-only CLI, avoid relying on +that path for lifecycle reporting; initialize/health-check the native client at +an owned boundary or verify a later operation and map the error through the +CLI's diagnostic transport. + +Driver options and peer ranges are version-sensitive. The table is not a reason +to pass every option supported by the native backend through configuration. +Expose only product-owned, schema-validated choices. + +### Metadata, TTL, watch, and disposal + +`getMeta()` combines native driver metadata with custom metadata stored under a +`$`-suffixed item unless `nativeOnly` is requested. Native fields and TTL support +vary by driver. Do not use an `mtime`, `ttl`, or custom metadata field for lease +correctness until the backend's atomic update and clock semantics are proven. + +`watch(callback)` returns an async-compatible unwatch function. If a driver has +no native watcher, the core can emit changes caused through the same storage +instance; it cannot observe external processes changing that backend. A watch +event contains only `"update" | "remove"` and a key. It is not a durable event +stream, replay log, or guaranteed exactly-once notification. + +Always call `storage.dispose()` from the composition root. It disposes mounted +drivers and clears the default memory driver's data. Disposing a storage is not +equivalent to flushing application work unless the selected driver explicitly +documents such a contract. + +### Snapshots and custom drivers + +`snapshot(storage, base)` enumerates keys and reads them in parallel. +`restoreSnapshot(storage, snapshot, base)` writes entries in parallel. The +snapshot does not include a transaction boundary, metadata protocol, or +concurrent-writer exclusion. Use it for controlled fixtures, migrations under a +lock, or best-effort cache transfer, not as a database backup or crash-consistent +checkpoint. + +Define a custom driver only to adapt a backend whose semantics are understood: + +```ts +interface DriverOptions { + readonly namespace: string; +} + +export const customDriver = defineDriver((options: DriverOptions): Driver => { + const values = new Map(); + + return { + name: "example", + options, + hasItem(key) { + return values.has(key); + }, + getItem(key) { + return values.get(key) ?? null; + }, + setItem(key, value) { + values.set(key, value); + }, + removeItem(key) { + values.delete(key); + }, + getKeys(base) { + return [...values.keys()].filter((key) => key.startsWith(base)); + }, + clear(base) { + for (const key of values.keys()) { + if (key.startsWith(base)) values.delete(key); + } + }, + dispose() { + values.clear(); + }, + }; +}); +``` + +The normal driver `setItem` receives the core's serialized string, not the +original object. Normalize keys to the package convention, release watchers and +handles in `dispose`, expose native capabilities only when the backend actually +has them, and add real backend integration tests. A custom driver is an adapter, +not a place to simulate unsupported transactions. + +## ohash + +### Exact v2 surface + +The verified 2.0.11 imports are: + +```ts +import { digest, hash, isEqual, serialize } from "ohash"; +import { diff } from "ohash/utils"; +``` + +The root package also exports `ohash/crypto` through a conditional Node/JS +implementation, but ordinary consumers should use the root `digest` export. + +In v2.0.11: + +- `serialize(input)` creates a best-effort stable string representation; +- `digest(string)` applies SHA-256 and Base64URL encoding; +- `hash(input)` is `serialize` followed by `digest`; +- `isEqual(a, b)` first checks `===`, then compares serialized values; +- `diff(a, b)` returns `DiffEntry` objects from `ohash/utils` with `key`, + `type`, `newValue`, optional `oldValue`, and string/JSON renderers. + +The v2 documentation explicitly says serialization is not designed for +security and intentional collisions remain possible. SHA-256 does not repair +ambiguity in the serializer or make low-entropy secrets safe to expose. + +### Fingerprint envelope + +Hash a versioned, schema-validated, normalized envelope rather than a mutable +request object: + +```ts +const fingerprint = hash({ + fingerprintVersion: 2, + operation: "catalog-import", + source: normalizedSourceIdentity, + requestedMode: request.mode, + schemaVersion: request.schemaVersion, +}); + +const checkpointKey = `checkpoint:import:v3:${fingerprint}`; +``` + +Define which inputs are material. Exclude correlation IDs, timestamps, and +presentation-only values unless they change semantic identity. Include tenant, +authorization scope, selected account, input artifact identity, and operation +version when omitting them could make two unsafe operations collide. + +Never use `hash()` as: + +- a password hash; +- a message authentication code or request signature; +- proof that untrusted content is authentic; +- an authorization token; +- an entire idempotency/recovery protocol; +- a compatibility guarantee across package upgrades without a golden test. + +For server-side idempotency, the server must atomically bind an identity to an +operation/result and reject incompatible reuse. `ohash` can produce a component +of that identity after its collision and disclosure properties are accepted. + +### Comparison and diffs + +Use `isEqual` for the package's structural serialization semantics, not domain +equality such as URL identity, timestamps within tolerance, database row +identity, or secret comparison. Use `diff` for diagnostics after redacting or +projecting sensitive fields: + +```ts +const changes = diff( + redactForDiff(previousConfig), + redactForDiff(nextConfig), +); + +for (const change of changes) { + diagnostics.configChange({ + key: change.key, + type: change.type, + summary: change.toString(), + }); +} +``` + +`redactForDiff` and `diagnostics` are application-owned. Do not log +`newValue`/`oldValue` blindly; the diff object can retain original values. + +### Version boundary + +`ohash` v2 has different documentation and outputs from the maintained v1 line. +Persist a fingerprint algorithm/version beside the value. On upgrade, either +retain the old reader, migrate under an explicit policy, or invalidate the +cache. Do not silently recompute a persisted recovery identity with a new major +version. + +## hookable + +### Exact v6 surface + +The verified 6.1.1 root exports include: + +```ts +import { + Hookable, + HookableCore, + createDebugger, + createHooks, + flatHooks, + mergeHooks, +} from "hookable"; +``` + +Use `createHooks()` or `new Hookable()` for typed product hooks: + +```ts +interface CommandContext { + readonly command: string; + readonly signal: AbortSignal; + readonly requestId: string; +} + +interface CliHooks { + "command:before": (context: CommandContext) => void | Promise; + "command:after": ( + context: CommandContext, + outcome: "succeeded" | "failed" | "cancelled", + ) => void | Promise; +} + +const hooks = createHooks(); + +const unregister = hooks.hook("command:before", async (context) => { + await extension.prepare(context); +}); + +try { + await hooks.callHook("command:before", context); +} finally { + unregister(); +} +``` + +`extension` is application-owned. Hook payloads should carry the owned signal +and immutable identifiers rather than giving extensions an entire mutable +composition root. + +`hook()` and `hookOnce()` return unregister functions. `addHooks()` accepts a +nested object, flattens names with `:`, and returns one unregister function. +`removeHook`, `removeHooks`, `removeAllHooks`, and v6 `clearHook` provide other +cleanup paths. + +### Ordering and failures + +`callHook(name, ...args)` invokes registered handlers sequentially in +registration order. It awaits an async handler before starting the next. A +thrown/rejected handler rejects the call and later handlers do not run. + +`callHookParallel(name, ...args)` starts handlers through `Promise.all`. It can +reduce latency only when handlers are independent. It does not roll back a +handler that completed before another rejected, establish deterministic +completion order, limit concurrency, or aggregate all failures. + +Choose and document one policy per hook: + +- fail-fast required hook; +- best-effort hook whose failures are collected and reported; +- independently bounded parallel hook; +- compensatable hook with an application-owned rollback protocol. + +Hookable supplies only the invocation primitive. Add timeout/cancellation in +the handler contract, and do not swallow hook failure at the command boundary. + +`beforeEach` and `afterEach` register synchronous spy callbacks. In v6 the +`afterEach` callbacks are run from `finally` when an async hook call rejects. +They are suitable for light instrumentation, not awaited work, output rendering, +or resource cleanup. + +### Minimal core and version differences + +`HookableCore` is a v6 smaller surface with `hook`, `removeHook`, and +`callHook`. Use it only when those operations are sufficient. It is absent from +the verified 5.5.3 package. + +The attached dependency graphs contain both 5.5.3 and 6.1.1. Common operations +such as `createHooks`, `hook`, `hookOnce`, sequential `callHook`, and parallel +`callHookParallel` exist in the inspected v5.5.3 line, but v6 adds +`HookableCore` and `clearHook` and changes the low-level `callHookWith` calling +shape. Keep `callHookWith` behind an adapter or avoid it; inspect the installed +declaration before writing a custom caller. + +Since v5, a hook failure rejects `callHook` rather than being redirected to a +global error hook. Do not write code assuming the older pre-v5 swallow-and-log +behavior. + +### Output and deprecation traps + +The verified v6 `createDebugger()` uses `console.time`, `console.timeLog`, and +`console.timeEnd`. Registering a handler through a deprecated hook name can use +`console.warn`. +Hookable v5 removed its old custom logger parameter. Therefore, in a CLI where +LogTape is the sole output transport: + +- do not enable `createDebugger` in production command paths; +- do not delegate public deprecation rendering to Hookable; +- normalize deprecated hook names at the application/plugin boundary and emit + the warning through the owned LogTape category; +- capture stdout/stderr in tests to prove no package helper bypasses transport. + +### Plugin boundary + +Hookable is not a plugin loader or sandbox. If third-party code registers hooks, +the application must define: + +- discovery, allowlisting, version negotiation, and integrity; +- hook names and runtime-validated payloads; +- registration and teardown ownership; +- execution order, reentrancy, concurrency, timeout, and cancellation; +- exception and partial-side-effect policy; +- filesystem, network, environment, and process capability restrictions; +- secret and diagnostic exposure; +- compatibility and deprecation policy. + +Prefer explicit service interfaces when one extension owns a coherent +capability. Use hooks for genuine many-listener lifecycle extension, not as a +replacement for dependency injection or ordinary function calls. + +## Integration patterns + +### HTTP cache with validated values + +```text +application normalizes request and authorization scope + -> ohash fingerprints a versioned non-secret identity + -> unstorage reads cache value + -> runtime schema validates cached envelope and expiry + -> ofetch performs a signal-bound, explicitly retried request on miss + -> runtime schema validates remote response + -> unstorage writes through a driver with proven retention semantics + -> LogTape records redacted cache/request outcome +``` + +Do not assume every driver honors `ttl`. Store an application expiry in the +validated envelope when expiry correctness must be portable, and treat backend +TTL as cleanup optimization unless its semantics were verified. + +Avoid a cache stampede with an application-owned single-flight or lease. Neither +`ohash` nor `unstorage` gives cross-process compare-and-set through the generic +surface. + +### Resumable import + +```text +CLI validates request and resolves source identity + -> ohash produces versioned request fingerprint + -> unstorage loads and schema-validates committed checkpoint + -> application rejects incompatible fingerprint/version + -> ofetch resumes from the remote cursor with one composed signal + -> domain commit succeeds + -> checkpoint advances after commit, never before + -> reconciliation proves ambiguous external effects +``` + +This can support application recovery only when the selected driver survives the +claimed failure, writes meet the required consistency boundary, domain effects +are idempotent or reconcilable, and crash tests prove the order. Calling the +state a checkpoint does not make it durable. + +### Extensible fetch lifecycle + +Keep transport and product extension separate: + +- use `ofetch` interceptors for transport-local request/response mechanics; +- use Hookable for documented application extension points; +- pass a projected immutable payload to product hooks; +- do not register the same concern in both systems; +- map hook and transport failures through the command's one public error owner. + +For example, an `import:before-commit` product hook should not mutate +`FetchOptions`; an `onRequest` transport interceptor should not decide whether a +domain import may commit. + +### Shutdown + +At the composition root: + +```text +signal stops new work + -> active fetches observe the composed signal + -> hook registration closes to new extensions + -> bounded in-flight hooks settle or cancel + -> committed state is flushed/verified according to driver contract + -> unstorage.dispose() releases mounted drivers + -> network dispatcher/client resources close + -> LogTape sinks dispose last +``` + +Do not call `process.exit()` before asynchronous disposal. Bound shutdown and +report which resources were closed, timed out, or left ambiguous. + +## Exclusions and non-capabilities + +Do not use these packages as substitutes for: + +- runtime response/config/checkpoint schemas; +- an OAuth, signing, secret-storage, or credential-refresh implementation; +- a transactional database or compare-and-set API; +- a durable queue, event log, workflow engine, or Temporal service; +- exactly-once effects; +- distributed leases without backend atomic operations and clock policy; +- a cryptographic MAC, password hash, signature, or integrity proof; +- a plugin loader, permission sandbox, module integrity system, or dependency + injection architecture; +- stable CLI result/diagnostic transport; +- domain retry, reconciliation, and recovery policy. + +Choose native `fetch` when the application does not need `ofetch` parsing, +retry, base/query, raw-response, or interceptor behavior. Choose a direct backend +client when unstorage hides a required transaction, conditional write, streaming, +query, or consistency primitive. Choose Web Crypto or a reviewed security +construction when an adversary is in the threat model. Choose explicit service +interfaces instead of Hookable when extension cardinality is known and ordered. + +## Failure signatures + +| Signature | Likely cause | Next verification | +|---|---|---| +| Request ignores configured deadline | `signal` and `timeout` supplied together under ofetch 1.5.1 | Inspect options at the adapter and test composed cancellation with a stalled server | +| Mutation runs twice | numeric ofetch retry enabled for a payload method | Capture attempts and prove remote idempotency/reconciliation | +| 404 is treated as valid data | `ignoreResponseError` suppressed status mapping | Test status matrix and require explicit expected-status policy | +| Typed response fails later | generic type asserted without runtime parse | Fetch `unknown` and validate hostile fixtures | +| Secret appears in diagnostics | complete FetchError/options/body or hook payload logged | Test redaction and project logged properties | +| Checkpoint vanishes after restart | default memory or other process-local driver | Kill the process and resume from the installed artifact | +| Write resolves but value is unchanged | driver lacks `setItem` or uses `readOnly` no-op | Read back through a fresh driver instance and verify identity | +| Clear reports success but data remains | filesystem driver has `noClear` or read-only policy | Exercise clear under selected options and assert retained keys | +| Watch misses external change | driver has no native watcher or only same-instance events observed | Change backend from another process/client | +| Batch partially applies | experimental batch/fallback has no atomic contract | Inject failure at each write and inspect all keys | +| Snapshot mixes generations | concurrent writes during enumerate/read | Run snapshot under concurrent mutation and use backend-native backup if required | +| TTL never expires | selected driver ignores TTL options | Inspect driver flags/source and test real elapsed expiry | +| Resume accepts a different request | fingerprint omits tenant/input/version | Mutate each material field and assert rejection | +| Stable data becomes unreachable after upgrade | persisted ohash output changed across major/version | Run golden vectors and version the key algorithm | +| Later hook never runs | earlier sequential hook rejected | Assert ordering and select fail-fast versus collection policy | +| Parallel hooks leave partial effects | one Promise.all branch rejected after another committed | Inject per-handler failure and add compensation or serialize | +| Raw console output bypasses LogTape | Hookable debugger/deprecation or Redis `preConnect` path used | Capture stdout/stderr on all error/deprecation paths | +| Typecheck rejects Hookable helper | copied v6 `HookableCore`, `clearHook`, or `callHookWith` shape into v5 | Inspect installed declaration and keep version adapter local | +| Alpha package behaves differently | stable manual applied to ofetch/unstorage 2 alpha | Inspect exact alpha exports/source and write a separate migration plan | + +## Testing and verification + +### Version and export verification + +Run against the repository's package manager and lockfile. Useful independent +checks are: + +```bash +npm view ofetch dist-tags --json +npm view unstorage dist-tags --json +npm view ohash dist-tags --json +npm view hookable dist-tags --json +``` + +For reproducible source inspection, fetch the exact package version, record its +registry integrity, and inspect the packaged `package.json`, declarations, +README, and implementation. Do not inspect `main` and assume it matches a lock. + +Typecheck exact imports in the owning workspace package. Verify the selected +runtime and packaged CLI, not only an editor language server. + +### ofetch tests + +Use a local controlled HTTP server and assert: + +- JSON, text, binary, empty, malformed, and schema-invalid responses; +- 2xx and every mapped 4xx/5xx class; +- actual attempt count for safe and payload methods; +- retry delay/deadline interaction; +- root cancellation, deadline cancellation, and cancellation during retry + delay; +- headers/query/baseURL without logging credentials; +- `onResponse`/`onResponseError` order and `ignoreResponseError` behavior; +- `raw` headers plus `_data` parsing; +- stream cancellation and reader cleanup; +- proxy/dispatcher behavior only in the runtime that owns it. + +### unstorage tests + +Run the same contract suite for every supported driver, plus driver-specific +checks: + +- missing item is `null` and `undefined` writes remove; +- value round trip followed by runtime schema validation; +- canonical keys, mount precedence, prefix views, and traversal rejection; +- read-only/no-clear behavior; +- batch partial failure and lack of assumed atomicity; +- metadata and TTL only where documented; +- same-instance and external-process watch behavior; +- process kill/restart at each checkpoint boundary; +- corrupted and old-version values; +- optional peer missing, authentication failure, network partition, and + permission denial; +- unmount and final disposal leave no open handles; +- read-back from a fresh client proves a critical write. + +### ohash tests + +Maintain golden vectors per fingerprint version and package version. Prove: + +- object key ordering behaves as expected for the normalized input; +- every material field changes the fingerprint; +- presentation-only fields do not when intentionally excluded; +- old checkpoint/cache keys remain readable or are deliberately invalidated; +- no secret or low-entropy sensitive value is exposed through a public hash; +- `diff` diagnostics are redacted before rendering. + +Golden vectors detect compatibility changes; they do not prove collision +resistance for the serializer or make the construction secure. + +### Hookable tests + +Assert: + +- registration order and sequential awaiting; +- fail-fast rejection and which later handlers did not run; +- parallel partial effects and the chosen policy; +- unregister, `hookOnce`, bulk add/remove, and shutdown cleanup; +- signal/timeout propagation in every async handler; +- reentrant calls and recursive hook policy; +- typed payload plus runtime validation at external plugin boundaries; +- no `console.*` output on production paths; +- compatibility against every supported installed major. + +Do not run a broad Markdown formatter as part of these checks. Preserve authored +reference layout and inspect the focused diff. + +## Sources and freshness + +Verified 2026-07-17 from primary package artifacts and official documentation: + +- `ofetch@1.5.1` registry artifact: + , integrity + `sha512-2W4oUZlVaqAPAil6FUg/difl6YhqhUR7x2eZY4bQCko22UXg3hptq9KLQdqFClV+Wu85UX7hNtdGTngi/1BxcA==`. +- Official ofetch repository/tag and v1 documentation boundary: + and + . +- `unstorage@1.17.5` registry artifact: + , integrity + `sha512-0i3iqvRfx29hkNntHyQvJTpf5W9dQ9ZadSoRU8+xVlhVtT7jAX57fazYO9EHvcRCfBCyi5YRya7XCDOsbTgkPg==`. +- Official unstorage guide and custom-driver contract: + and + . +- `ohash@2.0.11` registry artifact: + , integrity + `sha512-RdR9FQrFwNBNXAr4GixM8YaRZRJ5PUWbKYbE5eOsrwAjJW0q2REGcf79oYPsLyskQCZG1PLN+S/K1V00joZAoQ==`. +- Official ohash v2.0.11 source and migration boundary: + . +- `hookable@6.1.1` registry artifact: + , integrity + `sha512-U9LYDy1CwhMCnprUfeAZWZGByVbhd54hwepegYTK7Pi5NvqEj63ifz5z+xukznehT7i6NIZRu89Ay1AZmRsLEQ==`. +- Official Hookable v6.1.1 source: + . +- Attached version evidence: `live-browser-cli(41).zip/deno.lock`, + `new-finance-app(1).zip/aube-lock.yaml`, + `old-finance-app(1).zip/aube-lock.yaml`, + `better-auth.zip/aube-lock.yaml`, + `kaiju-website(6).zip/pnpm-lock.yaml`, + `kaiju-site-scope(17).zip/pnpm-lock.yaml`, + `solid-primitives(2).zip/pnpm-lock.yaml`, + `solid-motion-experiments.zip/aube-lock.yaml`, and + `thunderstrike-blog(4).zip/pnpm-lock.yaml`. + +Freshness limitations: + +- npm dist-tags and official `main` documentation can change after the review + date; exact-version package artifacts control the examples above. +- the stable `ofetch` and `unstorage` examples do not describe their v2 alpha + lines. +- only representative unstorage drivers were inspected in detail; verify the + exact selected driver, peer package, and backend behavior. +- package presence in attached locks is not evidence that application code uses + the package directly or relies on the behavior documented here. +- no claim is made that a generic storage driver supplies transactions, + compare-and-set, durable watches, or workflow recovery without backend-level + proof. diff --git a/skills/build-clis/references/unjs-runtime-config.md b/skills/build-clis/references/unjs-runtime-config.md new file mode 100644 index 0000000..f115fc1 --- /dev/null +++ b/skills/build-clis/references/unjs-runtime-config.md @@ -0,0 +1,1269 @@ +# UnJS runtime and configuration cluster manual + +## Contents + +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Verified versions and evidence boundary](#verified-versions-and-evidence-boundary) +- [Capability ownership](#capability-ownership) +- [Recommended integration order](#recommended-integration-order) +- [jiti runtime module loading](#jiti-runtime-module-loading) +- [c12 configuration loading](#c12-configuration-loading) +- [defu merge behavior](#defu-merge-behavior) +- [destr boundary parsing](#destr-boundary-parsing) +- [confbox structured formats](#confbox-structured-formats) +- [pkg-types repository metadata](#pkg-types-repository-metadata) +- [pathe filesystem paths](#pathe-filesystem-paths) +- [ufo URLs and query strings](#ufo-urls-and-query-strings) +- [Complete integration patterns](#complete-integration-patterns) +- [Failure signatures](#failure-signatures) +- [Testing and verification](#testing-and-verification) +- [Sources and freshness](#sources-and-freshness) + +## When to load this reference + +Load this reference when a CLI or developer tool uses, evaluates, discovers, +merges, reads, writes, or validates runtime configuration with any of these +packages: + +- jiti; +- c12; +- defu; +- destr; +- confbox; +- pkg-types; +- pathe; +- ufo. + +Also load it when a project says only “use the UnJS ecosystem” for config, +paths, package metadata, or URLs. That phrase is not an implementation plan. +Select packages by capability, assign one owner to each boundary, and verify the +installed version before copying an API. + +Read [c12-defu.md](c12-defu.md) as well when the application needs a +field-specific merge algebra, provenance, declarative array operations, or +separate authoring, sparse-patch, and runtime schemas. Read +[output.md](output.md) when a dependency can write directly to stdout or +stderr. + +## Outcome + +Following this reference should produce a configuration subsystem in which: + +- repository and workspace ownership is discovered deliberately; +- file and URL operations use the correct namespace; +- every enabled config source and precedence edge is explicit; +- executable config is treated as trusted code, never as data parsing; +- syntax parsing is separated from runtime schema validation; +- merge semantics are product policy rather than an accidental defu default; +- config edits state their lossiness and use the product's write-safety policy; +- package versions, export paths, and optional peers are verified; +- library diagnostics do not silently violate the CLI's output contract; +- failure cases are exercised through the public application resolver. + +## Verified versions and evidence boundary + +The APIs in this reference were rechecked on 2026-07-17 against the published +npm artifacts, including each package's export map, README, declarations, and +runtime bundle where behavior was ambiguous. + +| Package | Version inspected | npm tag observed | Important status | +| --- | --- | --- | --- | +| jiti | 2.7.0 | `latest` | The synchronous CommonJS-style call is deprecated; use `await jiti.import()` | +| c12 | 4.0.0-beta.5 | `latest` | v4 is still a beta version; npm also exposes `3x` as 3.3.4 | +| defu | 6.1.7 | `latest` | Array concatenation and nullish skipping are default behavior | +| destr | 2.0.5 | `latest` | “Strict” is not identical to the JSON grammar | +| confbox | 0.2.4 | `latest` | JSONC parsing is fault tolerant unless errors are inspected | +| pkg-types | 2.3.1 | `latest` | Types and readers do not perform application schema validation | +| pathe | 2.0.3 | `latest` | Normalizes filesystem paths to `/`; it is not a containment check | +| ufo | 1.6.4 | `latest` | Parsing and normalization are permissive utilities, not URL authorization | + +The attached `live-browser-cli(41).zip` independently proves these application +facts: + +- the root resolves c12 4.0.0-beta.5 and defu 6.1.7 while its Deno import map + also contains c12 3.3.4; +- `packages/config/deno.json` selects jiti 2.7.0 and c12 4.0.0-beta.5; +- `packages/config/src/index.ts` uses c12 behind an application resolver, + disables unwanted sources, validates authored layers, keeps patches sparse, + and applies Zod defaults only after merging; +- `packages/config/src/merge.ts` uses `createDefu()` only as the recursive + engine beneath explicit array and atomic-union policy; +- the ClickHouse tooling uses both `createJiti()` and c12, then validates the + imported value before migrations, seeds, or connections can consume it. + +Those codebases are examples of ownership, not universal APIs. Their local +helpers, schemas, array operation language, and package aliases are not exports +from UnJS packages. + +Do not cross-apply c12 v4 examples to c12 3 without checking the resolved +declarations. In v4, `chokidar`, `giget`, `jiti`, `magicast`, and some dotenv +support moved behind optional peer paths. In c12 3.3.4, several of those are +ordinary dependencies. A lockfile containing both versions does not make their +runtime behavior interchangeable. + +## Capability ownership + +| Owner | Owns | Does not own | +| --- | --- | --- | +| application schema | Accepted authored values, sparse patches, complete runtime config, transformations, defaults | File discovery or module evaluation | +| application resolver | Enabled sources, trust, precedence, merge policy, provenance, path bases, reload policy | Format-specific parser implementation | +| jiti | TypeScript/ESM-compatible module resolution, transformation, and evaluation | Sandboxing, config trust, precedence, or runtime validation | +| c12 | Config discovery, supported format loading, source layers, `extends`, environment branches, optional dotenv, watch/update plumbing | Domain schema, safe remote-code policy, or product merge semantics | +| defu | Recursive leftmost-priority default merge and custom merger callback | A complete patch language, deletion semantics, or atomic union ownership | +| destr | Convenient JSON-like scalar/value parsing with prototype-pollution defenses | Exact JSON validation, runtime type validation, or shell tokenization | +| confbox | JSON, JSON5, JSONC, YAML, TOML, and INI parsing/serialization | Semantic validation or lossless comment-preserving edits | +| pkg-types | Package, tsconfig, git config, lockfile, and workspace discovery/read/write helpers | Repository ownership truth, package-manager policy, or atomic mutation | +| pathe | Cross-platform filesystem path string operations with `/` normalization | Filesystem authorization, symlink resolution, or URL operations | +| ufo | URL/path/query encoding, parsing, joining, and normalization helpers | Origin allowlisting, SSRF prevention, signature identity, or filesystem paths | + +Keep these boundaries even though c12 depends on defu, confbox, pathe, and +pkg-types. A transitive dependency relationship does not transfer product +policy to the package. + +## Recommended integration order + +Use this order for a full configuration resolver: + +```text +explicit invocation cwd + -> pkg-types repository/workspace evidence + -> application chooses the owning root + -> pathe resolves configured filesystem paths + -> c12 discovers only authorized sources + -> confbox parses structured formats + -> native import or jiti evaluates trusted code config + -> c12 resolves explicitly allowed environment/extends layers + -> application validates every retained authored layer + -> application converts sources to sparse patches + -> explicit defu-based merger applies product field semantics + -> application removes loader-only keys + -> application validates the resolved sparse patch + -> runtime schema applies defaults and transformations once + -> ufo handles URL/query fields at their boundary + -> complete runtime config plus provenance reaches commands +``` + +`destr` is not a mandatory stage. Use it only for a deliberately ergonomic +single value, such as an environment or CLI value that may be `true`, `42`, +`null`, an array, or an object. Do not run complete config documents through +destr when confbox, c12, or an exact JSON parser owns the format. + +## jiti runtime module loading + +### Current imports and APIs + +The verified ESM entry point is: + +```ts +import { createJiti } from "jiti"; + +const jiti = createJiti(import.meta.url, { + interopDefault: true, + moduleCache: true, + sourceMaps: false, +}); + +const value = await jiti.import("./tool.config.ts", { + default: true, +}); +``` + +The second argument to `jiti.import()` accepts resolution conditions, +`parentURL`, `try`, and the `{ default: true }` shortcut. That shortcut returns +`module.default ?? module`; it does not validate the export. + +Use the async API for new code: + +```ts +const module = await jiti.import(absoluteConfigPath); +const resolved = jiti.esmResolve("./schema.ts"); +``` + +The callable `jiti(id)` and `jiti.resolve()` APIs emulate CommonJS `require()` +and are deprecated in the 2.7.0 declarations. Do not introduce them merely to +avoid making a loader async. + +Other verified entry points are: + +| Import | Purpose | Boundary | +| --- | --- | --- | +| `jiti/register` | Global Node module hook | Requires Node newer than 20 according to the published docs; affects the process globally | +| `jiti/native` | The same high-level API backed by native `import()` and `import.meta.resolve()` | Use only when the runtime natively accepts the selected syntax | +| `jiti/static` | Static entry supplied by the package | Confirm the installed behavior before treating it as a dynamic transformer replacement | + +### Verified option surface + +| Option | Verified 2.7.0 behavior | Decision rule | +| --- | --- | --- | +| `fsCache?: boolean \| string` | Enabled by default; `true` selects `node_modules/.cache/jiti` when available or a temp cache | Choose an explicit writable cache for constrained runtimes; treat cached transformed code as execution state | +| `rebuildFsCache?: boolean` | Rebuilds the filesystem transform cache | Use for controlled invalidation, not on every production invocation | +| `moduleCache?: boolean` | Enabled by default and integrates with the native CommonJS cache | Disable deliberately for reload tests; understand repeated module side effects first | +| `debug?: boolean` | Emits verbose jiti diagnostics | It writes through jiti's own console path, so it can violate a LogTape-only output contract | +| `sourceMaps?: boolean` | Adds inline source maps to transformed output | Enable when stack fidelity outweighs transformed-source size | +| `interopDefault?: boolean` | Defaults to true and proxies module/default exports for mixed ESM/CJS compatibility | Prefer explicit export contracts for config; test namespace and default behavior during upgrades | +| `extensions?: string[]` | Controls resolvable/transformed extensions | Narrowing is safer than accepting syntax the product never documents | +| `transform` and `transformOptions` | Replace/configure transformation | This is compiler ownership; use only with executable tests for the selected syntax | +| `alias?: Record` | Rewrites module IDs during resolution | Validate aliases and allowed roots; aliases are not a security boundary | +| `tsconfigPaths?: boolean \| string` | Disabled by default; `true` discovers a tsconfig, string selects one | Prefer an explicit path in monorepos to avoid adopting a neighboring package's aliases | +| `nativeModules?: string[]` | Adds modules to the native-load set | Do not use it to bypass validation of a loaded config export | +| `transformModules?: string[]` | Forces named modules through transformation | Pin and test; transforming dependencies can change runtime and cache behavior | +| `tryNative?: boolean` | Tries native loading before transformation; enabled by default when Bun is detected | Exercise native and transformed paths if the product supports several runtimes | +| `importMeta?: ImportMeta` | Supplies parent import metadata for `jiti/native` | Use the actual owning module, not a synthetic unrelated base | +| `esmEvalTempFile?: boolean` | Forces the ESM fallback through a temporary file | Account for temp-directory write permissions and cleanup | +| `jsx?: boolean \| JSXOptions` | Opt-in JSX transform | Config files should not gain JSX unless it is an explicit product feature | +| `virtualModules?: Record` | Returns preloaded values for matching module IDs | Useful for bundled binaries; validate the map as part of the build contract | + +`cache` and `requireCache` remain deprecated aliases for `fsCache` and +`moduleCache`. Do not teach new code the old names. + +The published README and declarations do not fully agree about the documented +default list for `nativeModules`. The 2.7.0 runtime builds an internal native +set containing at least `typescript` and `jiti`, then adds user entries. Treat +the runtime bundle as version-specific evidence and do not encode the internal +list into application policy. + +### Trust and isolation + +jiti executes code with the process's authority. A TypeScript config can read +files, inspect environment variables, open sockets, spawn processes, mutate +globals, and import any dependency available to it. Transformation is not +sandboxing. + +Apply these rules: + +- evaluate only files within the documented trust model; +- resolve the file and allowed root before importing it; +- do not accept a remote URL or arbitrary package name as a config path; +- validate the returned value immediately as `unknown`; +- keep secrets out of loader diagnostics and thrown source excerpts; +- decide cache behavior for watch mode and tests; +- run genuinely untrusted plugins in an isolated process or stronger sandbox + with a narrow protocol rather than in jiti. + +For default exports, prefer the explicit shortcut and schema boundary: + +```ts +const imported = await jiti.import(absolutePath, { default: true }); +const authored = AuthoringConfigSchema.parse(imported); +``` + +Do not write `jiti.import()` and treat the generic as validation. It +only changes the TypeScript view of the result. + +## c12 configuration loading + +### v4 default behavior that must be made explicit + +For c12 4.0.0-beta.5, the published runtime establishes these defaults: + +- `cwd` resolves from `process.cwd()`; +- `name` defaults to `config`; +- `configFile` defaults to `config`, or `.config` for another name; +- `rcFile` defaults to `.rc`, so local RC loading is enabled unless set + to `false`; +- `globalRc` is opt-in, but can add workspace and user sources when enabled; +- `packageJson` and `dotenv` are disabled unless selected; +- `envName` defaults to `process.env.NODE_ENV`, enabling matching `$test`, + `$development`, `$production`, or `$env[name]` branches when it is set; +- `extends` processing is enabled unless `extend: false` is passed; +- `omit$Keys` defaults to false; +- defu is the default merger. + +This means `loadConfig({ name: "tool" })` is not a neutral “load one file” +call. It can load `.toolrc`, select an environment branch, and resolve +`extends`. State each source policy. + +### Narrow application-owned loader + +Use a deliberately narrow call when only an explicit project config is part of +the contract: + +```ts +import { loadConfig, type ConfigLayer } from "c12"; +import { createJiti } from "jiti"; + +const jiti = createJiti(import.meta.url, { + interopDefault: true, + moduleCache: false, + fsCache: true, +}); + +const loaded = await loadConfig>({ + cwd: projectRoot, + name: "tool", + configFile: explicitConfigPath, + configFileRequired: true, + rcFile: false, + globalRc: false, + packageJson: false, + dotenv: false, + envName: false, + extend: false, + omit$Keys: true, + context: { + cwd: projectRoot, + configPath: explicitConfigPath, + }, + import: (id) => jiti.import(id), + resolveModule: (module) => module?.default ?? module, + merger: mergeConfigInputs, +}); + +for (const layer of loaded.layers ?? []) { + validateAuthoredLayer(layer as ConfigLayer); +} + +const filePatch = SparseConfigSchema.parse(loaded.config); +``` + +The application functions in this example are not c12 exports: + +- `mergeConfigInputs` owns field-specific semantics; +- `validateAuthoredLayer` reports source-local schema failures; +- `SparseConfigSchema` prevents defaults from entering precedence early. + +If no custom `import` is passed, c12 4 first tries native `import()`, then falls +back to jiti if native loading fails and jiti is installed. `jitiOptions` only +controls that fallback. It has no effect when `import` is supplied. + +The default module resolver is effectively `module.default || module`. Use a +custom nullish-aware resolver if falsy default exports are meaningful, then +validate the result. + +### Source priority and defaults + +c12 documents and implements high-to-low priority in this order: + +```text +overrides + > main config file + > local/global RC sources + > selected package.json field + > defaultConfig + > extended layers + > defaults +``` + +Environment-specific data is applied within each loaded layer. `defaults` is +merged after extension and remains lowest priority. `defaultConfig` participates +before extension. `overrides` is highest priority. + +Do not infer that this order matches the product's desired CLI precedence. A +typical application still needs to merge explicit CLI and environment patches +outside c12: + +```text +CLI patch + > supported environment patch + > c12 file patch + > runtime defaults +``` + +Keep every input sparse until the final merge. Applying a defaulted runtime +schema to each layer causes a high-priority layer's defaults to mask authored +values from lower-priority layers. + +### Extends and remote sources + +The verified `extend` option is either `false` or an object with +`extendKey?: string | string[]`. c12 recognizes local files/directories, +resolvable packages, and remote prefixes supported through giget. + +Remote extension is an execution and supply-chain boundary: + +- c12 can download a git/HTTP source through the optional giget peer; +- a source option can request dependency installation; +- private-source auth can be supplied; +- downloaded configuration may then be evaluated as code; +- cached clone location, ref mutability, redirects, integrity, credentials, + offline behavior, and transitive install scripts become product concerns. + +Use `extend: false` when inheritance is not a product feature. When only local +or installed presets are allowed, set `giget: false`, validate the authored +`extends` grammar before resolution where possible, pin installed preset +versions, and test traversal/alias cases. A custom c12 `resolve` callback that +returns `null` does not by itself deny a source; the default resolver continues +after an unresolved custom result. + +Do not expose remote `extends` to untrusted structured config and hope the final +schema will reject it. Fetching and loading happen before the application sees +the final value. + +### Structured formats and dynamic modules + +The verified v4 supported extension set is: + +```text +.js .ts .mjs .cjs .mts .cts .json .jsonc .json5 .yaml .yml .toml +``` + +JavaScript and TypeScript config are executable. JSONC, JSON5, YAML, and TOML +are parsed through confbox. c12 does not make either category schema-valid. + +Dynamic config functions receive the supplied `context`: + +```ts +export default async function config(context: ConfigContext) { + return { + root: context.cwd, + mode: context.mode, + }; +} +``` + +Factories should be deterministic for a given context, side-effect-light, and +validated immediately after evaluation. c12 does not impose a timeout or +cancellation contract on a config factory. + +`createDefineConfig()` is a type-oriented identity helper for object input. +It is not runtime validation and its published type does not cover an +application-specific async factory union. Define an application helper only +when its type exactly matches the authored export contract. + +### Dotenv + +c12 v4 exposes `loadDotenv()` and `setupDotenv()` and accepts +`dotenv: true | DotenvOptions` in `loadConfig()`. + +Verified options include: + +- `cwd`; +- `fileName: string | string[]`; +- `interpolate`; +- target `env` object; +- `expandFileReferences` for `_FILE` variables. + +When several files are listed, later dotenv files can override values loaded +by earlier dotenv files while pre-existing target environment values are +preserved. `setupDotenv()` writes selected values into the target environment; +`loadDotenv()` returns an object. + +Treat dotenv as an explicit mutation and secret-loading policy. `_FILE` +expansion reads and trims file contents. Do not log those values, include them +in provenance output, or enable the feature merely because the runtime is in a +container. + +On runtimes without `node:util.parseEnv`, c12 v4 may require the optional +`dotenv` peer. The v4 migration notes specifically call out legacy/Deno +support. Test the exact target runtime rather than assuming Node behavior. + +### Watching and updating + +`watchConfig()` returns a promise. The correct shape is: + +```ts +import { watchConfig } from "c12"; + +const watcher = await watchConfig({ + cwd: projectRoot, + name: "tool", + rcFile: false, + globalRc: false, + packageJson: false, + dotenv: false, + extend: false, + onWatch(event) { + recordConfigFilesystemEvent(event.type, event.path); + }, + acceptHMR({ getDiff }) { + return getDiff().length === 0; + }, + onUpdate({ oldConfig, newConfig, getDiff }) { + applyValidatedReload(oldConfig, newConfig, getDiff()); + }, +}); + +try { + useInitialConfig(watcher.config); +} finally { + await watcher.unwatch(); +} +``` + +The current published README omits `await` in one watcher snippet, but the +4.0.0-beta.5 declaration and runtime are asynchronous. Follow the installed +type. + +v4 requires the optional `chokidar` peer for watching. `debounce` accepts a +number or `false`; the runtime default is 100 milliseconds. Decide how invalid +reloads affect the last good config and command lifecycle. + +The v4 watcher and some extension paths currently issue internal +`console.warn()` calls for reload/extension failures. This matters for CLIs that +promise LogTape as the sole output transport. Either avoid those paths, isolate +and adapt the library output, or explicitly document the exception after an +executable test. Do not claim complete output routing merely because +application callbacks use LogTape. + +`updateConfig()` is imported from `c12/update`, requires the optional magicast +peer, and is marked experimental: + +```ts +import { updateConfig } from "c12/update"; + +const result = await updateConfig({ + cwd: projectRoot, + configFile: "tool.config", + createExtension: ".ts", + onCreate({ configFile }) { + return userApprovedCreation(configFile) + ? "export default {}\n" + : false; + }, + onUpdate(config) { + config.enabled = true; + }, +}); +``` + +Wrap mutation in the product's consent, dry-run, diff, validation, backup, and +atomic-write contract. The c12 API being able to update a file does not prove +that a particular failure is crash-safe or that every source format is +lossless. + +## defu merge behavior + +### Exact default semantics + +The verified import is: + +```ts +import { createDefu, defu, defuArrayFn, defuFn } from "defu"; +``` + +`defu(source, ...defaults)` gives the leftmost values higher priority: + +```ts +const result = defu( + { server: { host: "127.0.0.1" } }, + { server: { host: "0.0.0.0", port: 8080 } }, +); + +// { server: { host: "127.0.0.1", port: 8080 } } +``` + +Verified behavior that changes config semantics: + +- plain objects merge recursively; +- arrays concatenate as `[...higherPriority, ...lowerPriority]`; +- `null` and `undefined` in the higher-priority input are skipped; +- `__proto__` and `constructor` assignments are skipped; +- inputs are not mutated; +- functions, promises, regular expressions, and non-plain objects are treated + as values rather than recursively merged records. + +This is default assignment, not JSON Merge Patch. `null` cannot delete an +inherited value. A plain array does not replace an inherited array. If those +semantics are required, implement them explicitly and test every field class. + +### Custom merger callback + +`createDefu()` accepts a callback with the current destination object, key, +higher-priority value, and parent namespace. Return `true` only after handling +the field: + +```ts +import { createDefu } from "defu"; + +const replaceArrayFields = new Set([ + "plugins", + "output.targets", +]); + +export const mergeConfigInputs = createDefu( + (object, key, incoming, namespace) => { + const path = [namespace, String(key)] + .filter(Boolean) + .join("."); + + if (replaceArrayFields.has(path) && Array.isArray(incoming)) { + object[key] = [...incoming]; + return true; + } + + return false; + }, +); +``` + +The namespace is the parent path, not the complete key path. Construct the +complete path with `String(key)`. defu types the key generically and it may be a +symbol; interpolate only after conversion. + +Use exact-path checks for discriminated unions and special arrays. A check for +only the final property name can accidentally replace unrelated fields with +the same name. + +`defuFn` invokes higher-priority function values with an inherited default. +`defuArrayFn` does so only when the inherited value is an array. These helpers +are executable merge behavior. Do not use them on untrusted authored data or +as an implicit patch language. + +### Product-level merge rules + +At minimum, decide and test: + +| Field shape | Common policy | Why default defu may be wrong | +| --- | --- | --- | +| scalar | highest authored value wins | nullish skipping may hide an intended clear operation | +| nested options | recursive merge | safe only when partial nested values are meaningful | +| ordered array | replace, append, prepend, or declared operation | default concatenation chooses append without user intent | +| set-like array | union with stable order | default concatenation preserves duplicates | +| discriminated union | atomic replacement | recursive merge can produce an impossible mixed variant | +| secret/reference | atomic replacement or explicit clear | partial merge can retain stale credential fields | +| path-bearing object | merge with per-layer base metadata | merging values alone loses the owning directory | + +Validate authoring operations before merging, resolve them into ordinary +values, strip operation objects, validate the sparse result, then apply runtime +defaults once. + +## destr boundary parsing + +### Exact APIs + +```ts +import { destr, safeDestr } from "destr"; + +const loose = destr(rawValue); +const strictish = safeDestr(rawValue); +``` + +`destr()` and `safeDestr()` default to `unknown`. A generic type argument +is a compile-time assertion, not validation. + +The verified 2.0.5 behavior includes: + +- non-string input passes through unchanged; +- case-insensitive `true`, `false`, `undefined`, `null`, `NaN`, `Infinity`, + and `-Infinity` receive built-in values; +- a quoted string without escapes is unwrapped quickly; +- loose `destr()` returns an unparseable plain string unchanged; +- `safeDestr()` throws for input it attempts to parse and cannot parse; +- prototype-polluting `__proto__` and dangerous `constructor.prototype` keys + are rejected or removed depending on strictness. + +“Safe” does not mean exact JSON. `safeDestr("TRUE")` returns `true`, and +`safeDestr("undefined")` returns `undefined`. Non-string values still pass +through. Use `JSON.parse()` or confbox `parseJSON()` when exact JSON syntax is +the contract. + +Use destr for an explicitly ergonomic boundary, followed by a schema: + +```ts +import { safeDestr } from "destr"; +import * as z from "zod"; + +const DefineValueSchema = z.union([ + z.string(), + z.number().finite(), + z.boolean(), + z.null(), + z.array(z.unknown()), + z.record(z.string(), z.unknown()), +]); + +export function parseDefineValue(input: string): z.output { + return DefineValueSchema.parse(safeDestr(input)); +} +``` + +Do not use destructuring terminology to confuse destr with a config loader. It +does not discover files, track sources, apply precedence, or provide a schema. + +In non-strict mode, destr 2.0.5 can emit an internal `console.warn()` when it +drops suspicious keys. That can violate a structured-output contract. For +untrusted values, use a strict path plus application error mapping, or isolate +the parser behavior and verify the actual output channels. + +## confbox structured formats + +### Current imports + +All verified parsers and serializers are exported from `confbox`: + +```ts +import { + parseINI, + parseJSON, + parseJSON5, + parseJSONC, + parseTOML, + parseYAML, + stringifyINI, + stringifyJSON, + stringifyJSON5, + stringifyJSONC, + stringifyTOML, + stringifyYAML, +} from "confbox"; +``` + +Format-specific subpaths are also public: + +```ts +import { parseJSONC } from "confbox/jsonc"; +import { parseYAML } from "confbox/yaml"; +``` + +c12 v4 uses those subpaths internally for structured config formats. + +### Parse, validate, then resolve + +Every confbox generic is type-only. Parse as `unknown`, inspect syntax errors +where the parser supports them, then run the application schema. + +JSONC is deliberately fault tolerant. Reject collected errors explicitly: + +```ts +import { + parseJSONC, + type JSONCParseError, +} from "confbox/jsonc"; + +export function parseConfigJSONC(text: string): unknown { + const errors: JSONCParseError[] = []; + const value = parseJSONC(text, { + allowTrailingComma: true, + disallowComments: false, + allowEmptyContent: false, + errors, + }); + + if (errors.length > 0) { + throw new SyntaxError( + `Invalid JSONC at offsets ${errors.map((error) => error.offset).join(", ")}`, + ); + } + + return value; +} +``` + +If `errors` is omitted, malformed JSONC can return a partial value. A later +runtime schema with defaults may then accept the partial value and hide the +syntax error. c12 4's built-in JSONC loader calls confbox without exposing the +error array. If strict JSONC syntax is a product requirement, preflight the +selected file, restrict the format, or provide a loader path that preserves +parse diagnostics. + +### Format capabilities and lossiness + +| Format | Parser options verified | Round-trip warning | +| --- | --- | --- | +| JSON | `reviver` plus indentation/whitespace metadata options | Indentation and surrounding whitespace can be preserved on the parsed object; comments are not JSON | +| JSONC | comment/trailing-comma/empty-content flags and error collection | Comments and trailing commas are not preserved after parse/stringify | +| JSON5 | reviver; serializer replacer, space, and quote | Do not assume source comments or exact lexical choices survive | +| YAML | filename, warning hook, schema, JSON compatibility, listener; many serializer layout options | Multi-document input is rejected; comments are not preserved | +| TOML | parse and stringify | Comments and indentation are not preserved | +| INI | bracketed-array parse option; whitespace, alignment, section, sort, newline, platform, and bracketed-array output options | Style and indentation are not preserved currently | + +Common format metadata options include `indent`, +`preserveIndentation`, `preserveWhitespace`, and `sampleSize`. They preserve +limited formatting metadata, not a complete syntax tree. Cloning or replacing +the parsed object can also sever metadata used by a later serializer. + +Do not advertise confbox as a lossless editor. For human-authored config: + +1. obtain explicit authorization to edit; +2. parse and validate the original; +3. disclose format lossiness; +4. produce a reviewable diff or dry run; +5. validate the new text; +6. write through an application-owned atomic strategy; +7. reload through the public resolver. + +## pkg-types repository metadata + +### Verified package and workspace APIs + +```ts +import { + findPackage, + findWorkspaceDir, + readPackage, + resolveLockfile, + type PackageJson, +} from "pkg-types"; + +const workspaceCandidate = await findWorkspaceDir(invocationCwd, { + tests: ["workspaceFile", "gitConfig", "lockFile", "packageJson"], + workspaceFile: "furthest", + gitConfig: "closest", + lockFile: "furthest", + packageJson: "furthest", +}); + +const packageFile = await findPackage(invocationCwd); +const unvalidatedPackage: PackageJson = await readPackage(invocationCwd); +const lockfile = await resolveLockfile(invocationCwd); +``` + +`findWorkspaceDir()` checks these marker classes by default: + +1. farthest workspace file, including pnpm, Lerna, Turbo, Rush, Deno JSON/JSONC; +2. closest `.git/config`; +3. farthest supported lockfile; +4. farthest package file. + +The first successful heuristic wins. That is useful evidence, not repository +ownership truth. A nested independent project, a checked-in lockfile, or a +monorepo tool file above the intended package can change the result. Compare +the candidate with repository instructions, workspace membership, manifests, +and the operation's intended scope. + +`findPackage()` and `readPackage()` support `package.json`, `package.json5`, and +`package.yaml`. The JSON-only variants are `resolvePackageJSON()`, +`readPackageJSON()`, and `writePackageJSON()`. + +Other verified helpers include: + +- `readTSConfig()`, `writeTSConfig()`, and `resolveTSConfig()`; +- `findFile()`, `findNearestFile()`, and `findFarthestFile()`; +- `readGitConfig()`, `writeGitConfig()`, `parseGitConfig()`, and + `stringifyGitConfig()`; +- `sortPackage()` and `normalizePackage()`; +- `updatePackage()`; +- identity type helpers `definePackageJSON()`, `defineTSConfig()`, and + `defineGitConfig()`. + +### Validation, cache, and writes + +The `PackageJson` and `TSConfig` interfaces are broad static types. The readers +return values with those types but do not prove application invariants. Parse +the fields the command depends on: + +```ts +const requestCache = new Map>(); + +const manifest = OwnedManifestSchema.parse( + await readPackage(packageRoot, { cache: requestCache }), +); +``` + +`cache` accepts a boolean or a map. With `true`, pkg-types uses its module-level +cache. A long-lived watch process or a command that writes then rereads a file +can observe stale data unless it owns and invalidates a request-scoped map. + +`updatePackage()` reads the package, gives the callback a proxy, and writes the +result in the same format. Accessing common map properties such as +`dependencies` or `scripts` can auto-create an empty object. This is convenient +but still a mutation: + +```ts +import { updatePackage } from "pkg-types"; + +await updatePackage(packageRoot, (pkg) => { + pkg.scripts.verify = "deno task verify"; + return pkg; +}); +``` + +The verified 2.3.1 implementation writes directly with `writeFile`; it does not +claim an atomic temp-file/rename protocol. Its parsers and serializers also +inherit confbox's lossiness. For crash-safe or comment-sensitive edits, build a +product-owned mutation transaction instead of claiming `updatePackage()` alone +is safe. + +Do not let pkg-types choose the package manager, authorize dependency changes, +or decide which workspace is publishable. It reports metadata and path +evidence. + +## pathe filesystem paths + +### Exact imports and behavior + +```ts +import { + basename, + delimiter, + dirname, + extname, + isAbsolute, + join, + matchesGlob, + normalize, + parse, + relative, + resolve, + sep, +} from "pathe"; +``` + +pathe 2.0.3 mirrors the Node path API shape while normalizing operations to +forward slashes across platforms. `sep` is always `/`. `delimiter` remains +platform-specific (`;` on Windows and `:` elsewhere). Explicit `posix` and +`win32` variants are exported. + +Additional alias helpers live under `pathe/utils`: + +```ts +import { + filename, + normalizeAliases, + resolveAlias, + reverseResolveAlias, +} from "pathe/utils"; +``` + +Do not use a filesystem path utility for URLs. A colon, query string, percent +encoding, UNC prefix, or drive letter has different meaning across namespaces. + +### Containment is application policy + +String resolution is only the first containment stage: + +```ts +import { isAbsolute, relative, resolve } from "pathe"; + +export function resolveContainedPath(root: string, authored: string): string { + const candidate = resolve(root, authored); + const fromRoot = relative(root, candidate); + + if ( + fromRoot === ".." || + fromRoot.startsWith("../") || + isAbsolute(fromRoot) + ) { + throw new TypeError(`Path escapes the configured root: ${authored}`); + } + + return candidate; +} +``` + +This rejects obvious lexical traversal only. For security-sensitive paths, +also account for: + +- symlinks and junctions using the host filesystem's canonical path; +- a not-yet-created file whose existing parent is a symlink; +- Windows drive and UNC semantics; +- case-insensitive filesystems; +- race conditions between validation and use; +- the difference between a display path and canonical identity. + +`normalize()`, `resolveAlias()`, and `matchesGlob()` do not authorize a path. +Aliases can redirect imports and globs can match more than expected. Apply +allowed-root and file-kind checks after resolution. + +## ufo URLs and query strings + +### Current parsing and composition APIs + +```ts +import { + encodeParam, + filterQuery, + getQuery, + hasProtocol, + isScriptProtocol, + joinRelativeURL, + joinURL, + normalizeURL, + parseQuery, + parseURL, + resolveURL, + stringifyParsedURL, + stringifyQuery, + withBase, + withQuery, +} from "ufo"; +``` + +Use helpers according to the component being manipulated: + +- `encodeParam()` encodes a path parameter and also encodes `/`; +- `encodePath()` preserves path separator meaning; +- `encodeQueryKey()` additionally encodes `=`; +- `encodeQueryValue()` uses query-value rules, including space/plus behavior; +- `parseQuery()` returns string or string-array values and ignores + `__proto__` and `constructor` keys; +- `stringifyQuery()` repeats keys for arrays and omits `undefined` values; +- `withQuery()` merges existing query values with the supplied object, with + supplied keys replacing existing keys; +- `joinRelativeURL()` resolves `.` and `..` segments; +- `joinURL()` joins segments without relative traversal semantics; +- `resolveURL()` preserves a base URL's query and fragment while resolving + further segments according to ufo's rules. + +Example for a validated application query: + +```ts +const parsedQuery = QueryInputSchema.parse( + parseQuery(rawQuery), +); + +const requestPath = withQuery( + joinURL("/api", encodeParam(resourceId)), + { + cursor: parsedQuery.cursor, + limit: parsedQuery.limit, + }, +); +``` + +The schema still owns cardinality and types. `parseQuery<{ limit: string }>()` +does not turn repeated `limit` keys into a single value at runtime. + +### Security and identity + +ufo is permissive. `parseURL("example.com/path")` returns a pathname unless a +default protocol is supplied. Use native `URL` plus application policy when an +absolute network destination is required: + +```ts +const base = new URL(AllowedEndpointSchema.parse(config.endpoint)); +const target = new URL(joinURL(base.href, encodeParam(resourceId))); + +if (target.origin !== base.origin) { + throw new TypeError("Resolved endpoint changed origin"); +} +``` + +Then apply scheme, hostname, port, DNS/IP, redirect, credential, and private +network policy appropriate to the operation. An origin comparison alone is not +a complete SSRF defense. + +The 1.6.4 README shows `isScriptProtocol("javascript:alert(1)")`, but the +published implementation matches an exact protocol token ending in `:`. A +runtime smoke test produced `false` for that full URL and `true` for +`"javascript:"`. Use the parsed protocol: + +```ts +const parsed = parseURL(authoredURL); + +if (isScriptProtocol(parsed.protocol)) { + throw new TypeError(`Disallowed URL protocol: ${parsed.protocol}`); +} +``` + +Even this is a denylist for `blob:`, `data:`, `javascript:`, and `vbscript:`. +Use a positive scheme/origin allowlist for privileged requests or links. + +Do not normalize identity-bearing URLs casually: + +- `normalizeURL()` changes encoding and path presentation; +- `cleanDoubleSlashes()` can change embedded URL-like path data; +- `isSamePath()` ignores trailing-slash and encoding differences; +- `isEqual()` ignores leading slash, trailing slash, and encoding differences + unless strict comparison options are enabled; +- `decode()` returns the original string when decoding fails. + +These conveniences can be wrong for signatures, OAuth redirect matching, +cache keys, crawl deduplication, opaque user identifiers, or presigned URLs. +Preserve the original string and define canonicalization as a versioned product +contract before hashing, signing, or comparing. + +`$URL` and `createURL()` are deprecated in 1.6.4. Use native `URL` or +`parseURL()` as the declarations direct. + +## Complete integration patterns + +### Trusted TypeScript config in a monorepo CLI + +```text +invocation cwd + -> pkg-types returns workspace candidates and manifest evidence + -> repository instructions select the owning package/root + -> pathe resolves the explicit config under that root + -> lexical and canonical containment checks pass + -> c12 loads one required config with RC/package/dotenv/env disabled + -> custom jiti import evaluates the trusted TypeScript module + -> c12 inheritance is disabled or restricted by declared policy + -> every authored layer is schema-validated + -> defu custom merger resolves explicit field semantics + -> sparse-patch schema rejects loader metadata and operation objects + -> runtime schema applies defaults once + -> command receives runtime config plus config/layer provenance +``` + +Tests must cover a nested package invocation, an explicit missing file, a +config outside the root, a module with side effects, a bad default export, +array replacement, atomic union replacement, and both c12 lines if both remain +supported. + +### Human-authored JSONC config + +```text +read original text + -> confbox parseJSONC with error collection + -> reject every syntax error + -> authored schema validation + -> application mutation on a copy + -> runtime/sparse schema validation + -> show lossiness and exact diff + -> approved atomic write + -> public c12 resolver reload + -> compare effective config and provenance +``` + +Because comments and trailing commas are not preserved, do not call this a +source-preserving edit. If preserving comments is a requirement, select an AST +editor that supports the exact format and verify it with fixtures. + +### Config-derived endpoint + +```text +c12/confbox/jiti yields unknown config value + -> URL schema accepts an absolute allowed scheme + -> native URL parses the authority + -> ufo encodes application path/query components + -> native URL resolves against the approved base + -> origin/network policy validates the result + -> original and canonical forms remain distinct + -> redacted diagnostic records the approved destination class +``` + +Never feed an unvalidated config URL into a fetch client merely because ufo +parsed or normalized it. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +| --- | --- | --- | +| Required config cannot be resolved | `configFileRequired` is true but the base/path or extension does not resolve | Log the redacted resolved candidate and inspect c12's `configFile`/`_configFile` for the installed version | +| TypeScript config loads on one runtime only | Native import succeeded in one environment while another needed jiti or a different syntax transform | Pin jiti, inspect `jitiOptions`/custom import, and execute both runtime paths | +| Config code runs twice | Module cache disabled, watch reload, or multiple jiti instances/loaders | Trace loader instance and cache ownership; make side effects idempotent or remove them | +| jiti debug lines bypass LogTape | `debug` or `JITI_DEBUG` enabled | Disable it or explicitly adapt/isolate output before claiming a sole transport | +| Unexpected `.toolrc` values | c12 generated an RC name because `rcFile` was not false | Make the RC source policy explicit and report provenance | +| Test/production values appear unexpectedly | `envName` defaulted from `NODE_ENV` | Pass `envName: false` or a validated explicit environment | +| Network or package installation occurs during config load | c12 `extends` reached giget or an install-enabled source | Disable extends/giget, pin presets, and audit source options before loading | +| Watch mode fails to start | c12 v4 optional `chokidar` peer is absent | Install/pin the peer only if watch is a supported capability | +| Watcher object is a promise or fields are undefined | `watchConfig()` was not awaited | Follow the installed declaration, not the faulty README snippet | +| Reload warning bypasses structured diagnostics | c12 watcher emitted internal `console.warn()` | Isolate/adapt the dependency or document and test the exception | +| Array contains inherited and authored entries | defu concatenated arrays | Add a field-specific replace/operation rule and a conflict test | +| Discriminated union contains fields from two variants | defu recursively merged an atomic union | Replace the complete union at its exact path | +| `null` did not clear a value | defu skips nullish higher-priority values | Add an explicit clear operation or choose a patch algebra that defines deletion | +| “Strict JSON” accepted `TRUE`, `NaN`, or `undefined` | `safeDestr()` was mistaken for exact JSON | Use exact JSON parsing and a runtime schema | +| A warning appears while parsing an untrusted scalar | loose destr dropped a suspicious key | Use strict parsing and map the error through the output owner | +| Malformed JSONC becomes `{}` or a partial object | confbox parse errors were not collected and rejected | Pass `errors`, reject non-empty results, and test malformed fixtures | +| Comments disappear after a config edit | confbox serialization is not a lossless AST round trip | Restore the original, disclose lossiness, or use a verified format-specific editor | +| Package metadata stays stale after mutation | pkg-types global/read cache was enabled | Use an invocation-scoped cache and invalidate the changed path | +| Package file is truncated after interruption | pkg-types direct write was treated as atomic | Use an application-owned temp-file, sync, rename, and recovery policy | +| Operation runs at the wrong monorepo root | `findWorkspaceDir()` heuristic found an outer marker | Reconcile markers with workspace membership, instructions, and requested scope | +| Path passes string containment but escapes through a symlink | pathe resolves lexical strings only | Canonicalize the existing root/parent with host filesystem APIs and address races | +| Windows path comparisons differ from display output | pathe normalized separators while another layer used native/canonical form | Select one internal identity and test drive, UNC, case, and separator behavior | +| `isScriptProtocol(fullURL)` returns false | ufo 1.6.4 expects the protocol token, despite its README example | Parse first and call `isScriptProtocol(parsed.protocol)`, then apply an allowlist | +| Signed URL or cache key changes after cleanup | ufo normalization changed encoding or slash identity | Preserve the original; use only the product's versioned canonicalization | + +## Testing and verification + +### Version and export verification + +Record the owning manifest and lockfile result, then inspect exact artifacts: + +```bash +npm view jiti version dist-tags +npm view c12 version dist-tags peerDependencies +npm view defu version +npm view destr version +npm view confbox version +npm view pkg-types version +npm view pathe version +npm view ufo version +``` + +For reproducible source inspection, use the versioned published tarball or a +matching immutable repository tag. Verify imports with the project's own +typechecker; do not infer exports from a website navigation list. + +### Resolver matrix + +Exercise the application resolver, not only each library: + +| Axis | Required cases | +| --- | --- | +| invocation | workspace root, nested package, outside repository, symlinked path | +| source | missing explicit file, TS module, JSON, malformed JSONC, YAML/TOML if supported | +| c12 policy | RC disabled/enabled, package field disabled/enabled, env disabled/named, extends disabled/local/remote-denied | +| jiti | native success, transform fallback, cache on/off, bad export, thrown side effect | +| merge | scalar conflict, nested object, replace/append/prepend array, atomic union, explicit clear | +| validation | bad authored layer, bad merged sparse patch, defaulted runtime config | +| mutation | dry run, rejected consent, validation failure, interrupted write, reload | +| output | stdout result remains clean, diagnostics redacted, dependency warnings accounted for | + +### Package-specific executable checks + +At the verified versions, smoke tests should prove at least: + +```text +defu({ items: ["high"] }, { items: ["low"] }) + -> { items: ["high", "low"] } + +defu({ value: null }, { value: 1 }) + -> { value: 1 } + +safeDestr("TRUE") + -> true + +parseJSONC(malformed, { errors }) + -> returns a value and fills errors; application rejects it + +withQuery("/x?a=1", { a: 2, b: "ok" }) + -> "/x?a=2&b=ok" + +isScriptProtocol("javascript:alert(1)") + -> false in ufo 1.6.4 + +isScriptProtocol(parseURL("javascript:alert(1)").protocol) + -> true in ufo 1.6.4 +``` + +These outputs were executed against the version-pinned npm packages while +writing this reference. Keep them as regression expectations only for the +pinned versions; rerun them when versions change. + +### Completion standard + +Do not report this subsystem verified until: + +1. installed versions and optional peers are known; +2. the public resolver passes the source/precedence matrix; +3. every loaded value crosses a runtime schema; +4. merge decisions pass field-specific conflict tests; +5. config edits pass recovery and reload tests; +6. path and URL policies pass adversarial cases; +7. dependency-originated stdout/stderr is observed and reconciled with the + CLI output contract; +8. the actual command uses the resolved config successfully. + +## Sources and freshness + +Verified on 2026-07-17 from primary published artifacts and attached source. + +- jiti 2.7.0: [official repository](https://github.com/unjs/jiti), + [published tarball](https://registry.npmjs.org/jiti/-/jiti-2.7.0.tgz). +- c12 4.0.0-beta.5: [official repository](https://github.com/unjs/c12), + [published tarball](https://registry.npmjs.org/c12/-/c12-4.0.0-beta.5.tgz). +- c12 3.3.4 comparison line: npm `3x` tag and + [published tarball](https://registry.npmjs.org/c12/-/c12-3.3.4.tgz). +- defu 6.1.7: [official repository](https://github.com/unjs/defu), + [published tarball](https://registry.npmjs.org/defu/-/defu-6.1.7.tgz). +- destr 2.0.5: [official repository](https://github.com/unjs/destr), + [published tarball](https://registry.npmjs.org/destr/-/destr-2.0.5.tgz). +- confbox 0.2.4: [official repository](https://github.com/unjs/confbox), + [published tarball](https://registry.npmjs.org/confbox/-/confbox-0.2.4.tgz). +- pkg-types 2.3.1: [official repository](https://github.com/unjs/pkg-types), + [published tarball](https://registry.npmjs.org/pkg-types/-/pkg-types-2.3.1.tgz). +- pathe 2.0.3: [official repository](https://github.com/unjs/pathe), + [published tarball](https://registry.npmjs.org/pathe/-/pathe-2.0.3.tgz). +- ufo 1.6.4: [official repository](https://github.com/unjs/ufo), + [published tarball](https://registry.npmjs.org/ufo/-/ufo-1.6.4.tgz). +- Attached `live-browser-cli(41).zip`: root `package.json` and `deno.jsonc`; + `packages/config/deno.json`; `packages/config/src/index.ts`; + `packages/config/src/merge.ts`; `packages/config/src/merge.test.ts`. +- Attached ClickHouse source under `kaiju-site-scope(17).zip`: + `libs/clickhouse/src/kit/node.ts` and `libs/clickhouse/src/kit/loader.ts` for + independently observed jiti/c12 loading and immediate schema validation. + +Freshness rules: + +- recheck c12 before every update while the selected v4 line remains beta; +- recheck package exports and declarations rather than relying on README code; +- rerun the ufo protocol smoke case until its README and runtime agree; +- rerun confbox malformed JSONC behavior and comment-loss fixtures on update; +- rerun jiti default-export, cache, and runtime-fallback tests on update; +- keep c12 3 and c12 4 expectations separate wherever both resolve in one + workspace; +- label any API not present in the installed declarations as unresolved rather + than reconstructing it from ecosystem familiarity. diff --git a/skills/build-clis/references/unjs.md b/skills/build-clis/references/unjs.md new file mode 100644 index 0000000..9450721 --- /dev/null +++ b/skills/build-clis/references/unjs.md @@ -0,0 +1,333 @@ +# Focused UnJS adapters for CLI applications + +## Contents + +- [Selection rule](#selection-rule) +- [Capability map](#capability-map) +- [Configuration cluster](#configuration-cluster) +- [Environment and path cluster](#environment-and-path-cluster) +- [HTTP and URL cluster](#http-and-url-cluster) +- [Storage, hashing, and hooks](#storage-hashing-and-hooks) +- [Package and build cluster](#package-and-build-cluster) +- [Documentation and release cluster](#documentation-and-release-cluster) +- [Alternative parser and reporter](#alternative-parser-and-reporter) +- [Integration sequences](#integration-sequences) +- [Testing and failure signatures](#testing-and-failure-signatures) +- [Sources and freshness](#sources-and-freshness) + +## Selection rule + +Treat UnJS as an ecosystem of focused packages, not a framework to install as a +bundle. Verify each package's current exports, runtime matrix, maintenance state, +and relationship to installed siblings. Assign one owner to every capability. + +Use an UnJS package when it replaces application-specific host plumbing behind +a clear contract. Do not replace domain schemas, durability semantics, risk +policy, output contracts, or lifecycle ownership with an ecosystem brand. + +After selecting a cluster, load the versioned implementation manual instead of +extrapolating from this map: + +- [runtime and configuration packages](unjs-runtime-config.md); +- [fetch, state, hashing, and hooks](unjs-fetch-state.md); +- [build, release, content, and project operations](unjs-build-release.md). + +The productionized CLI guidebook and its audit provide the capability map below. +The attached CLI proves c12, defu, jiti, and related transitive packages are +present; it does not prove every mapped package is installed or used. + +## Capability map + +| Package | Capability | Put behind | Important exclusion | +|---|---|---|---| +| c12 | Config discovery, layers, extends, environments, provenance, watch/update | `ConfigLoader` | Does not own domain defaults or trust policy | +| defu | Recursive default-style pair merging and custom merger | merge function | Default array behavior is not product policy | +| jiti | Runtime JavaScript/TypeScript module loading | config/module loader | Not a sandbox | +| rc9 | XDG-aware user RC read/write/update | `UserConfigStore` | Not project config discovery | +| std-env | CI/provider/debug/color/minimal-environment signals | `EnvironmentPolicy` | Does not replace per-stream TTY probes | +| pathe | Normalized cross-platform paths | `PathPolicy` | Does not define ownership/security | +| ofetch | Cross-runtime fetch, parsing, timeout, retry, interceptors | `HttpClient` | Does not decide idempotency or business retry policy | +| ufo | URL parsing, joining, normalization, query composition | URL boundary helper | Do not use normalization that changes domain identity silently | +| unstorage | Async key-value API, drivers, mounts, metadata, watch/snapshot/hydration where supported | `CheckpointStore` or `Cache` | Key-value persistence alone is not durability semantics | +| ohash | Deterministic hashing over canonicalizable inputs | `Fingerprint` | A hash is not an idempotency/recovery protocol | +| hookable | Typed/application hook mechanism | extension adapter | Do not create a plugin system without lifecycle/error policy | +| magicast | Source-preserving JS/TS config edits | `ConfigEditor` | Requires consent, validation, and atomic write | +| confbox | JSONC/YAML/TOML and structured config handling | format adapter | Does not replace authored schema | +| pkg-types | Package discovery, metadata, exports, version | `PackageMetadata` | Does not choose release truth automatically | +| nypm | Package-manager detection, dependency/script operations | `ProjectTooling` | Does not own the CLI's installer or uninstall policy | +| unbuild | Library/package builds, declarations, externals, development stubs | build pipeline | Not a standalone executable compiler | +| changelogen | Changelog/release-note generation | release workflow | Generated notes require review and version policy | +| automd | Generated Markdown regions | documentation workflow | Do not broad-format human-authored Markdown | +| giget | Template/repository fetching used in some ecosystem flows | trusted fetch adapter | Remote source needs pinning/integrity/offline policy | +| Citty | Lightweight CLI parser/runner | parser adapter | Alternative to Optique, not another layer | +| Consola | Console logger/reporter | output adapter | Alternative to LogTape, not a second transport | + +Confirm version-specific details from primary documentation or installed type +declarations. Several packages have much broader APIs than a CLI needs. + +## Configuration cluster + +Use c12, defu, and jiti together only through the application's config owner: + +```text +c12 discovers layers + -> jiti evaluates allowed JS/TS modules + -> application validates authored exports + -> defu-based explicit merger resolves layers + -> application validates sparse patch + -> runtime schema applies defaults +``` + +Use rc9 for a distinct user-level RC source. Do not make c12 and rc9 both read +and merge the same file. Record user RC below project and invocation sources +unless the product defines another order. + +Use magicast or confbox for consentful editing: + +- magicast when preserving JavaScript/TypeScript source structure matters; +- confbox for a supported structured format; +- c12 update/create hooks when their version and file type fit; +- authored schema validation after the edit regardless of writer. + +Never silently edit a shell profile or unrelated configuration. Show the target +and diff, support dry run, write atomically, reload, and report the path. + +## Environment and path cluster + +Use std-env to detect broad host context such as CI provider, debug convention, +color support, or minimal runtime. Continue to probe stdin, stdout, and stderr +independently because one stream can be redirected while another is a TTY. + +Combine std-env signals with explicit CLI policy: + +```text +explicit --color/--no-color + > NO_COLOR/FORCE_COLOR and product environment + > stream-specific TTY capability + > CI/provider convention + > safe no-color fallback +``` + +Use pathe inside host adapters for normalized path operations. Keep path policy +explicit for: + +- project-relative versus layer-relative paths; +- XDG config, data, cache, state, and log directories; +- Windows drive/UNC behavior; +- symlink and traversal checks; +- display path versus canonical filesystem identity. + +Do not let string normalization authorize a path. Validate containment and +ownership after resolution. + +## HTTP and URL cluster + +Use ofetch behind a structural client contract: + +```ts +export interface HttpClient { + request(options: { + readonly url: URL; + readonly method: string; + readonly signal: AbortSignal; + readonly timeoutMs: number; + readonly idempotencyKey?: string; + }): Promise; +} +``` + +Configure timeouts for every request. Retry only methods/operations whose +semantics are safe. ofetch's defaults are an implementation detail; define +allowed methods, attempts, backoff, deadline, and retryable errors in schemas. +Propagate the root abort signal. + +Use interceptors for correlation and structured LogTape events, not to hide +global mutable policy. Redact credentials and query parameters before logging. + +Use ufo at URL boundaries where its parsing/composition helpers materially +reduce mistakes. Preserve URL identity rules for signing, cache keys, crawls, +and user-provided opaque URLs. A “cleaner” URL can be a different resource. + +## Storage, hashing, and hooks + +Use unstorage as an adapter for caches or checkpoints when its driver matrix +matches the runtime. Define schema and semantics above it: + +```ts +export interface CheckpointStore { + read(key: string): Promise; + write(key: string, value: T): Promise; + remove(key: string): Promise; +} +``` + +For recovery, also define: + +- versioned checkpoint schema; +- committed versus attempted work; +- atomicity or compare-and-set requirements; +- lease ownership/expiry; +- idempotency and reconciliation; +- incompatible-request rejection; +- corruption and migration behavior. + +In-memory and browser drivers do not provide cross-process durability. A remote +driver does not automatically provide transactions. Match driver guarantees to +the workflow claim. + +Use ohash over normalized, schema-validated canonical data for plan/request +fingerprints. Include version and relevant input identities. Do not hash secrets +into public identifiers without analyzing leakage. Test key-order and runtime +stability for the exact version. + +Use hookable only when extension points are a product requirement. Define: + +- hook names and payload schemas; +- serial versus parallel execution; +- ordering and reentrancy; +- cancellation and timeout; +- exception aggregation; +- plugin trust and version compatibility; +- cleanup/unregistration. + +Callbacks without these rules create hidden control flow and make recovery hard. + +## Package and build cluster + +Use pkg-types to inspect the manifest and export graph that actually owns the +package. Use it for version/build provenance only after deciding whether root, +workspace package, tag, or injected revision is authoritative. + +Use nypm when a CLI operates on another project's dependencies or scripts: + +```text +detect package manager from manifests and locks + -> respect project choice and workspace root + -> show planned operation + -> invoke through injected subprocess and signal + -> stream redacted diagnostics + -> verify manifest/lock outcome +``` + +Do not install with npm in a pnpm/Yarn/Bun/Deno project merely because the CLI +itself runs on Node. Do not let nypm determine whether a dependency mutation was +authorized. + +Use unbuild for distributable libraries/packages when declaration output, +externals, multiple entrypoints, and development stubs fit. Verify: + +- clean public export map imports; +- type declarations and source maps; +- runtime dependencies versus externals; +- side effects and tree shaking; +- packed tarball contents; +- Deno/Node/browser conditions claimed. + +Use `deno compile` or the chosen standalone pipeline when the product needs a +self-contained executable. Use Node SEA only with its current limitations and +target verification. unbuild does not replace either executable pipeline. + +## Documentation and release cluster + +Use changelogen to assist release notes, not define compatibility. Compare +command names, flags, environment variables, config keys, exit codes, output +schemas, persisted state, and installation behavior before classifying a change. + +Use automd only for marked generated regions. Preserve human-authored wrapping, +tables, and code blocks outside those regions. The user's explicit Markdown +layout rule overrides broad formatter defaults. + +When fetching templates or presets through giget or a similar adapter: + +- pin a tag, commit, digest, or package version; +- define allowed hosts/protocols; +- respect proxies and offline mode; +- set timeouts and size limits; +- validate extracted paths and reject traversal; +- show overwritten files before applying; +- verify the generated project rather than trusting fetch success. + +## Alternative parser and reporter + +Citty can be the command owner when a lightweight grammar, nested/lazy commands, +aliases, generated usage, hooks, and plugins are sufficient. Choose it instead +of Optique after comparing requirements. Do not parse some subcommands with +Optique and others with Citty without an explicit stable boundary. + +Consola can be the output owner for applications that choose its reporter model. +Do not add it for spinners or friendly messages when LogTape already owns +transport. Build a LogTape formatter/sink or interaction renderer over the same +structured event instead. + +## Integration sequences + +### Project-aware install command + +```text +Optique parses package and dry-run terms + -> c12 loads project policy once + -> pkg-types locates workspace/package metadata + -> nypm detects the selected package manager + -> command creates an operation plan + -> LogTape transports plan/result and diagnostics + -> authorized apply invokes package manager with root abort signal + -> manifest and lockfile changes are verified +``` + +### Recoverable network import + +```text +Optique parses semantic timeout and resume selection + -> c12 resolves endpoint and cache policy + -> ufo validates/composes URLs + -> ofetch applies deadline, signal, and safe retry policy + -> unstorage persists versioned committed checkpoints + -> ohash binds checkpoint to normalized request/input identity + -> LogTape records redacted progress and exact stable result +``` + +### Consentful config initialization + +```text +Optique/Clack gathers missing non-secret values only on a TTY + -> c12 selects the owned config target + -> magicast or confbox builds a source-preserving edit + -> dry-run result shows the diff + -> atomic writer applies authorized change + -> c12 reloads the file + -> authored and runtime schemas validate it +``` + +## Testing and failure signatures + +Use real integration fixtures for package-manager locks, config formats, fetch +failures, storage drivers, and packed artifacts. Mocking every package at the +adapter boundary can prove domain isolation but not ecosystem compatibility. + +| Signature | Likely ownership error | Verification | +|---|---|---| +| Two different config values by command | c12/rc9/Optique env overlap | Trace one source algebra and winners | +| CI receives color or prompt | std-env signal replaced explicit per-stream policy | Run with redirected streams and CI env | +| POST executes twice | ofetch retries without domain idempotency policy | Capture request attempts and method rules | +| Resume accepts different inputs | ohash fingerprint omits normalized identity/version | Mutate one material field and assert rejection | +| “Durable” run disappears after restart | unstorage driver is memory/local-only | Kill process and resume from claimed boundary | +| Generated config loses comments | structured serializer used on TS/JSONC source | Compare source-preserving edit and diff | +| Wrong package manager changes lockfile | nypm detection/workspace root unchecked | Exercise npm/pnpm/Yarn/Bun/Deno fixtures | +| Package works in repo but not consumer | unbuild/pkg-types export or packed-file drift | Install packed tarball in a clean project | +| Static binary lacks commands/templates | dynamic graph invisible to compiler | Run every installed command from artifact | +| Markdown diff rewrites guidebook | automd/formatter touched unowned regions | Restrict generation markers and inspect diff | +| Logs split between two policies | Consola added beside LogTape | Search output imports and test exact routes | +| Unknown command changes meaning later | catch-all/lazy parser accepts abbreviations | Test explicit command registry and aliases | + +## Sources and freshness + +- Normative ecosystem audit: `cli-guidelines-audit-and-expansion(1).md`, reviewed 2026-07-17. +- Normative architecture: `productionized-cli-pattern-guidebook-v1.1(1).md`, reviewed 2026-07-17. +- Observed dependency graph: `live-browser-cli(41).zip/deno.lock` and package manifests, reviewed 2026-07-17. +- Official ecosystem index: , discovery pointer for maintained projects and current package documentation. +- Official organization source: , discovery pointer for individual package repositories and release histories. + +Freshness status: the capability assignments are grounded in the attached +guidebooks. They are not proof that every package is installed, supports every +runtime, or retains the same API. Verify the selected package's official docs, +exports, version, driver guarantees, and maintenance status before adoption. diff --git a/skills/build-data/SKILL.md b/skills/build-data/SKILL.md index 45c37cc..50ebdb0 100644 --- a/skills/build-data/SKILL.md +++ b/skills/build-data/SKILL.md @@ -47,8 +47,13 @@ ORM-shaped API or a repository README that contradicts the deployed query path. decision model. - [postgres-drizzle.md](references/postgres-drizzle.md): transactional schemas, Drizzle, migrations, drivers, and resource lifetime. +- [drizzle-architecture.md](references/drizzle-architecture.md): load when reviewing + Drizzle internals, dialects, drivers, sessions, prepared queries, result mapping, + ORM/Kit boundaries, or designing a new dialect. - [clickhouse.md](references/clickhouse.md): analytics, MergeTree design, ingestion, deduplication, mutation, and custom adapters. +- [clickhouse-adapter.md](references/clickhouse-adapter.md): load when implementing, + auditing, publishing, or extending the Kaiju custom Drizzle-like ClickHouse adapter. - [projections.md](references/projections.md): Typesense, QLever/Blazegraph, synchronization, rebuild, and reconciliation. - [artifacts.md](references/artifacts.md): JSONL, Parquet, raw evidence, diff --git a/skills/build-data/references/artifacts.md b/skills/build-data/references/artifacts.md index 612cfda..1e7c45a 100644 --- a/skills/build-data/references/artifacts.md +++ b/skills/build-data/references/artifacts.md @@ -1,31 +1,308 @@ -# Data artifacts and staging +# Data artifacts, manifests, and staging -## JSONL +Use this reference when a pipeline writes JSONL, Parquet, RDF, CSV, snapshots, profiles, or other files that must survive process loss or feed another system. A file extension is not a contract. The artifact contract includes identity, schema, commit protocol, provenance, reader compatibility, retention, and recovery. -Use JSONL for streaming, appendable, line-oriented evidence and interchange. -Define one record schema/version per line, newline escaping through serialization, -compression, file/run naming, manifests, checksums, and partial-file handling. +## Contents -## Parquet +- Authority and lifecycle +- Artifact identity and directory layout +- JSONL contracts +- Parquet contracts +- Raw evidence and derived records +- Completion manifests +- Atomic publication and resume +- Configuration model +- Integration sequence +- Failure and recovery +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -Use Parquet for typed columnar batches and interoperable analytical staging. -Define schema evolution, nullability, row-group size, partition layout, -compression, dictionary behavior, and reader compatibility. +## Authority and lifecycle + +Classify every artifact before choosing its format: + +| Class | Authority | Required recovery path | +|---|---|---| +| Raw capture | Immutable evidence of what a source returned | Reparse with a newer parser without refetching | +| Normalized staging | Derived, versioned interchange | Regenerate from raw capture and parser version | +| Transfer artifact | Contract between independently deployed systems | Compatibility test with every supported reader | +| Checkpoint | Authority for committed progress only | Reject or migrate incompatible state | +| Projection input | Rebuild source for a search, graph, or analytical sink | Replay deterministically and reconcile | +| Report/export | User-facing result | Reproduce from named source and query versions where required | + +Do not let a convenient staging file silently become the source of truth. Record which upstream object, transaction, or retrieval produced it and whether it is replaceable. + +The retained PopModern code demonstrates both the useful intent and the incomplete implementation. It writes raw MediaWiki pages, daily staging JSONL, Parquet, RDF, Typesense documents, profiles, run-state YAML, and metrics. However, many writes are best-effort, several exceptions are swallowed, daily JSONL is appended across runs, and the run state contains counts rather than committed artifact identities. Treat that code as counterexample evidence for why a manifest is necessary, not as a production commit protocol to copy. + +## Artifact identity and directory layout + +An artifact identity should be stable enough to answer “is this the same input and transformation?” without trusting a filename: + +```ts +interface ArtifactIdentity { + runId: string + artifactId: string + kind: 'raw' | 'normalized' | 'checkpoint' | 'projection-input' | 'export' + source: { + system: string + objectId: string + version?: string + etag?: string + retrievedAt: string + } + schema: { name: string; version: string } + producer: { name: string; version: string; configDigest: string } + content: { sha256: string; bytes: number; records?: number } +} +``` + +Prefer run-isolated directories over date-only append targets: + +```text +artifacts/ + mediawiki/ + 2026-07-17T142233Z_01J.../ + raw/ + normalized/ + rejected/ + profile/ + manifest.in-progress.json + manifest.json +``` + +A date is not a run identity. Two retries on the same date must not ambiguously append into one logical artifact unless the append log has its own transactional framing and committed offsets. + +## JSONL contracts + +JSONL is useful for line-oriented replay, streaming exchange, and append-only evidence. Define all of these: + +- UTF-8 encoding and exactly one serialized JSON value per terminated line; +- object envelope and schema version per record or per artifact manifest; +- whether blank lines are forbidden, ignored, or meaningful; +- maximum line size and decompression limits; +- ordering and duplicate rules; +- newline and control-character handling through the JSON serializer, never manual interpolation; +- compression format and whether concatenated compressed members are supported; +- corrupt/truncated final-line policy; +- committed byte offset or record identity used for resume; +- file checksum and, for append logs, segment checksums. + +Use an envelope when provenance or multiple record types must travel with each item: + +```json +{"schema":"mediawiki.toy.normalized","version":2,"record_id":"tfwiki:123:456","source":{"page_id":123,"revision_id":456},"data":{"name":"Example"}} +``` + +Do not infer a production table schema by scanning the first thousand JSONL lines. The retained PopModern `pg_loader.py` does exactly that, flattens top-level keys, stringifies through CSV, skips invalid JSON, and interpolates table and column identifiers. Its own docstring says it is unsuitable for nested production structures. A production loader requires an explicit schema, safe identifier ownership, typed conversion, a rejected-record channel, and a count/hash reconciliation. + +## Parquet contracts + +Parquet is useful for typed columnar batches, analytical staging, and interoperability. Specify: + +- explicit Arrow/Parquet schema rather than relying on whichever records appear first; +- nullability and distinction between absent, null, empty, and defaulted values; +- logical types for timestamps, dates, decimals, UUID-like values, and nested fields; +- timestamp unit and timezone semantics; +- decimal precision and scale; +- dictionary encoding choices for bounded-cardinality fields; +- compression codec and supported reader versions; +- target row-group size based on scan and memory behavior; +- partition keys based on common pruning, without creating tiny partitions; +- field-add, field-remove, rename, widening, and incompatible-change policy; +- empty artifact behavior and whether an empty schema is valid. + +The retained `write_records_to_parquet()` materializes `list(records)` and derives a table from the resulting Python objects. Its comment correctly warns that large streams need chunking. Do not call that helper bounded merely because the surrounding pipeline is iterative. Use a `ParquetWriter` or equivalent and flush bounded record batches: + +```py +writer = pq.ParquetWriter(temp_path, declared_schema, compression="zstd") +try: + for batch in bounded_batches(records, 10_000): + table = pa.Table.from_pylist(batch, schema=declared_schema) + writer.write_table(table, row_group_size=10_000) +finally: + writer.close() +``` + +Validate the resulting metadata and read it with every supported consumer. A successful writer call does not prove schema compatibility. ## Raw evidence and derived records -Retain source provenance, retrieval time, content identity, parser/version, and -license/privacy constraints. Raw, normalized, derived, and projected forms should -be distinguishable and reproducible. +Raw capture should preserve enough context to replay and audit: + +- retrieval timestamp and source endpoint; +- source object identity, revision, ETag, cursor, or version; +- request parameters that affect the response; +- response status and relevant headers; +- content digest and byte count; +- license, privacy, retention, and redaction classification; +- collection error if no payload was obtained. + +Normalized records should add: + +- deterministic record identity; +- parser, cleaner, mapper, ontology, and schema versions; +- raw artifact and source-object references; +- normalization warnings and rejected-field reasons; +- event time separately from processing time; +- transformation configuration digest. + +Never overwrite raw capture during normalization. Never call a normalized field “source” if it actually identifies the parser or current website. + +## Completion manifests + +The manifest is the durable statement of what committed. A useful model is: + +```ts +interface RunManifest { + schemaVersion: 1 + runId: string + state: 'in-progress' | 'complete' | 'failed' | 'cancelled' + startedAt: string + finishedAt?: string + inputs: ArtifactIdentity[] + stages: Array<{ + name: string + version: string + state: 'pending' | 'running' | 'complete' | 'failed' | 'skipped' + checkpoint?: { inputIdentity: string; committedOffset: string } + outputs: ArtifactIdentity[] + accepted: number + rejected: number + errors: number + }> + sinks: Array<{ + name: string + required: boolean + state: 'pending' | 'complete' | 'failed' + receipt?: Record + }> +} +``` + +Counts alone are not completion evidence. Include output identity, checksum, schema version, and sink receipt/checkpoint. A global `errors: 3` does not reveal which records failed, whether a required sink failed, or whether retry is safe. + +## Atomic publication and resume + +For local filesystems: + +1. Write to a run-scoped temporary path on the same filesystem. +2. Flush the application writer. +3. Close it and fsync the file when durability requires it. +4. Validate content, counts, and checksum. +5. Atomically rename to the final artifact path. +6. Atomically publish the final manifest last. + +For object storage, a rename may be copy-plus-delete and not atomic. Write immutable content-addressed or run-scoped objects and publish a small final manifest/pointer after validation. Readers only consume objects named by a complete manifest. + +A checkpoint must describe committed output. Advancing the input cursor before the file segment or sink batch commits can lose data. Advancing it afterward may replay the last unit after a crash, so the sink must accept deterministic identities or another idempotency mechanism. + +On resume: + +- verify input identity and producer/schema/config versions; +- validate the last committed artifact segment and checksum; +- reject ambiguous or incompatible checkpoints; +- replay from the last committed boundary; +- reconcile outputs before marking the resumed run complete. + +## Configuration model + +Keep artifact configuration explicit and validated. The consumer may choose Zod, Standard Schema, Effect Config, Pydantic, or another owner; this reference does not require one validator. + +```yaml +artifacts: + root: ./var/artifacts + run_id: auto + raw: + retain_days: 90 + compression: gzip + jsonl: + schema: mediawiki.toy.normalized + version: 2 + max_line_bytes: 1048576 + parquet: + compression: zstd + rows_per_group: 10000 + schema_file: schemas/toy.arrow.json + publication: + required_fsync: false + manifest_version: 1 +``` + +Record the resolved configuration digest in the manifest. Redact secrets before recording configuration. + +## Integration sequence + +```text +discover source version + -> create run identity and in-progress manifest + -> capture immutable raw evidence + -> validate/decode into bounded batches + -> normalize with explicit schema and provenance + -> quarantine rejected records + -> write staged temporary artifacts + -> close, validate, checksum, and publish artifacts + -> commit sink receipts and checkpoints + -> publish final complete manifest +``` + +If a downstream Typesense, QLever, ClickHouse, or PostgreSQL load is required, the manifest remains incomplete until the required sink receipt and reconciliation succeed. + +## Failure and recovery + +| Failure | Unsafe behavior | Recovery contract | +|---|---|---| +| Process dies mid-JSONL line | Append from attempted input offset | Truncate/discard uncommitted segment and resume from committed identity | +| Parquet writer dies before footer | Publish unreadable file | Keep temporary name; final manifest never references it | +| Schema changes during retry | Coerce old checkpoint silently | Reject or run a named checkpoint/schema migration | +| Invalid source record | Broad catch and continue | Quarantine raw identity, safe reason, stage, and retry disposition | +| Optional profile fails | Hide failure in global success | Mark optional stage failed and keep run complete only by explicit policy | +| Required projection load fails | Print error then “complete” | Keep run incomplete and persist retryable sink state | +| Daily file already exists | Append another run ambiguously | Use run identity or an append-log segment protocol | +| Checksum mismatch | Reprocess downstream anyway | Quarantine artifact and rebuild from its authoritative input | + +## Test matrix + +Test at least: + +- zero records, one record, and a batch boundary plus one; +- embedded newlines, Unicode normalization, very large values, and invalid encoding; +- truncated JSONL final line and corrupt middle line; +- stable Parquet schema with all-null early batches; +- decimals, timestamps, nested values, and reader round trips; +- process interruption before close, after close, before rename, and before manifest publication; +- duplicate delivery and replay of the last committed batch; +- schema/config/source-version mismatch on resume; +- required versus optional sink failure; +- disk full, permission denied, and object-store timeout; +- memory ceiling against a synthetic source larger than RAM; +- artifact retention without deleting objects still named by a live manifest. + +## Executable verification + +Adapt commands to the repository and installed tools: + +```bash +jq -c . artifacts/run/normalized/*.jsonl >/dev/null +wc -l artifacts/run/normalized/*.jsonl +sha256sum -c artifacts/run/checksums.sha256 +``` + +Use PyArrow or the chosen reader to assert the declared schema, row groups, counts, statistics, and a full scan of test artifacts. Then run the real downstream loader against a disposable store and compare accepted, rejected, and projected identities with the final manifest. + +Verification is incomplete until an interruption test proves that no final manifest names a partial file and resume does not skip a committed record. -## Completion manifest +## Deliberate exclusions -Record run and input identities, schema versions, stage outputs, per-sink status, -counts, rejected items, checkpoints, checksums, and completion state. Write the -manifest atomically or use an explicit in-progress/final protocol. +- Do not require JSONL or Parquet when a database transaction or object is the actual appropriate boundary. +- Do not prescribe Zod, LogTape, Effect, Python, Deno, or a particular storage provider. Preserve the consumer's chosen schema, logging, runtime, and storage owners. +- Do not infer schema from sample records for a production load. +- Do not treat a date-based filename, file existence, or non-zero size as identity or completion. +- Do not swallow required-stage errors to keep a batch moving. +- Do not promise exactly-once processing from a checkpoint alone. +- Do not publish temporary or partially validated artifacts. -## Bounded processing +## Sources and freshness -Stream and batch. Do not accumulate all records for profiling or deduplication -without a measured bound. Test large synthetic input, partial writes, truncated -files, resume, duplicate records, schema changes, and corrupt artifacts. +Grounded in the retained PopModern `DATA_PIPELINE.md`, `infra/mediawiki_ingest/etl_runner.py`, `infra/importer/utils/{parquet.py,pg_loader.py,profiling.py,state.py}`, the general importer and its JSONL/N-Triples sinks, reviewed 2026-07-17. The codebase is observational and includes acknowledged prototypes; its swallowed errors, materialized Parquet writes, inferred PostgreSQL loader, and non-transactional run state are counterexamples, not endorsed APIs. Format and storage-provider behavior must be rechecked against the versions installed by the consumer. diff --git a/skills/build-data/references/clickhouse-adapter.md b/skills/build-data/references/clickhouse-adapter.md new file mode 100644 index 0000000..44c6780 --- /dev/null +++ b/skills/build-data/references/clickhouse-adapter.md @@ -0,0 +1,302 @@ +# Custom ClickHouse Drizzle-like adapter design + +## Contents + +- Status and evidence +- Capability matrix +- Module architecture +- Schema DSL +- Query AST and dialect +- Driver, session, and results +- Writes and mutations +- Migration and seed system +- Public API shape +- Unsupported semantics +- Test matrix +- Delivery sequence +- Sources and freshness + +## Status and evidence + +The uploaded Kaiju `libs/clickhouse` package is an inspectable custom adapter, not a verified package published for general use. Its source implements native ClickHouse columns, engine/table metadata, a Drizzle-SQL-based dialect, select/insert/mutation builders, a session and database facade, migration snapshots/diffs/generation/application, seed execution, an Optique CLI, and live integration tests. + +Do not invent an import specifier for it. Reuse the architecture only after confirming the consuming repository contains or publishes the package. + +## Capability matrix + +Use an explicit matrix in the package documentation and release gates: + +| Surface | Uploaded implementation | Required production proof | +|---|---|---| +| native table/columns | implemented | DDL and round-trip breadth | +| MergeTree engine helpers | implemented | server-version grammar tests | +| settings, codecs, TTL, indexes, projections | implemented | snapshot/diff/live DDL | +| select/CTE/joins/sets | implemented | SQL goldens and live results | +| `PREWHERE`, `FINAL`, `SAMPLE`, `QUALIFY`, totals, limit-by, settings | implemented | version matrix | +| prepared placeholders | implemented over compiled queries | typed binding and repeated execution | +| streaming iterator | implemented | early-return disposal/backpressure | +| values insert | implemented through native client | batch/type/default behavior | +| insert-select | implemented | live integration | +| `ALTER UPDATE/DELETE` mutation | implemented and requires `WHERE` | async completion/error behavior | +| multi-statement transaction | explicitly rejected | rejection test remains | +| relational query API | not established | do not advertise | +| FK/unique OLTP constraints | not ClickHouse semantics | do not emulate in types | +| `RETURNING` parity | not established | do not advertise | +| snapshot/diff/generate | implemented | transition/property tests | +| migration apply/history | implemented | interrupted-file repair | +| introspection/push/studio | not established | mark unsupported | +| seed journal/hash | implemented | crash/retry/idempotency | + +## Module architecture + +Recommended ownership: + +```text +schema/ + columns.ts native builders, encode/decode, DDL metadata + engines.ts structured engine arguments + table.ts ClickHouse table identity and physical config + ddl.ts CREATE TABLE renderer +params.ts bound value + ClickHouse type identity +dialect.ts Drizzle SQL AST -> ClickHouse SQL +session.ts prepare/execute/iterate + result mapping +result.ts ordered selection and decoder mapping +select.ts typed SELECT builder +insert.ts values and insert-select +mutation.ts ALTER UPDATE/DELETE with safety guard +db.ts public facade + unsafe escape hatch + close +migrations/ snapshot, diff, types, SQL splitting, runner +kit/ config loader and generation filesystem adapter +seeds/ seed definition, hash/history, runner +cli/ generate/migrate/seed/config commands and man page +``` + +Keep the native client behind a structural `ClickHouseClientLike` interface so unit tests do not require a server. Avoid exporting Drizzle internal compiler types as the adapter's public contract where a small structural interface suffices. + +## Schema DSL + +The uploaded package attaches a ClickHouse-specific table config to a Drizzle `Table` subclass and uses native `ColumnBuilder`/`Column` subclasses. The table config covers: + +- engine; +- `PARTITION BY`, `ORDER BY`, `PRIMARY KEY`, and `SAMPLE BY` expressions; +- TTL actions; +- data-skipping indexes; +- projections; +- engine settings and comments; +- database and cluster. + +Use typed engine arguments: + +```ts +type EngineArgument = + | { kind: "identifier"; value: string } + | { kind: "string"; value: string } + | { kind: "number"; value: number } + | { kind: "expression"; value: string }; +``` + +This prevents a replica path string, a version column identifier, a number, and `cityHash64(id)` from sharing an unsafe string-rendering rule. Keep `unsafeEngine()` or raw expression entrypoints visibly unsafe and limited to trusted static fragments. + +Example design, using names from the uploaded local source only: + +```ts +const events = clickhouseTable( + "events", + { + id: chUInt64("id"), + tenantId: chUUID("tenant_id"), + occurredAt: chDateTime64("occurred_at", { precision: 3, timezone: "UTC" }), + kind: chLowCardinality("kind", chString()), + }, + (column) => ({ + engine: mergeTree(), + partitionBy: sql`toYYYYMM(${column.occurredAt})`, + orderBy: [column.tenantId, column.occurredAt, column.id], + }), +); +``` + +Verify actual factory signatures before copying this example: the source may evolve. + +Column coverage should be grouped and tested by behavior: + +- integer widths and signedness, big integer mapping; +- floating, Decimal, Boolean; +- String, FixedString, UUID; +- Date, Date32, DateTime, DateTime64/timezone; +- Enum8/Enum16; +- LowCardinality, Nullable; +- Array, Map, Tuple, Nested; +- IPv4/IPv6; +- JSON/Object/Variant where server support permits; +- aggregate-function state types; +- aliases, materialized/default expressions, codecs, comments, statistics. + +## Query AST and dialect + +Build on Drizzle's SQL AST, not a second ad hoc expression language. The dialect must compile ClickHouse clause order exactly: + +```text +WITH -> SELECT -> FROM -> SAMPLE -> JOIN/ARRAY JOIN -> PREWHERE -> WHERE +-> GROUP BY -> HAVING -> QUALIFY -> ORDER BY -> LIMIT BY -> LIMIT/OFFSET +-> SETTINGS -> FORMAT +``` + +Keep format ownership in the session when the result mapper requires `JSONCompactEachRow`; avoid allowing user `FORMAT` to invalidate decoding accidentally. + +Parameterize values with ClickHouse's typed query parameters. Preserve explicit type identity for placeholders and column-bound parameters. Reject unsafe inferred types when ambiguity would change semantics. + +Test the dialect independently of transport: + +```ts +const compiled = query + .prewhere(eq(events.tenantId, sql.placeholder("tenant"))) + .where(gte(events.occurredAt, from)) + .orderBy(desc(events.occurredAt), desc(events.id)) + .limit(100) + .settings({ max_execution_time: 10 }) + .toSQL(); +``` + +Assert exact SQL, named parameter values, and ClickHouse parameter types. + +## Driver, session, and results + +Define four native operations: + +- `query({ query, format, query_params, clickhouse_settings, query_id, abort_signal })`; +- `command(...)` for DDL/mutations; +- `insert({ table, values, format, ... })` for batches; +- `close()` for lifetime. + +The uploaded structural interface also models a result set with `json()`, `stream()`, and `close()`. The adapter session should close it after array materialization and in an iterator's `finally` path. + +Use positional rows such as `JSONCompactEachRow` only when `orderSelectedFields(...)` and result positions are deterministic. Apply each selected column's decoder. For outer joins, return a nested joined object as `null` when every selected field for that joined table is null; do not leave a misleading object full of null values. + +The database factory should return both the typed facade and reachable native client ownership. Logger hooks should receive SQL and parameters but redact secrets and high-volume payloads. + +## Writes and mutations + +Values inserts should: + +- require at least one row; +- map property names to physical columns; +- run column encoders; +- omit server-default columns when absent; +- reject runtime SQL defaults the native insert format cannot express; +- avoid serializing `undefined` as a value accidentally; +- support request settings, query IDs, and cancellation; +- encourage batches and idempotency tokens. + +Insert-select compiles through the dialect and executes as a command. It is not the same transport as `client.insert(values)`. + +Mutation builders must require `WHERE` before execution. Accept only known target columns, encode assignments through those columns, and expose settings such as synchronous mutation waiting deliberately. + +Do not add `update()`/`delete()` names that imply immediate row-level OLTP semantics without documenting that they issue ClickHouse mutations. + +## Migration and seed system + +The uploaded design snapshots physical table metadata and diffs named columns, indexes, projections, TTL, settings, and layout keys. It classifies operations: + +- `safe`: additive/retryable DDL; +- `caution`: type/index/projection/TTL work needing explicit acceptance; +- `destructive`: data/object removal; +- `manual`: physical layout changes requiring replacement/backfill/swap. + +Generation writes a pending metadata record before atomically installing SQL, snapshot, and journal. Preserve this crash-recovery protocol. Never advance the snapshot when risk gates reject the plan. + +The migration runner: + +- sorts and de-duplicates identifiers; +- hashes normalized statements; +- rejects applied-ID hash mismatch by default; +- executes statements one at a time with stable query-ID prefixes; +- records successful or failed attempts; +- cannot roll back prior statements in the file. + +Seeds need their own identifiers/hashes/history. Make callbacks idempotent or rely on a proven ClickHouse deduplication boundary. A failed seed may have written data before the process died. + +## Public API shape + +Export by capability, not internal file layout: + +```ts +// runtime +export { drizzleClickHouse, alias, bindValue } from "./src/index.ts"; +export type { ClickHouseDatabase, ClickHouseQueryOptions } from "./src/index.ts"; + +// schema +export { clickhouseTable, mergeTree, replacingMergeTree } from "./src/schema/index.ts"; + +// migration generation/application +export { defineClickHouseConfig, generateClickHouseMigration } from "./src/kit/index.ts"; +export { migrateClickHouse } from "./src/migrations/index.ts"; +``` + +These paths illustrate ownership. Use the actual package export map, not deep imports, in consumers. + +## Unsupported semantics + +Reject or clearly omit: + +- transactional ORM sessions and multi-statement rollback; +- foreign-key or unique-constraint enforcement not provided by ClickHouse; +- PostgreSQL/MySQL conflict and returning clauses; +- relational-query API unless fully implemented; +- implicit row-by-row save/delete methods; +- schema introspection, push, Studio, or Kit compatibility not implemented; +- automatic migration of engine/sorting/partition keys; +- exactly-once claims spanning external effects; +- server features not covered by the supported version matrix. + +An explicit throw is a feature. Silent fallback to unsafe SQL is not. + +## Test matrix + +Unit and property tests: + +- every engine argument renderer and injection rejection; +- every column type DDL plus encode/decode; +- table DDL for partition/order/primary/sample/TTL/index/projection/settings; +- SQL clause order and parameter types; +- aliases, CTEs, joins, set operations, partial/nested selections; +- prepared execution and result mapping; +- iterator early close; +- empty/mixed/default inserts; +- mutation missing-`WHERE` rejection; +- snapshot determinism and diff transitions; +- SQL statement splitting across strings/comments; +- pending-generation recovery; +- migration/seed hash mismatch and failed history; +- explicit transaction rejection and client close. + +Live container tests: + +- create all supported DDL against minimum and target servers; +- values insert, insert-select, select, join, aggregation, and prepared query; +- timestamps/timezones, UInt64/Decimal, nested types; +- synchronous mutation and system-table observation; +- async insert/dedup/retry behavior; +- materialized view and projection use; +- clean migration, representative upgrade, interrupted migration repair; +- restart persistence and client shutdown. + +## Delivery sequence + +1. Publish a capability matrix and supported server/Drizzle/client versions. +2. Stabilize native client and parameter interfaces. +3. Complete schema/DDL breadth with container tests. +4. Stabilize select/result mapping and streaming disposal. +5. Stabilize inserts and bounded mutations. +6. Freeze snapshot schema version and build exhaustive diff tests. +7. Harden generation/history/crash recovery. +8. Add seeds, CLI, man/completion, and configuration tracing. +9. Run compatibility consumers outside the monorepo. +10. Only then declare a stable public package. + +## Sources and freshness + +- Attachment: `kaiju-site-scope(17).zip/libs/clickhouse/src` and `tests`, inspected 2026-07-17. This is the direct source for the described implementation. +- Primary connected systems: [ClickHouse documentation](https://clickhouse.com/docs/) and [Drizzle ORM documentation](https://orm.drizzle.team/docs/overview), verified 2026-07-17 for public database and ORM contracts. + +The adapter's package identity, publication status, compatibility range, and external API are unverified and version-sensitive. Do not invent an import specifier or claim parity with Drizzle dialects that have upstream support. diff --git a/skills/build-data/references/clickhouse.md b/skills/build-data/references/clickhouse.md index b5324d3..64f5d75 100644 --- a/skills/build-data/references/clickhouse.md +++ b/skills/build-data/references/clickhouse.md @@ -1,36 +1,282 @@ -# ClickHouse analytics and custom adapters +# ClickHouse storage, ingestion, and operations -## Table design +## Contents -Choose engine, ordering key, primary-key index, partitioning, retention/TTL, -compression/codecs, and projections/materialized views from query and ingestion -patterns. An ordering key is a physical access/deduplication decision, not an -OLTP uniqueness constraint. +- Mental model +- MergeTree-family selection +- Sorting, primary index, partitions, and granules +- Inserts, batching, and deduplication +- Corrections, mutations, and consistency +- Materialized views and projections +- TTL and storage lifecycle +- Replication and distribution +- Observability and diagnosis +- Schema evolution and verification +- Failure signatures +- Sources and freshness -Define event identity, version/correction semantics, late arrival, duplicate -handling, batch size, async insert behavior, and mutation/deletion cost. +## Mental model -## PostgreSQL relationship +ClickHouse is an analytical, column-oriented database. A MergeTree-family table stores inserted rows in immutable data parts. Each part is sorted by the table's `ORDER BY` expression, split into granules, compressed by column, and merged with other compatible parts in the background. An insert creates parts; it does not update a B-tree one row at a time. -PostgreSQL can own organizations, billing, workflow state, and other transactional -records while ClickHouse owns observation analytics. Define the durable event or -outbox/change-capture path, projection lag, retry, backfill, and reconciliation. -Avoid dual writes without a repair contract. +Keep these terms distinct: -## Custom Drizzle-shaped adapter +| Term | Operational meaning | Common mistake | +|---|---|---| +| `ORDER BY` | Physical sort key and main data-skipping design | Treating it as display order or uniqueness | +| `PRIMARY KEY` | Sparse index expression; defaults to the sorting key | Expecting an OLTP uniqueness constraint | +| `PARTITION BY` | Coarse lifecycle and pruning boundary | Partitioning by a high-cardinality identifier | +| part | Immutable sorted unit written and merged | Assuming a row is updated in place | +| granule | Smallest block selected through the sparse index | Expecting point lookup precision | +| mark | Index/offset entry for a granule | Assuming one index entry per row | +| merge | Background consolidation and engine-specific reconciliation | Treating it as a synchronous commit step | +| mutation | Asynchronous rewrite of affected parts | Using it as an OLTP update loop | -Inspect actual source and exports for dialect, table/column builders, SQL -generation, session, driver, query, insert, migrator, and Kit integration. Prove -each capability independently. A MySQL-shaped session or familiar query builder -does not establish ClickHouse transactions, constraints, returning behavior, or -migration semantics. +The default index granularity is commonly 8,192 rows, but inspect the deployed server and table settings. Sparse indexing is efficient only when filters align with the leading sorting-key expressions. -Do not invent private adapter APIs. If source is unavailable, state the exact -inspection required and use the native client/query surface supported by evidence. +## MergeTree-family selection -## Verification +Choose the engine from the correction and aggregation model, not from the table name. -Capture generated DDL/SQL, create a clean database, apply schema, seed, batch -insert, query representative filters/aggregates, inject duplicates and late -records, exercise retention/corrections, and test failure/retry/backfill. Measure -with representative cardinality and ordering, not a one-row smoke test. +| Engine | Use when | Required correctness rule | +|---|---|---| +| `MergeTree` | Append-only facts or events | Duplicates remain unless ingestion prevents them | +| `ReplacingMergeTree(version[, deleted])` | New versions of a logical row arrive append-only | Query-time `FINAL`, `argMax`, or version-aware view is needed until merges reconcile | +| `SummingMergeTree(columns...)` | Numeric states can be safely summed by sorting key | Non-summed columns require deterministic semantics | +| `AggregatingMergeTree` | Store aggregate-function states | Insert and query with matching `*State`/`*Merge` functions | +| `CollapsingMergeTree(sign)` | Paired state/cancel rows are produced correctly | Unbalanced or reordered sign rows corrupt meaning | +| `VersionedCollapsingMergeTree(sign, version)` | Collapsing data can arrive out of order | Identity, sign, and version must be stable | +| replicated variants | Self-managed replicas use Keeper coordination | Replication is not sharding and does not remove retry design | +| `Distributed` | Route queries/inserts across shards | Local tables, sharding key, replica topology, and failure semantics remain explicit | + +Do not select a specialized engine merely to obtain “upsert” behavior. Write the query that must be correct before merges, then test it with multiple versions in separate parts. + +Example starting point for observations: + +```sql +CREATE TABLE observations +( + tenant_id UUID, + observed_at DateTime64(3, 'UTC'), + host String, + kind LowCardinality(String), + event_id UUID, + payload String, + ingested_at DateTime64(3, 'UTC') DEFAULT now64(3) +) +ENGINE = MergeTree +PARTITION BY toYYYYMM(observed_at) +ORDER BY (tenant_id, host, kind, observed_at, event_id); +``` + +This favors tenant/host/kind/time-range access. It is wrong for a workload whose dominant filters begin with another dimension. Validate with real predicates and `EXPLAIN indexes = 1`. + +## Sorting, primary index, partitions, and granules + +Design in this order: + +1. List high-value query shapes with filter frequency, selectivity, time range, and latency target. +2. Put frequently filtered low-to-moderate-cardinality dimensions early in `ORDER BY` when they cluster useful ranges. +3. Put time after stable scoping dimensions for tenant/entity time-series queries. +4. Add a deterministic identity/tie-breaker where ordering, versioning, or pagination needs it. +5. Use a different `PRIMARY KEY` only when a shorter sparse index has measured value; it must remain a prefix-compatible expression for the engine/version. +6. Partition for bounded retention, replacement, or backfill—not as a substitute for the sorting key. +7. Add data-skipping indexes only after query plans prove the primary sort cannot prune enough. + +Partition cautions: + +- A partition key creates separate part sets. Inserts touching many partitions create many parts. +- High-cardinality partitions increase filesystem/metadata/merge overhead. +- Monthly partitions are a common event-data starting point, not a universal default. +- A logical row's versions should stay in the same partition if query-time `FINAL` may process partitions independently. +- Partition pruning helps only when predicates can be related to the partition expression. + +Inspect pruning rather than inferring it: + +```sql +EXPLAIN indexes = 1 +SELECT count() +FROM observations +WHERE tenant_id = {tenant:UUID} + AND host = {host:String} + AND observed_at >= {from:DateTime64(3)} + AND observed_at < {to:DateTime64(3)}; +``` + +Check `system.parts` for active part count, rows, bytes, partition spread, and merge pressure. Check query logs for rows/bytes read versus returned. A query that returns ten rows after reading billions is a schema/query contract failure even when it is syntactically valid. + +## Inserts, batching, and deduplication + +Prefer client-side batches of at least 1,000 rows and commonly 10,000–100,000 rows for synchronous ingestion, subject to row width, memory, and latency. Tiny frequent inserts create small parts faster than background merges can consolidate them. + +When client batching is not feasible, evaluate asynchronous inserts: + +```sql +SET async_insert = 1; +SET wait_for_async_insert = 1; +``` + +`wait_for_async_insert = 1` acknowledges after a successful buffer flush and returns flush errors. Fire-and-forget acknowledgement can lose in-memory buffered data and conceal errors. ClickHouse 26.3 enables async inserts by default, so record the server version and explicit settings instead of assuming historical defaults. + +Define retry identity separately from engine reconciliation: + +- preserve a stable upstream event or batch identity; +- reuse the same deduplication token for the exact same retry payload; +- do not reuse a token for changed data; +- bound the deduplication window and document what happens after it; +- test retry after “server committed, client timed out”; +- verify dependent materialized-view behavior at the deployed version. + +From ClickHouse 26.1, async-insert deduplication can extend consistently through dependent materialized views. Do not project that behavior onto older versions. + +Avoid “exactly once” as an unqualified claim. State the boundary: accepted request, durable source part, dependent views, replicated copies, downstream export, or externally visible effect. + +## Corrections, mutations, and consistency + +Choose a correction strategy: + +| Strategy | Best fit | Visibility and cost | +|---|---|---| +| Append corrected fact and select latest | Event/version model | Immediate if query is version-aware; extra rows remain | +| `ReplacingMergeTree` | Versioned logical rows | Background merge is eventual; `FINAL`/`argMax` may be needed | +| lightweight `DELETE`/`UPDATE` where supported | Sparse corrections | Version and workload dependent; test actual semantics | +| `ALTER TABLE ... UPDATE/DELETE` mutation | Bounded bulk correction | Rewrites affected parts asynchronously; monitor completion | +| replacement table + backfill + swap | Sorting/partition/engine change | Operationally explicit; needs dual-read/write or cutover plan | + +Never run a broad mutation without estimating affected parts and bytes. Use a required predicate in adapter builders. Decide whether the caller waits using settings such as `mutations_sync`, polls system tables, or returns an operation identifier. + +ClickHouse consistency is not a single switch. Specify: + +- whether an insert acknowledgement covers one replica or a quorum; +- which replica a subsequent read may reach; +- whether distributed inserts queue locally; +- whether materialized views and projections are current; +- whether version reconciliation needs `FINAL`; +- how replica lag, failed parts, and Keeper availability surface. + +Do not expose a generic ORM `transaction()` that suggests multi-statement rollback. The reviewed Kaiju adapter correctly rejects transactional ORM sessions and requires idempotency/compensation. + +## Materialized views and projections + +Distinguish the acceleration mechanisms: + +| Mechanism | Ownership | Write/rebuild behavior | +|---|---|---| +| Incremental materialized view | Insert-triggered transformation into a target table | Processes newly inserted blocks; historical backfill is separate | +| Refreshable materialized view | Periodically recomputed query result | Suitable for bounded recomputation and joins whose sources change independently | +| Projection | Alternate stored layout attached to a table | Optimizer may select it; materialization and compatibility require proof | +| Data-skipping index | Per-granule metadata | Prunes granules but does not replace a correct sorting key | + +An incremental materialized view observes inserted blocks, not a magical current-state table. If the transformation joins another table, later changes to that other table do not retroactively update existing target rows. Define target engine, deduplication, backfill, cutover, and repair. + +Before adding a projection or view: + +1. capture the slow query and plan; +2. prove sorting-key and predicate fixes are insufficient; +3. define storage and insert amplification; +4. materialize historical data explicitly; +5. verify the optimizer selects the projection or consumers query the target; +6. test schema changes and rebuild time; +7. retain a base-table correctness path. + +## TTL and storage lifecycle + +TTL can delete, move, or recompress data and can aggregate expired rows for supported table designs. Treat TTL changes as data migrations: an altered rule may make old rows immediately eligible. + +Record: + +- timestamp expression and timezone; +- retention by tenant/data class; +- delete versus move/recompress action; +- merge scheduling and lag expectations; +- legal hold and user-deletion exceptions; +- backup/recovery interaction; +- how to prove expired data is gone from replicas and projections. + +Do not use `OPTIMIZE TABLE ... FINAL` as routine cleanup. It can create large write amplification and parts that normal merges will not combine efficiently. Reserve partition-scoped optimization for measured, bounded maintenance. + +## Replication and distribution + +Replication copies parts; sharding distributes rows. A `Distributed` table is usually a routing/query surface over local MergeTree-family tables. + +Define: + +- shard and replica counts plus failure domains; +- sharding expression and consequences of changing it; +- local and distributed table names; +- insert path (`Distributed` versus direct shard) and queue durability; +- quorum and sequential-read requirements; +- schema DDL coordination (`ON CLUSTER`) and partial-node recovery; +- Keeper path/macros for replicated engines; +- duplicate blocks and retry tokens; +- cross-shard aggregation, joins, and limits; +- failover and replica-lag observability. + +ClickHouse Cloud uses shared-storage architecture rather than the same local-part replication model as self-managed clusters. Verify deployment type before prescribing ZooKeeper/Keeper paths or local-disk repair. + +## Observability and diagnosis + +Inspect, with version-appropriate columns: + +- `system.parts`: active parts, partitions, rows, bytes, levels; +- `system.merges`: active merge/mutation work and progress; +- `system.mutations`: completion and failure reason; +- `system.replicas` and replication queues: lag and read-only state; +- `system.query_log`: duration, rows/bytes read, memory, exceptions, query IDs; +- `system.asynchronous_insert_log`: buffered insert/flush outcomes where enabled; +- `system.errors`: recurring server failures; +- materialized-view and projection sizes/usage through tables, parts, and query plans. + +Propagate stable `query_id`/correlation IDs. Log normalized operation metadata, not credentials or raw sensitive SQL parameters. Capture rows, bytes, elapsed time, retries, endpoint, and settings relevant to semantics. + +Failure signatures: + +| Signature | Likely cause | Next evidence | +|---|---|---| +| `Too many parts` | tiny inserts or too many touched partitions | insert rate/batch size and `system.parts` | +| `FINAL` dominates CPU | versioned engine used without query-friendly reconciliation | versions per key, partitions, `argMax` alternative | +| query reads most rows | sort key does not align with predicate | `EXPLAIN indexes = 1`, query log | +| mutation never finishes | huge rewrite, blocked replica, or resource pressure | `system.mutations`, merges, replica queue | +| source count differs from view | backfill omitted, retry duplicated, transform filtered | source blocks, MV target, version-specific dedup | +| distributed read is stale | replica lag/read routing | distributed topology and replication queues | +| TTL deletes unexpected rows | expression/timezone/change applied to old data | DDL history and affected partitions | + +## Schema evolution and verification + +Classify changes: + +- additive metadata-only or safe column addition; +- cautionary type/default/index/projection/TTL change; +- destructive drop or narrowing; +- physical layout change to engine, partition, sorting, primary, or sampling key; +- server-version-dependent operation. + +Physical layout changes generally require a replacement table, controlled backfill, validation, and swap. A migration generator must emit “manual” rather than fabricating a safe `ALTER`. + +Executable verification: + +```bash +docker compose up -d clickhouse +deno test -A libs/clickhouse/tests +deno test -A libs/clickhouse/tests/integration +``` + +Then prove a clean lifecycle: + +1. create schema from an empty server; +2. apply migrations twice and verify idempotent history/hash behavior; +3. insert representative batches, duplicates, late data, and corrected versions; +4. execute prepared selects, joins, aggregates, and streaming iteration; +5. execute bounded update/delete mutations and observe completion; +6. inspect generated SQL and `EXPLAIN indexes = 1`; +7. exercise view/projection backfill and TTL on disposable partitions; +8. interrupt migration/seed/insert and verify repair/resume; +9. close result sets and the underlying client; +10. repeat against the minimum and target ClickHouse versions. + +## Sources and freshness + +- Primary: [ClickHouse documentation](https://clickhouse.com/docs/) and linked ClickHouse engineering material, verified 2026-07-17 for parts, granules, sparse indexes, insert batching, async inserts, deduplication, mutations, views, projections, TTL, replication, and the stated 26.1/26.3 boundaries. +- Attachment: `kaiju-site-scope(17).zip/libs/clickhouse` source and tests, inspected 2026-07-17 as one custom-adapter implementation. + +Server settings and SQL behavior are version-sensitive. Recheck the deployed ClickHouse version; do not promote the private Kaiju adapter's names or capabilities into public ClickHouse contracts. diff --git a/skills/build-data/references/drizzle-architecture.md b/skills/build-data/references/drizzle-architecture.md new file mode 100644 index 0000000..ce2c46b --- /dev/null +++ b/skills/build-data/references/drizzle-architecture.md @@ -0,0 +1,198 @@ +# Drizzle architecture and dialect ownership + +## Contents + +- Mental model +- Package and layer map +- Schema and columns +- SQL AST and dialect +- Session, driver, and prepared queries +- Query builders and results +- Migrations and Drizzle Kit +- Resource lifetime +- Version boundaries +- Conformance checklist +- Sources and freshness + +## Mental model + +Drizzle is not one generic database wrapper. It is a set of packages and dialect-specific layers that share SQL AST types, table/column metadata, query-building conventions, and result mapping. A familiar `db.select().from(...)` surface does not imply that every dialect supports the same SQL, transactions, constraints, migrations, relational queries, or driver behavior. + +Use this ownership model: + +```text +application schema + -> dialect-specific table and column builders + -> Drizzle table/column metadata + -> query builder creates Drizzle SQL AST + -> dialect compiles AST to SQL + typed parameters + -> session prepares and executes through one driver + -> result mapper applies column decoders and join nullability + +schema source + -> snapshot/serializer + -> schema diff + -> reviewed migration artifacts + -> migrator applies artifacts and records history +``` + +Keep runtime ORM and migration-tool responsibilities distinct even when they share schema objects. + +## Package and layer map + +| Layer | Owns | Must not silently own | +|---|---|---| +| `drizzle-orm` core | SQL AST, expressions, aliases, tables/columns base types, selection mapping | Database-specific syntax or network transport | +| dialect module | quoting, placeholders, SQL clauses, feature grammar | Credentials, pooling, deployment policy | +| driver integration | native client adaptation and result shape | Schema diff policy | +| session | prepare/execute/iterate, parameter filling, result mapping, transaction surface | Pretending unsupported atomicity exists | +| schema module | table, column, index, constraint and relation declarations | Runtime connection construction | +| relational query layer | relation metadata and nested result assembly | Cross-dialect feature parity | +| Drizzle Kit | config, introspection, snapshot, diff, generation, push/studio workflows | Application request execution | +| migrator | ordered artifact application and history | Automatic rollback of non-transactional DDL | + +Inspect imports from the installed version. Drizzle uses internal symbols and entity kinds to identify objects. Multiple incompatible `drizzle-orm` instances in a monorepo can break identity assumptions even when TypeScript types look structural. Centralize the version and, where needed, the re-export owner. + +## Schema and columns + +A dialect-specific column builder owns at least: + +- compile-time data, driver-parameter, not-null, default, generated, enum, and table-name metadata; +- runtime column SQL type; +- application-to-driver encoder; +- driver-to-application decoder; +- default/generated/alias behavior; +- dialect-specific DDL metadata such as codecs, compression, statistics, or timezone. + +The table builder must build columns against the final table identity and attach the symbols Drizzle query utilities inspect. Preserve a dialect identity rather than masquerading as PostgreSQL or MySQL to reuse types. + +Schema-first does not mean “types are the database.” Verify generated DDL, runtime inserts, selected driver values, nullability, defaults, dates, big integers, decimals, arrays, maps, tuples, enums, and custom types. + +## SQL AST and dialect + +Prefer Drizzle's `SQL`, `SQL.Aliased`, column, table, subquery, view, placeholder, and parameter nodes over interpolated strings. The dialect decides: + +- identifier escaping; +- parameter placeholder syntax and typed parameter transport; +- casing policy; +- selection aliases; +- `WITH`, `SELECT`, joins, filters, grouping, sets, order, limits, settings; +- insert/update/delete grammar; +- DDL grammar; +- unsupported features. + +Raw SQL is an escape hatch, not a substitute for a grammar. Separate trusted static fragments from values and identifiers. For ClickHouse, a typed engine argument must distinguish identifiers, strings, numbers, and deliberately unsafe expressions so the renderer does not quote all four the same way. + +Dialect compilation tests should assert SQL and ordered parameters. Include aliases, nested selections, placeholders, casing, reserved identifiers, Unicode, nulls, joins, CTEs, set operators, limit/offset, and dialect-only clauses. + +## Session, driver, and prepared queries + +The driver boundary should be small enough to fake in unit tests. Define the actual client operations required: query, command, insert, result streaming, close, request settings, cancellation, and query identifiers. + +The session owns: + +1. dialect compilation; +2. named-placeholder filling; +3. application-value encoding and type binding; +4. native driver invocation; +5. row normalization; +6. selected-field decoder mapping; +7. joined-object nullification; +8. result-set disposal; +9. transaction behavior, including explicit rejection. + +A prepared-query object should preserve SQL, parameter metadata, selected fields, join nullability, and custom result mapping. `prepare()` need not mean a server-side prepared statement: document whether preparation only caches a compiled query and fills parameters per execution. + +If the native driver returns strings for 64-bit numbers, decimals, timestamps, arrays, or tuples, decode through the owning column. Do not globally coerce values based on JavaScript guesses. + +## Query builders and results + +Implement the smallest honest capability surface. A useful core may include: + +- select from table, alias, subquery, view, and raw SQL; +- CTEs; +- inner/left/right/full/cross joins plus dialect strictness/global modifiers; +- `PREWHERE`, `WHERE`, `GROUP BY`, `WITH ROLLUP/CUBE/TOTALS`, `HAVING`, `QUALIFY`; +- order, limit, offset, limit-by, settings, final, sample; +- union/intersect/except where server support is verified; +- insert values and insert-select; +- explicit mutation builders; +- unsafe query/command for unsupported syntax. + +Do not advertise Drizzle's relational query API unless schema relations, dialect SQL, and nested result assembly are implemented and tested. Flat SQL joins are not relational-query parity. + +Result mapping must handle: + +- explicit and inferred selections; +- aliases and SQL expressions; +- nested selection paths; +- nullable joined objects; +- column decoders; +- compact row formats whose positions must align with ordered selected fields; +- iterator early return and result-set close. + +## Migrations and Drizzle Kit + +Treat these as independent proof obligations: + +| Capability | Proof | +|---|---| +| schema serialization | deterministic snapshot contains all physical metadata | +| diff | property tests plus golden transitions | +| risk classification | safe/caution/destructive/manual cases cannot be bypassed accidentally | +| generation | atomic files, journal, pending-write recovery, stable ordering | +| application | clean server and representative upgrade | +| history | hash mismatch, duplicate ID, failed attempt, retry behavior | +| introspection | round trip only if actually implemented | +| push/studio | absent unless separately supported | + +ClickHouse cannot wrap a migration file in a cross-statement transaction. Execute statements in deterministic order, record query IDs and success/failure, make each statement retryable, and preserve a repair path when statement three fails after statements one and two committed. + +Physical changes—engine, partition key, sorting key, primary key, sampling key—should be `manual`: create replacement, backfill, validate, and swap. A generator that emits plausible invalid `ALTER` syntax is more dangerous than one that stops. + +## Resource lifetime + +Return or retain the underlying client/pool handle. Construction and environment loading belong to the composition root, not import time. The runtime owner must be able to: + +- close/drain in tests, CLI exit, worker shutdown, and server termination; +- cancel in-flight queries; +- close streaming result sets after completion, error, or early iterator return; +- configure logging without mutating global state during import; +- create isolated clients for integration tests. + +## Version boundaries + +Pin compatible versions of Drizzle ORM, Kit, native driver, TypeScript, and runtime. For every upgrade: + +1. inspect changed public and internal imports; +2. run SQL golden tests; +3. run type-level API tests; +4. run native driver mapping tests; +5. generate migrations without accepting them; +6. compare snapshots and risk classifications; +7. apply clean and upgrade paths; +8. run live integration tests against supported server versions. + +Avoid deep imports unless there is no public alternative. If unavoidable, isolate them behind one adapter module and lock a conformance test to the expected symbols. + +## Conformance checklist + +- [ ] Every public builder method either executes correctly or rejects explicitly. +- [ ] SQL and parameter order are deterministic. +- [ ] All value types round-trip through driver encoders/decoders. +- [ ] Prepared placeholders retain database type metadata. +- [ ] Join nullability maps nested objects correctly. +- [ ] Iterators release native results on early return. +- [ ] Transactions have real semantics or throw immediately. +- [ ] Schema snapshots cover all dialect DDL metadata. +- [ ] Diff tests cover every supported metadata transition. +- [ ] Manual/destructive migrations require explicit review. +- [ ] Clean install, upgrade, failed migration, and retry are tested. +- [ ] Underlying clients close and no global import side effects remain. + +## Sources and freshness + +- Primary: [Drizzle ORM documentation](https://orm.drizzle.team/docs/overview), verified 2026-07-17 for public schema, SQL, dialect, driver/session, query, migration, relation, and transaction concepts. +- Attachment: `kaiju-site-scope(17).zip/libs/clickhouse`, inspected 2026-07-17 for the custom Drizzle-like implementation and its dependency on Drizzle SQL internals. + +Drizzle internals and deep imports are version-sensitive. The uploaded Kaiju adapter is architecture evidence, not a published universal package contract; its package identity and public API are unverified. diff --git a/skills/build-data/references/failures.md b/skills/build-data/references/failures.md index 6cb59d7..626a826 100644 --- a/skills/build-data/references/failures.md +++ b/skills/build-data/references/failures.md @@ -1,19 +1,203 @@ -# Data failure signatures +# Data-system failure diagnosis and recovery -| Signature | Likely defect | Required evidence | +Use this reference when a data system is wrong, stale, incomplete, slow, leaking scope, or unrecoverable. Diagnose the violated owner/commit contract before adding retries or another abstraction. A green typecheck or completed process is not evidence that data committed correctly. + +## Contents + +- Triage protocol +- Authority and consistency failures +- Artifact and pipeline failures +- PostgreSQL and Drizzle failures +- Search, graph, and ClickHouse failures +- Query failures +- Resource and operational failures +- Recovery decision model +- Failure-injection matrix +- Verification and incident evidence +- Deliberate exclusions +- Sources and freshness + +## Triage protocol + +Preserve evidence before retrying or repairing: + +1. Stop destructive cleanup, compaction, retention, or automated retry if it can erase the failure. +2. Record incident time, affected tenant/range/run/change IDs, deployed versions, resolved config digest, and topology. +3. Identify the authoritative source for each disputed fact. +4. Capture manifests, checkpoints, migration history, queue/outbox state, projection receipts, relevant system tables, and redacted diagnostics. +5. Determine the last proven committed boundary for every required sink. +6. Classify impact: missing, duplicate, stale, extra, corrupted, unauthorized, unavailable, or slow. +7. Reproduce on a copy/fixture where possible. +8. Choose forward repair, replay, rebuild, rollback, or restore based on authority and identity evidence. +9. Reconcile after repair; do not infer success from command exit alone. + +Ask three questions first: + +```text +What is authoritative? +What is the last committed identity/version? +Can reapplication be proven safe? +``` + +If any answer is unknown, a blind retry can create more damage. + +## Authority and consistency failures + +| Signature | Likely cause | Inspect | Safe next step | +|---|---|---|---| +| Two stores disagree after “successful” request | direct dual write or premature success | authority transaction, outbox/change ID, sink receipts | replay durable change or targeted repair; add handoff | +| Both stores contain independent edits | dual authority | writers, timestamps/versions, conflict policy | stop one writer; resolve facts under explicit policy | +| Search grants access after membership revoke | projection used for authorization | current membership, server base filter, indexed tenant data | block through authority; delete/repair projection | +| Rebuild resurrects deleted data | source snapshot lacks tombstones/deletion boundary | snapshot identity, delete log, artifact retention | rebuild from complete boundary including deletes | +| Projection checkpoint is ahead of data | checkpoint committed before sink | receipt/change IDs | rewind to last proven change and replay idempotently | +| Projection data is ahead of checkpoint | crash after sink write | target version/change IDs | replay and detect already-applied change | + +Do not “pick the newest timestamp” unless clock ownership, ordering, and conflict semantics make that authoritative. + +## Artifact and pipeline failures + +| Signature | Likely cause | Recovery | +|---|---|---| +| JSONL parser fails on final line | process died mid-write | discard/truncate uncommitted segment; resume committed checkpoint | +| JSONL middle line corrupt | non-atomic append or external mutation | quarantine segment; rebuild from raw authority | +| Parquet cannot open/no footer | partial file published | remove pointer/reference; rebuild temporary artifact | +| Parquet schema varies by batch | inference from observed records | declare schema and rewrite compatible artifact | +| Memory rises with input size | materialized `list(records)` or global profiling set | bounded writer/profile sketches; large-input oracle | +| Run reports complete with errors | broad catches/best-effort stages | inspect required stage/sink states; mark incomplete | +| Daily staging file contains repeated retries | date-only append target | split by run identity; dedupe using record/source version | +| Resume skips records | checkpoint represents attempted input | rewind to committed output identity | +| Resume duplicates side effects | sink lacks idempotency/version | repair duplicates and add deterministic change identity | + +The retained PopModern MediaWiki pipeline catches and ignores failures in raw capture, cleaning, staging, profiling, Parquet, run state, PostgreSQL load, and metrics, yet prints completion. Its `records_for_profiling` also retains all staged records before profiling/Parquet. Use these as concrete failure signatures, not an endorsed resilience policy. + +## PostgreSQL and Drizzle failures + +| Signature | Likely cause | Evidence and repair | +|---|---|---| +| Migration generated but fails empty install | SQL never applied; artifact drift | apply full committed history to disposable DB; repair migration | +| Upgrade fails while empty install passes | data/backfill/lock assumption | restore representative prior snapshot; forward repair | +| Composite foreign key creation fails | referenced key is not an appropriate unique constraint | inspect `pg_constraint`; define named `UNIQUE` key | +| Follow-up migration recreates index/constraint | duplicate schema declarations or snapshot drift | compare Drizzle schema, snapshot, journal, introspection | +| Duplicate idempotent rows under concurrency | read-then-insert | unique constraint plus atomic insert/conflict policy | +| Timeline/sequence collisions | `existing.length + 1` | database-owned atomic allocation/lock/constraint | +| Queue item claimed twice | select-then-update claim window | atomic claim/conditional update test or `SKIP LOCKED` design | +| Process/test hangs | pool handle hidden/not closed | expose client close; drain and open-handle test | +| Raw SQL error reaches API | driver error leaked | stable constraint/code mapping and redacted diagnostics | +| Imports fail without env | module-scope config/client construction | lazy factory and import-safe schema | + +Before retrying deadlocks or serialization failures, ensure the whole transaction is safe to replay and bound attempts. Never retry uniqueness/check violations as transient errors. + +## Search, graph, and ClickHouse failures + +| Signature | Likely cause | Evidence and repair | |---|---|---| -| One store handles transactions and high-volume analytics through one model | OLTP/OLAP ownership collapsed | Workload and recovery map | -| Search/graph store called source of truth without rebuild policy | Projection authority unclear | Authoritative source and reconciliation | -| Drizzle-like API assumed to support transactions/migrations | Adapter shape mistaken for dialect support | Source, generated SQL, executable tests | -| Migration generates but clean install fails | Generation treated as application proof | Empty and upgrade database tests | -| Driver hidden behind DB object | Shutdown owner missing | Close/drain handle test | -| Raw DB message reaches client | Boundary leak | Stable problem and redacted cause | -| README and deployment name different graph engines | Documentation/source drift | Active manifests and query endpoint | -| Dual write partially succeeds | No durable projection protocol | Outbox/change identity and reconciliation | -| Pipeline reports success after sink failure | Required/optional state missing | Per-sink manifest and resume | -| JSONL append resumes at attempted offset | Checkpoint precedes commit | Committed output identity | -| All records retained for a statistic | Unbounded memory | Batch/stream oracle | -| Count is labeled exact but uses relation estimate | Query contract false | Explicit count strategy | -| Cursor repeats/skips under writes | Order lacks stable tie-breaker/context | Concurrent pagination test | - -Correct the owner and recovery contract before adding another abstraction layer. +| Search count is low after successful bulk call | per-document failures ignored | parse batch response, list rejected IDs, replay targeted records | +| Old field behavior after schema change | in-place collection incompatible or stale alias | inspect active collection/alias and schema version; rebuild/cut over | +| Deleted document remains | upsert-only projection | tombstone/delete feed or authoritative identity reconciliation | +| RDF query returns old and new values | append added correction but did not remove old triple | rebuild/replace graph or supported delete/update | +| Graph queries hit unexpected engine | docs/deployment drift | process/container/config/endpoint evidence | +| QLever index start fails | incomplete inputs/index artifacts or config mismatch | validate input manifest; rebuild named index, never route partial | +| ClickHouse duplicates appear | MergeTree semantics or retry dedupe misunderstood | inspect table engine/order/version, parts/merges; reconcile versions | +| ClickHouse mutation appears stuck | asynchronous mutation/part rewrite | inspect `system.mutations`, part errors, resource pressure | +| Analytics misses late records | window/backfill/checkpoint policy incomplete | replay affected range and define late-arrival reconciliation | + +Do not apply PostgreSQL update/uniqueness expectations to ClickHouse. Do not infer graph update capability from SPARQL query support. + +## Query failures + +| Signature | Likely cause | Inspect/correct | +|---|---|---| +| Cross-tenant results | missing server base predicate or graph scope escape | generated SQL/SPARQL and adversarial two-tenant fixture | +| Cursor repeats/skips | unstable order, wrong direction, mutable tiebreaker | compound boundary predicate and concurrent traversal | +| Cursor valid on unrelated filter | token lacks resource/filter/version binding | signed context digest and rejection | +| Exact total differs from rows | count/page predicates or snapshot differ | compile same authority/user filters and name consistency | +| Slow query after typed refactor | cast/expression prevents index pruning | real plan with representative values/data | +| SPARQL syntax/injection defect | raw expression/value concatenation | inspected term serializer, malicious literal tests | +| `IN []` returns surprising result | empty-list semantics undefined | reject or compile explicit true/false policy | +| New database field becomes public | wildcard not registry-bound | explicit public selection registry | + +Capture generated query and bound parameters safely. Never paste secrets or personal literal values into incident reports. + +## Resource and operational failures + +| Signature | Likely cause | Correction | +|---|---|---| +| Pool exhaustion | per-request construction, leaked transactions, wrong replica budget | acquisition metrics, stack/call-site audit, composition-root resource | +| Shutdown hangs | intake continues or driver close unreachable | stop intake, cancel/drain bounded work, close handle | +| Retry storm | unbounded retry without jitter/budget | central retry policy, circuit/load shedding, durable retry state | +| Disk full leaves “complete” artifact | publication not atomic/manifests premature | keep incomplete manifest, reclaim safe temp files, rebuild | +| Backfill overloads production | no rate/resource isolation | bounded batches, throttling, replica/maintenance plan | +| Health passes while migrations missing | health only checks process/socket | readiness includes schema/capability compatibility | + +## Recovery decision model + +Choose the smallest repair with complete evidence: + +| Condition | Prefer | +|---|---| +| Immutable authoritative input and deterministic projector | replay affected identities/range | +| Projection broadly corrupt or schema incompatible | versioned full rebuild and cutover | +| Authority data corrupted but valid backup/change log exists | restore/PITR plus downstream catch-up | +| Migration partially applied | forward repair using introspected state; rollback only if explicitly safe | +| Sink ahead of checkpoint | idempotent replay and checkpoint reconciliation | +| Checkpoint ahead of sink | rewind to proven receipt and replay | +| Authority unclear or independent writes conflict | stop mutation and require ownership decision | + +Repair records should include incident, operator, input boundary, tool/version, commands, before/after counts/hashes, rejects, and remaining uncertainty. + +## Failure-injection matrix + +Inject process loss: + +- before and after authority commit; +- after outbox/change insert; +- after sink write but before checkpoint; +- midway through artifact segment and before final manifest; +- during migration DDL/backfill; +- after queue lease and before acknowledgement; +- after wait/signal state change and before resume enqueue; +- during projection alias cutover; +- during shutdown with in-flight transaction/batch. + +Inject dependency failures: + +- timeout, connection reset, authentication failure, pool exhaustion; +- partial bulk rejection; +- malformed/corrupt input; +- disk full/permission denied; +- duplicate/out-of-order/late delivery; +- schema/config/version mismatch; +- stale authorization and privacy deletion. + +Every test needs a post-restart oracle: authoritative identities, sink receipts, checkpoints, manifests, projection state, and operator-visible error. + +## Verification and incident evidence + +Verification must include the system boundary that failed: + +- migration history plus PostgreSQL introspection and representative queries; +- artifact parser/full scan, schema, checksum, and manifest; +- search per-item receipts and identity reconciliation; +- graph input manifest, index version, representative SPARQL queries; +- ClickHouse system tables, convergence checks, and affected-range queries; +- complete cursor traversal under concurrent changes; +- restart/replay outcome and no skipped committed identities; +- privacy/tenant negative tests; +- process exit/open-handle check. + +Keep failed/blocked/passed distinct. If a registry, engine, credential, or realistic data volume is unavailable, record the blocked proof and do not convert structural inspection into a passing operational result. + +## Deliberate exclusions + +- Do not immediately retry before preserving last committed evidence. +- Do not repair projections by editing them manually without recording authoritative reconciliation. +- Do not force a specific database, logger, validator, or Effect runtime. +- Do not label swallowed optional errors harmless without an explicit required/optional contract. +- Do not use counts alone when identity/content/version can differ. +- Do not claim recovery from a unit test that never restarts the process or real dependency. +- Do not treat a type-level ORM test as migration, concurrency, or lifecycle evidence. +- Do not expose raw queries, credentials, or personal data in incident output. + +## Sources and freshness + +Grounded in the retained finance PostgreSQL/Drizzle schema, migration guide, query utilities, workflow store/worker implementation, and PopModern importer, MediaWiki ETL, artifact utilities, Typesense, QLever, SPARQL, and ClickHouse infrastructure, reviewed 2026-07-17. Several signatures come from explicit prototypes or incomplete code paths, including swallowed ETL failures, materialized profiling/Parquet, placeholder search sink, non-atomic workflow transitions, and hidden pool lifetime. Revalidate behavior in the consumer's installed versions and deployed topology. diff --git a/skills/build-data/references/postgres-drizzle.md b/skills/build-data/references/postgres-drizzle.md index 1bdfc40..a59bdb8 100644 --- a/skills/build-data/references/postgres-drizzle.md +++ b/skills/build-data/references/postgres-drizzle.md @@ -1,46 +1,302 @@ -# PostgreSQL and Drizzle +# PostgreSQL and Drizzle operational design -## Transactional model +Use this reference for PostgreSQL schema, Drizzle ORM runtime, Drizzle Kit migrations, `postgres.js` client ownership, cross-schema relations, concurrency, and production verification. Treat Drizzle's schema types, query runtime, driver, and Kit artifacts as related but distinct systems. -Use database constraints for identities and invariants that must hold under -concurrency. Define foreign keys, unique indexes, checks, nullability, defaults, -timestamps, deletion, tenant ownership, and transaction boundaries explicitly. +## Contents -## Drizzle ecosystem +- Ownership layers +- PostgreSQL invariant model +- Tenant and cross-schema design +- Drizzle schema and import ownership +- Client, session, and lifetime +- Configuration model +- Migration workflow +- Transactions and concurrency +- Error policy +- Integration sequence +- Failure and repair +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -Inspect ORM, Kit, dialect, driver, schema, migrator, relational/query APIs, and -generated artifacts at the installed versions. In a workspace, central re-export -can prevent incompatible duplicate class instances, but it becomes an explicit -dependency and version owner. +## Ownership layers -Prove separately: +| Layer | Owns | Does not prove | +|---|---|---| +| PostgreSQL schema | tables, columns, constraints, indexes, policies | application query scoping or migration success | +| Drizzle schema metadata | typed table/column/relation declarations | installed database state | +| Drizzle query builder/dialect | SQL AST and compilation | driver connectivity or plan quality | +| Driver/session | execution, parameters, result mapping, transactions | schema migration history | +| `postgres.js` | pool, sockets, prepared behavior, shutdown | Drizzle relational semantics | +| Drizzle Kit | snapshot/diff/generate/migrate tooling | reviewed correctness or rollback | +| Composition root | config, construction, logging adapter, readiness, close/drain | business authorization | -- schema/type generation; -- migration generation; -- migration contents and review; -- application against an empty database; -- upgrade from a representative prior schema; -- rollback or forward repair policy; -- seeding and idempotency; -- representative query and insert; -- transaction and isolation behavior; -- connection close/drain. +Avoid treating `db` as a magic unit that owns everything. The underlying client/pool must remain reachable for lifecycle and driver-specific behavior. -## Resource lifetime +## PostgreSQL invariant model -Separate database construction from environment loading. Avoid import-time -connections and global logger configuration. The composition root should own the -underlying driver/pool handle and make shutdown reachable for tests, workers, -CLIs, and graceful server termination. +Put concurrency-sensitive invariants in PostgreSQL when it is the authority: + +- primary keys and stable identity; +- `NOT NULL` and explicit defaults; +- unique constraints for business/idempotency keys; +- foreign keys and deletion/update actions; +- check constraints for bounded state machines and value rules; +- exact numeric precision/scale for money plus explicit currency; +- timestamp timezone and clock-owner policy; +- exclusion constraints where range overlap is the invariant; +- transactions and isolation for multi-row state transitions. + +Application validation improves errors but does not replace database constraints under concurrency. + +The retained finance schema separates `auth`, `finance`, and `workflows`; finance rows include `organization_id`; money uses `numeric(19,4)` with currency; state-like text fields have checks; and cross-schema relationships are explicit. Those are observed patterns, not universal table names. + +## Tenant and cross-schema design + +For tenant-owned relationships, consider composite keys so the database can prevent cross-tenant references: + +```ts +const account = finance.table('account', { + id: text().primaryKey(), + organizationId: text('organization_id').notNull(), +}, (t) => [ + unique('account_org_id_uq').on(t.organizationId, t.id), +]) + +const entry = finance.table('entry', { + organizationId: text('organization_id').notNull(), + accountId: text('account_id').notNull(), +}, (t) => [ + foreignKey({ + columns: [t.organizationId, t.accountId], + foreignColumns: [account.organizationId, account.id], + }), +]) +``` + +PostgreSQL foreign keys target a primary key, unique constraint, or another supported unique key shape. The retained finance guide documents a real issue: a standalone unique index was not the right ownership when another table referenced the composite columns; use a table `UNIQUE` constraint for referenced key semantics. Do not duplicate `.unique()` and `uniqueIndex()` on the same columns. + +Database-level tenant integrity does not replace authorization. Queries must still bind the current authorized organization and resource. + +## Drizzle schema and import ownership + +In monorepos, establish one dependency/version owner for Drizzle. The retained finance package exposes a central `@utils/db/drizzle` surface because Drizzle classes with private/protected fields can become TypeScript-incompatible when separate packages resolve different copies. Verify whether the consumer has this failure before centralizing imports, but avoid mixed versions and duplicate runtime class identities. + +Separate exports: + +```text +@app/db runtime factories +@app/db/schema schema aggregate +@app/db/schema/auth bounded schema surface +@app/db/drizzle controlled ORM/core symbols if needed +@app/db/types public runtime types +``` + +Do not create broad re-exports that accidentally make internal tables or unstable ORM internals a public package contract. Keep schema import paths used by Drizzle Kit stable and inspect the generated snapshot after moves. + +Drizzle relations are application query metadata, not PostgreSQL foreign keys. Define both when both behaviors are needed and verify generated SQL for actual constraints. + +## Client, session, and lifetime + +The composition root should own the client and Drizzle wrapper: + +```ts +interface DatabaseResource { + db: DB + client: Client + close: (options?: { timeoutSeconds?: number }) => Promise +} + +function createDatabaseResource(config: DatabaseConfig): DatabaseResource { + const client = postgres(config.url, { + max: config.maxConnections, + prepare: config.prepare, + connect_timeout: config.connectTimeoutSeconds, + idle_timeout: config.idleTimeoutSeconds, + }) + const db = drizzle({ client, schema, logger: config.logger }) + return { db, client, close: () => client.end({ timeout: 5 }) } +} +``` + +Use the installed driver's exact option and close signatures. This example is an ownership shape, not a copy-paste API guarantee. + +Decide: + +- pool maximum relative to replicas/workers and database limit; +- prepared statements behind transaction/session poolers; +- connect, idle, statement, and pool wait timeouts; +- TLS and certificate verification; +- application name and server settings; +- retry boundary; +- health/readiness query; +- shutdown order and in-flight drain; +- query logging/redaction owned by the application's observability layer. + +The retained `createDatabase()` constructs the client internally and returns only Drizzle. That can hide `client.end()`; repair the ownership rather than assuming the wrapper closes itself. It correctly separates optional LogTape sink configuration from database construction in the newer client file, while the retained server contains import-time logging configuration as a counterexample. Do not force LogTape if the consumer selected another observability owner. + +## Configuration model + +Validate lazily at the composition boundary: + +```ts +interface DatabaseConfig { + url: string + maxConnections: number + prepare: boolean + connectTimeoutSeconds: number + idleTimeoutSeconds: number + statementTimeoutMs: number + ssl: 'disable' | 'require' | 'verify-full' + logQueries: boolean +} +``` + +Importing schema or types should not require environment access, open a connection, or configure global logging. Tests and CLIs should be able to inject an explicit URL/client. Redact credentials from errors and diagnostic configuration. + +The retained package defaults `max` to 10 and `prepare` to false. Those are repository choices, not universal defaults. Derive them from the actual database/proxy/replica topology. + +## Migration workflow + +The retained finance package uses: + +```ts +defineConfig({ + out: './drizzle', + schema: './schemas', + dialect: 'postgresql', + dbCredentials: { url: DATABASE_URL }, +}) +``` + +At the consuming version, confirm config keys and CLI behavior. The operational sequence is: + +1. Change executable TypeScript schema. +2. Run pinned Drizzle Kit generation from the owning package/directory. +3. Inspect SQL, snapshot, and journal changes. +4. Review constraints, defaults, casts, backfills, locks, and destructive operations. +5. Apply to an empty disposable database. +6. Upgrade representative previous schemas/data. +7. Run application reads/writes and invariant tests. +8. Measure lock/duration implications for production-sized data. +9. Define rollback or forward-repair procedure. +10. Commit schema and generated artifacts together. + +Use generated SQL migrations as reviewed history. `push` can be useful for disposable exploration but bypasses the same reviewed artifact sequence; do not use it as durable production history by accident. + +If intentionally rebuilding the baseline, remove/regenerate the complete migration metadata set coherently. The retained guide notes that deleting only parts of `drizzle/meta` can leave a missing journal. Never repair migration state by guessing which generated files are disposable. + +## Transactions and concurrency + +Define transactions around authoritative state transitions, not repository method count. Test the actual driver/dialect transaction API and isolation. + +Patterns requiring special attention: + +- idempotent create: unique key plus insert/conflict or transaction, not read-then-insert; +- monotonic per-parent sequence: database-owned sequence/counter/locked row, not `existing.length + 1`; +- queue claim: one atomic claim such as `FOR UPDATE SKIP LOCKED` or conditional update with proven selection behavior; +- balance/ledger changes: immutable entries and transactionally maintained projections where applicable; +- outbox: business state and change record in the same transaction; +- serialization/deadlock: bounded retry of the entire safe transaction with jitter and observability; +- savepoints/nested transactions: verify installed behavior rather than assuming. + +Drizzle transaction types do not establish that a custom adapter or serverless driver supports every PostgreSQL behavior. ## Error policy -Map unique, foreign-key, serialization, timeout, unavailable, and unexpected -errors to stable domain/API failures. Do not put raw SQL/driver messages in -client responses. Keep redacted causes and correlation in diagnostics. +Classify at the database boundary: + +| Category | Example response policy | +|---|---| +| unique/check/foreign-key | stable conflict/unprocessable domain error after identifying owned constraint | +| serialization/deadlock | retry whole transaction if safe and within budget | +| statement/lock timeout | stable unavailable/timeout; preserve retry metadata internally | +| connection/pool unavailable | readiness/availability failure, not invalid client input | +| migration mismatch | fail startup or capability readiness explicitly | +| unexpected driver error | generic client problem; redacted structured cause internally | + +Do not send raw SQL, connection URLs, table layout, or driver messages to clients. Avoid parsing human messages when stable error codes/constraint names are available. + +## Integration sequence + +```text +composition root parses config + -> constructs owned postgres.js client + -> wraps it in Drizzle with one schema/version + -> verifies readiness/migration compatibility + -> injects DB capability into services/workers + -> runs parameterized transactions/queries + -> maps stable database errors + -> stops intake, drains work, closes client +``` + +Schema release: + +```text +schema edit -> generate -> review SQL/snapshot -> empty install -> upgrade fixture + -> application integration -> production rollout/observe -> forward repair if needed +``` + +## Failure and repair + +| Signature | Defect | Repair evidence | +|---|---|---| +| Migration generates but clean install fails | generation treated as application proof | empty DB apply and schema introspection | +| Composite FK fails despite unique index | index/constraint semantics confused | named table unique constraint and applied FK | +| Second migration recreates same uniqueness | duplicate declaration | inspect schema and generated diff | +| Two packages produce incompatible Drizzle types | duplicate version/runtime identity | lockfile resolution and central version owner | +| Process hangs after tests | client close inaccessible | explicit resource close and zero open-handle test | +| Importing schema requires env | config at module scope | import-safe schema and lazy config | +| Idempotency race creates duplicates/errors | read-before-insert | database constraint and atomic insert policy | +| Queue work executes twice | claim split or lease semantics incomplete | concurrent workers and atomic claim oracle | +| Raw constraint message reaches caller | boundary leakage | stable mapping plus redacted cause | + +## Test matrix + +Test: + +- schema import without environment/network permissions; +- config validation and credential redaction; +- connection/readiness and explicit close/drain; +- empty migration install and representative upgrades; +- generated schema matches introspected constraints/indexes; +- cross-schema and composite foreign keys; +- unique constraint versus unique-index intended ownership; +- check constraints with invalid direct SQL writes; +- transaction commit, rollback, nested/savepoint behavior if used; +- uniqueness/idempotency races with multiple connections; +- deadlock and serialization retry exhaustion; +- queue claim/lease concurrency if PostgreSQL owns work; +- tenant-isolated relationships and queries; +- pool exhaustion, database restart, statement/lock timeout; +- graceful shutdown with in-flight transactions; +- ORM query/result mapping, decimals, timestamps, arrays, JSON, and nulls; +- generated package/monorepo import identity if re-exporting Drizzle. + +## Executable verification + +Run pinned repository tasks, not generic commands if the project wraps Drizzle through Aube/mise/npm/Deno. The retained finance guide uses: + +```bash +aube --dir utils/db db:generate +aube --dir utils/db db:migrate +``` + +In a disposable database, apply committed migrations, inspect `pg_constraint`, `pg_indexes`, schemas, and migration history, then run real service queries. Upgrade a copied prior schema. Run two or more concurrent clients against uniqueness and claim paths. Stop the process during in-flight work and assert the pool closes or the documented timeout expires. + +## Deliberate exclusions + +- Do not force Drizzle when the consumer has another selected data layer. +- Do not force LogTape, Zod, or Effect; integrate with selected owners. +- Do not import environment or connect while importing schema/types. +- Do not equate Drizzle relations with foreign keys. +- Do not equate a unique index with every semantic of a table unique constraint. +- Do not treat generated SQL as reviewed/applied/upgrade-safe. +- Do not claim custom Drizzle-like adapters support PostgreSQL transactions or Kit. +- Do not use `push` as reviewed production migration history without explicit policy. +- Do not hide the driver/pool close handle. -## Concurrency verification +## Sources and freshness -Test idempotent creates, uniqueness races, sequence allocation, pagination under -writes, transaction rollback, deadlock/serialization retries, pool exhaustion, -and shutdown with in-flight work. +Grounded in the retained new/old finance `utils/db` README, `client.ts`, `env.ts`, `drizzle.ts`, `drizzle.config.ts`, generated migration, auth/finance/workflow schemas, package manifests, and workflow PostgreSQL store, reviewed 2026-07-17. Observed package versions include Drizzle ORM 0.45.x, Drizzle Kit 0.31.x, and postgres.js 3.4.x in the uploaded package manifest; APIs and configuration are version-sensitive. Verify installed versions, lockfile resolution, generated artifacts, and real PostgreSQL behavior before copying examples. diff --git a/skills/build-data/references/projections.md b/skills/build-data/references/projections.md index 2b1e4bf..3c793cf 100644 --- a/skills/build-data/references/projections.md +++ b/skills/build-data/references/projections.md @@ -1,33 +1,294 @@ -# Search and graph projections +# Search, graph, and analytical projections -## Search indexes +Use this reference when authoritative data is copied into Typesense or another search engine, QLever/Blazegraph or another graph query engine, ClickHouse, a materialized view, or a cache. A projection is operationally complete only when its identity, build, publication, lag, deletion, reconciliation, and rollback contracts are explicit. -Treat Typesense or another search engine as a rebuildable projection unless the -system explicitly assigns it authority. Define document identity, schema, -tokenization/faceting/sorting, bulk import, update/delete behavior, alias/index -replacement, lag, backfill, and tenant visibility. +## Contents -Use versioned index names and an atomic alias switch where supported. Preserve -the previous index until the new projection is validated and rollback is safe. +- Authority and projection ownership +- Change identity and projector contract +- Search projections +- RDF and SPARQL projections +- Analytical projections +- Versioned build and atomic publication +- Deletion, correction, and privacy +- Reconciliation and repair +- Integration sequence +- Failure and recovery +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Graph stores +## Authority and projection ownership -For QLever, Blazegraph, or another SPARQL engine, define RDF term identity, -namespaces, ontology/version provenance, serialization, bulk-load format, -inference expectations, update capability, and query endpoint ownership. +For each projected fact, name: -Do not choose the active engine from a README label. Inspect deployment manifests, -ingestion scripts, endpoint configuration, and consumer queries. +- the authoritative source and transaction boundary; +- the durable change identity or replay source; +- the projector version and configuration; +- the target document/subject/row identity; +- expected lag and freshness objective; +- deletion/tombstone behavior; +- whether the projection is required for product availability; +- who can trigger a backfill, cutover, rollback, or targeted repair. -## Reconciliation +Do not call a search or graph store authoritative merely because it serves reads. Serving authority and data authority are different. A store can own query availability while remaining rebuildable from PostgreSQL, immutable artifacts, or another source. -Record authoritative change identity, projection version, checkpoint, last -successful item/range, rejected records, and validation summary. Compare source -and projection counts/hashes or domain invariants. Support targeted repair and -full rebuild. +The retained PopModern repository illustrates why deployment evidence matters. Its README and data guide describe RDF N-Triples served through QLever, Typesense search, PostgreSQL/Supabase, and prototype ClickHouse infrastructure. Older migrations and comments mention other graph/export approaches. Inspect the active `Qleverfile`, container scripts, query endpoints, Typesense schemas/seeders, and ingestion recipes; do not select Blazegraph, QLever, or another engine from a stale label. -## Failure behavior +## Change identity and projector contract -A required projection failure keeps the run incomplete. An optional projection -failure remains visible with retry and operator action. Never swallow a sink -exception and report global success. +A projector should behave like this: + +```ts +interface ProjectionChange { + changeId: string + authorityVersion: string + entityId: string + operation: 'upsert' | 'delete' + occurredAt: string + schemaVersion: string + payload?: T +} + +interface ProjectionReceipt { + projection: string + projectionVersion: string + changeId: string + targetIdentity: string + status: 'applied' | 'already-applied' | 'rejected' + appliedAt: string +} +``` + +The projector must be idempotent by `changeId`, target version, or another proven mechanism. “Upsert” is not sufficient if applying an older event after a newer one can regress state. Carry authority version/event time and define ordering or compare-and-ignore behavior. + +If writes originate in PostgreSQL, use a transactional outbox, logical change stream, or another durable change source when losing the projection update is unacceptable. Writing PostgreSQL and Typesense/QLever/ClickHouse in one request handler is a dual write, not a transaction. + +## Search projections + +Define a search collection contract independently of the source record: + +```ts +interface SearchDocumentV3 { + id: string + tenantId: string + title: string + aliases: string[] + status: 'active' | 'archived' + authorityVersion: string + indexedAt: string +} +``` + +Decide and test: + +- stable document ID and tenant namespace; +- searchable versus facetable versus sortable fields; +- tokenization, locale, stemming, typo tolerance, synonyms, and infix behavior; +- optional/null/empty handling; +- nested and array field behavior; +- ranking and tie-break rules; +- per-tenant filter applied by the server, not trusted from the client; +- batch import/upsert response inspection per document; +- delete and tombstone propagation; +- schema evolution and rebuild triggers; +- alias or collection cutover support in the installed engine/version. + +Bulk APIs can return a successful HTTP response while individual documents fail. Parse every item result, quarantine rejected documents with safe reasons, and reconcile requested IDs against accepted IDs. + +The PopModern general-purpose `TypesenseSink` is explicitly a placeholder that logs what it would do; it does not convert RDF to documents or write them. Do not count its registration as a live projection. The separate MediaWiki sink calls Typesense bulk import, but does not inspect per-item results in the retained code. Treat that as an incomplete error contract. + +## RDF and SPARQL projections + +An RDF projection needs explicit term and ontology ownership: + +- base IRI and stable subject identity; +- namespace registry and prefix versions; +- source-to-RDF mapping version; +- literal datatype and language-tag rules; +- blank-node policy; use stable IRIs where cross-run identity matters; +- ontology and inference version; +- named graph or dataset ownership if used; +- serialization and parser version; +- duplicate triple and delete/correction model; +- engine update capability versus rebuild-only operation; +- query endpoint, timeout, result limits, and service owner. + +N-Triples append is easy to implement and hard to correct. Appending a changed record can leave the old triple present. Define whether the build is a full immutable dataset, per-source graph replacement, or an update-capable store. If full rebuild is the contract, publish a complete versioned dataset and atomically switch the serving index. + +The PopModern importer maps records into an in-memory RDFLib graph, flushes batches, and appends N-Triples. This bounds graph memory, but it does not provide record-level checkpoints, atomic multi-file publication, delete semantics, or a manifest. The QLever index must therefore be treated as a build from a named complete input set, not as proof that every recipe committed successfully. + +For SPARQL queries, test the active engine rather than assuming common semantics cover: + +- supported SPARQL version/features; +- timeout and result-size limits; +- property paths, full-text extensions, and service clauses; +- datatype comparisons and unbound variables; +- query plan behavior for the actual data distribution; +- update availability and authorization; +- inference/materialization behavior. + +## Analytical projections + +For ClickHouse, aggregate tables, or analytical materialized views, define: + +- event/row identity and deduplication policy; +- source event time, ingest time, and correction version; +- column types, null/default semantics, and schema evolution; +- partition and ordering keys derived from query/lifecycle needs; +- late-arrival window and backfill path; +- mutable correction semantics and convergence checks; +- aggregate refresh/materialization ownership; +- retention/TTL and legal-delete propagation; +- query consistency expected by callers. + +Do not promise immediate row uniqueness or synchronous update semantics merely because the projection exposes SQL. Keep transactional invariants in their authoritative owner and document analytical convergence. + +## Versioned build and atomic publication + +Prefer versioned targets: + +```text +authority snapshot/outbox boundary: 004182 +projection schema: search-comic-v7 +target: comics_20260717_142233_v7 +checkpoint: through change 004182 +``` + +Build sequence: + +1. Freeze or name the input snapshot/change boundary. +2. Create a new target/index/collection/dataset with the declared schema. +3. Stream bounded batches and inspect every batch receipt. +4. Record rejects without silently omitting them. +5. Reconcile counts, IDs, tenant/domain invariants, and sampled content. +6. Run representative queries and latency/error checks. +7. Atomically switch an alias/router/config pointer where supported. +8. Observe the new target through the real application. +9. Retain the previous target for a defined rollback window. +10. Delete old targets only after rollback and retention requirements expire. + +If the engine lacks an atomic alias, design a routing layer or a maintenance-window cutover. Do not rename a multi-step replacement “zero downtime” without a concurrency test. + +## Deletion, correction, and privacy + +Deletion is a first-class projection operation. Define: + +- hard delete, tombstone, or hide policy; +- propagation deadline; +- how a rebuild learns that a source record was deleted; +- how old immutable artifacts are retained or redacted under policy; +- how derived documents/triples/aggregates are located; +- how deletion is confirmed across all required projections; +- how a failed deletion is retried and escalated. + +Full-snapshot rebuilds naturally drop absent records only if the target is replaced wholesale. Incremental upserts do not. Maintain tombstones or compare authoritative and projected identity sets. + +Corrections need version-aware application. A late retry of source version 4 must not overwrite version 5. For RDF builds, correction may require rebuilding the relevant graph/dataset because an appended replacement triple does not remove the old one. + +## Reconciliation and repair + +Use several layers of reconciliation: + +| Layer | Example proof | +|---|---| +| Transport | Every requested batch item has a success or rejected receipt | +| Identity | Authoritative active IDs equal projected active IDs for a boundary | +| Counts | Per tenant/status/day counts agree within declared semantics | +| Content | Deterministic hashes of normalized projection records match | +| Domain | No public document references deleted/private authority rows | +| Query | Representative search/SPARQL/analytical queries return expected fixtures | +| Freshness | Checkpoint lag and oldest unapplied change stay within objective | + +Support both targeted repair and full rebuild. Targeted repair accepts an authoritative identity/range and is idempotent. Full rebuild uses a versioned target and safe cutover. Record who initiated repair, input boundary, projector version, results, and rejected items. + +## Integration sequence + +```text +authoritative transaction + -> durable change identity or named snapshot + -> normalize projection record with authority version + -> idempotent projector + -> inspect per-item sink receipts + -> commit projection checkpoint + -> expose lag/error metrics + -> reconcile identities/content/invariants + -> cut over or repair +``` + +For a pipeline with several sinks, keep independent state: + +```json +{ + "changeId": "004182", + "sinks": { + "typesense": {"state":"complete","target":"comics_v7"}, + "qlever": {"state":"failed","target":"kg_20260717","error":"..."}, + "clickhouse": {"state":"pending"} + } +} +``` + +Global success is false until every required sink is complete. + +## Failure and recovery + +| Failure | Required response | +|---|---| +| Projector crashes after sink write but before checkpoint | Replay same change safely; detect already-applied version | +| Checkpoint advances before sink acknowledgement | Reconcile gap and rewind/replay; fix ordering | +| Search bulk import partially rejects documents | Persist item-level rejects; keep build incomplete if required | +| Alias switches before validation | Roll back alias and invalidate incomplete target | +| Deleted authority record remains searchable | Apply tombstone/identity diff; test tenant/privacy filters | +| RDF append contains old and new values | Rebuild/replace graph or issue explicit deletes through supported update path | +| QLever/graph index build fails halfway | Never route to partial index; rebuild named target from manifest | +| Projection schema changes while backlog exists | Version event/normalizer and run explicit compatibility or rebuild path | +| Backfill races live updates | Use a boundary plus catch-up phase; compare versions before application | +| Optional projection fails | Show degraded capability and retry state; do not silently claim fresh | + +## Test matrix + +Test: + +- initial empty build, full build, no-op rebuild, and incremental catch-up; +- duplicate delivery and out-of-order source versions; +- inserts, updates, deletes, undeletes, and tenant moves; +- process loss after sink commit but before checkpoint; +- partial batch failures and retry of only rejected identities; +- schema-compatible and incompatible projection changes; +- backfill concurrent with live writes; +- cutover and rollback while queries are active; +- old and new target query equivalence on frozen fixtures; +- search ranking/facet/filter/sort behavior; +- RDF term identity, datatypes, correction, and representative SPARQL queries; +- analytical late arrival, correction, and retention behavior; +- privacy deletion across every projection; +- lag objective and alerting; +- targeted repair and full rebuild from the same authoritative boundary. + +## Executable verification + +Use engine-native commands and real application requests. Examples to adapt: + +```bash +curl -fsS "$TYPESENSE_URL/collections/comics/documents/search?q=fixture&query_by=title" +curl -fsS --get "$QLEVER_URL" --data-urlencode 'query=SELECT (COUNT(*) AS ?count) WHERE { ?s ?p ?o }' +``` + +Run a reconciliation program that exits non-zero for missing, extra, stale-version, tenant-leaking, or rejected required records. Verify the serving alias/pointer and then issue the application's real search/graph/analytics requests before retiring the previous target. + +## Deliberate exclusions + +- Do not mandate Typesense, QLever, Blazegraph, ClickHouse, or any named engine without repository/deployment evidence. +- Do not treat the PopModern placeholder Typesense sink as implemented. +- Do not assume search bulk HTTP success means every document succeeded. +- Do not use the projection as authority unless the architecture explicitly assigns it authority and recovery. +- Do not dual-write required state without a durable gap-repair mechanism. +- Do not claim exactly-once projection; specify idempotent replay and observable convergence. +- Do not force Zod, LogTape, Effect, or a specific queue/outbox implementation when the consumer chose another owner. +- Do not delete the previous target before rollback evidence and retention policy allow it. + +## Sources and freshness + +Grounded in the retained PopModern `DATA_PIPELINE.md`, importer recipes and sinks, MediaWiki ETL, QLever configuration/start scripts, Typesense seed/sink code, SPARQL endpoints, and legacy/current deployment artifacts, reviewed 2026-07-17. PopModern is source evidence with active, legacy, and placeholder paths; the actual deployed engine and capability set must be re-established from the consuming repository. Engine alias, bulk-result, update, and query behavior is version-sensitive and must be verified against installed official documentation and executable endpoints. diff --git a/skills/build-data/references/queries.md b/skills/build-data/references/queries.md index 1a5512b..ef7b646 100644 --- a/skills/build-data/references/queries.md +++ b/skills/build-data/references/queries.md @@ -1,38 +1,300 @@ -# Query contracts +# Query execution, safety, pagination, and count semantics -## Safe construction +Use this reference when translating a normalized query contract into SQL, SPARQL, search requests, or another storage protocol. The query layer must preserve server authority, public semantics, parameter safety, ordering, resource bounds, and error classification. A typed builder does not prove those properties. -Parse public filters, sorts, selected fields, and pagination into a validated -query specification. Map approved public fields/operators to expressions. Do not -interpolate arbitrary identifiers or predicates. +## Contents -Apply server-owned tenant, visibility, and lifecycle filters before user input. -Keep transport syntax separate from storage-specific execution. +- Ownership boundaries +- Normalized query model +- Field and operator registries +- Server-owned constraints +- SQL construction +- SPARQL construction +- Sorting and cursor pagination +- Field selection +- Count strategies +- Resource and error policy +- Integration sequence +- Failure cases +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Pagination +## Ownership boundaries -Cursor pagination requires a stable total order with a tie-breaker and an encoded -identity tied to relevant filter/order/version context. Offset pagination may be -adequate for bounded/admin views but can shift under concurrent writes. +Keep these layers separate: -Test duplicates, deletions, inserts between pages, invalid/expired cursors, empty -pages, descending and compound sorts, and tenant isolation. +```text +transport decoder + -> normalized query specification + -> endpoint/domain policy + -> storage field/operator registry + -> storage-specific expression compiler + -> query execution and cancellation + -> row/binding decoding + -> response pagination/count metadata +``` -## Count choices +Transport syntax does not own database identifiers. A public field such as `createdAt` maps through an allowlisted registry to a specific SQL column or SPARQL variable/pattern. The server owns tenant, organization, lifecycle, visibility, and other non-overridable constraints. -Use exact counts only when correctness and cost justify them. Planner estimates -should reflect the active filtered query; relation estimates are coarse hints. -Omit counts where continuation is sufficient. Name the mode in the API contract. +The retained finance `utils/query` code is detailed observed evidence for filters, sorts, fields, HMAC cursors, query/JSON/form adapters, per-field operator definitions, limits, and exact/planned/estimated/no-count modes. It is not automatically a public package contract. Confirm the exact exports and installed source in the consuming repository. -## SPARQL +## Normalized query model -For SPARQL, separate query construction, parameter/value serialization, -transport, result parsing, domain mapping, pagination, and engine error handling. -Use an inspected builder such as `@okikio/sparql` only from actual exports. Keep -query preview safe and redact credentials/endpoints where necessary. +Use one internal model after source decoding: -## Verification +```ts +type QuerySpec = { + filters: null | Array<{ + field: string + operator: 'eq' | 'ne' | 'gt' | 'gte' | 'lt' | 'lte' | + 'between' | 'in' | 'nin' | 'contains' | 'icontains' | + 'startswith' | 'endswith' | 'is_null' | 'is_not_null' + value?: unknown + }> + sorts: null | Array<{ field: string; direction: 'asc' | 'desc'; tiebreaker: boolean }> + fields: null | { type: 'simple'; fields: string[] } | + { type: 'jsonapi'; fields: Record } + pagination: + | { type: 'offset'; offset: number; limit: number } + | { type: 'cursor'; limit: number; cursor?: string; decodedCursor?: CursorData } +} +``` -Inspect generated SQL/SPARQL, run representative datasets and query plans, -exercise malformed filters and engine failures, and compare exact/estimated -count semantics. Type-level builders do not replace execution. +Disabled features need explicit semantics. The retained query factory normalizes disabled filters/sorts/fields to `null`; that is useful if handlers/executors distinguish “feature unavailable” from an empty client request. Do not silently accept a public filter and then ignore it. Either reject it or document the disabled contract. + +Cap complexity before compilation: number of filters/sorts/selected fields, values in `in`, nesting if supported, offset, limit, decoded cursor size, query timeout, and maximum response rows/bytes. + +## Field and operator registries + +Public fields need storage-owned mappings: + +```ts +const accountQueryFields = { + status: { + expression: account.status, + filter: { type: 'enum', operators: ['eq', 'in'], values: ACCOUNT_STATUS }, + sortable: true, + selectable: true, + }, + createdAt: { + expression: account.createdAt, + filter: { type: 'date', operators: ['gt', 'gte', 'lt', 'lte', 'between'] }, + sortable: true, + selectable: true, + }, + id: { expression: account.id, sortable: true, selectable: true }, +} as const +``` + +Registry rules: + +- unknown fields/operators fail closed; +- values are parsed according to the registry, not guessed by the compiler; +- enum membership is explicit; +- null operators reject values; +- `between` requires exactly two ordered/coercible values; +- array-consuming operators are enabled per field and have length limits; +- string operators define escaping, collation, and case behavior; +- identifier/expression values come from code-owned registry entries only; +- sortable/selectable/filterable are separate capabilities. + +An empty registry must have intentional semantics. The retained tests sometimes treat an empty allowlist as “allow any.” That is dangerous at a public storage compiler unless the expression mapping still prevents arbitrary identifiers. Prefer explicit deny-all or a separately named unrestricted internal mode. + +## Server-owned constraints + +Apply policy constraints before user input and make them impossible for the client to remove: + +```ts +const authority = and( + eq(account.organizationId, auth.organizationId), + ne(account.status, 'deleted'), +) + +const userPredicate = compileFilters(spec.filters, accountQueryFields) +const where = and(authority, userPredicate) +``` + +Never use a client-provided organization ID as authority after merely validating that it is a UUID. Check membership/permission and bind the authorized organization/resource pair in the same query where feasible. + +For SPARQL, “base patterns” must constrain the actual result graph. Adding a tenant triple pattern is insufficient if OPTIONAL/UNION/subquery structure lets data escape it. Test the generated query with adversarial data from two tenants. + +## SQL construction + +Compile normalized operators to the installed query builder/dialect: + +| Public operator | SQL concept | Edge contract | +|---|---|---| +| `eq` / `ne` | `=` / `<>` | null uses distinct null operator, not `= NULL` | +| range | `> >= < <= BETWEEN` | type coercion and inclusive semantics | +| `in` / `nin` | `IN` / `NOT IN` | empty list semantics; null behavior; max size | +| contains | escaped `LIKE`/`ILIKE` or full-text | wildcard escaping, collation, index support | +| null | `IS NULL` / `IS NOT NULL` | no client value | + +Use parameters for values. Registry-owned identifiers/expressions can be composed through the query builder. Do not interpolate arbitrary public fields, operators, sort fragments, table names, or loader column lists. + +Inspect generated SQL and parameter order. Test query plans for representative filter/sort combinations. Type compatibility does not prove an index is usable or that a cast avoids it. + +## SPARQL construction + +Separate: + +- term/variable/prefix construction; +- triple patterns and authoritative scoping; +- value serialization with datatype/language rules; +- filter expression construction; +- ordering and pagination; +- endpoint transport, timeout, and cancellation; +- SPARQL JSON result parsing; +- domain decoding and error mapping. + +Do not build `IN`, `between`, or cursor expressions by concatenating `.value` strings from a builder unless the library explicitly documents that as safe, correct serialization. The retained finance SPARQL executor uses `raw(...)` with joined expression values for several operators. That is valuable counterexample evidence: inspect the exact `@okikio/sparql` semantics and add malicious/typed literal tests before adopting it. + +Date values require correct RDF datatype and precision. A cursor value truncated to `YYYY-MM-DD` changes ordering if the field is actually `xsd:dateTime`. Tiebreaker comparisons through `STR(...)` can differ from numeric/IRI ordering. Match the cursor encoding and query term type exactly. + +Treat query preview text as sensitive. Redact credentials/endpoints and consider whether literals contain personal data. + +## Sorting and cursor pagination + +Cursor pagination requires one deterministic total order. If the public sort is not unique, append an immutable unique tiebreaker: + +```text +ORDER BY created_at DESC, id DESC +next page predicate: + created_at < :last_created_at + OR (created_at = :last_created_at AND id < :last_id) +``` + +Cursor payload should bind to the semantics it resumes: + +```ts +interface CursorV2 { + version: 2 + resource: 'accounts' + sort: Array<{ field: string; direction: 'asc' | 'desc' }> + boundary: Record + filterDigest: string + authorityDigest?: string + issuedAt: string + expiresAt: string +} +``` + +Sign or authenticate opaque cursors where clients must not tamper with boundaries. Canonicalize payloads before HMAC. Use constant-time signature comparison where the runtime provides it. Rotate secrets with a key/version identifier. Never include secret or private row data merely because the token is base64url encoded. + +The retained finance cursor includes one primary sort plus tiebreaker, direction, and creation time. It does not visibly bind the cursor to filter/resource context in the inspected schema. A cursor reused across filters can yield incorrect pages even with a valid HMAC. Add a context digest or enforce an equivalent server-side binding. + +Offset pagination can be appropriate for bounded admin views or snapshot-stable results. Define maximum offset and concurrent-write behavior. Do not present it as stable traversal under ongoing inserts/deletes. + +## Field selection + +Field selection affects storage cost and public data exposure. Define: + +- public-to-storage mapping; +- default selection; +- wildcard expansion semantics; +- always-required identity/tiebreaker fields; +- fields required for authorization/domain mapping but omitted from response; +- JSON:API resource-type handling if supported; +- relationship expansion separately from scalar selection; +- maximum selected fields. + +Do not treat `allowedFields: []` as unrestricted accidentally. Do not expose a new database column automatically through `*`. Expand wildcards against an explicit public registry. + +## Count strategies + +Name the strategy in code and response metadata: + +| Strategy | Meaning | Verification | +|---|---|---| +| Exact | Count of the same authoritative filtered relation | Execute count with identical base/user predicates | +| Planned | Planner estimate for the active query | `EXPLAIN`/planner output for that filtered query | +| Estimated | Coarse table/relation statistic | Document that filters may not be reflected | +| None | No total promised | Continuation/next cursor only | + +Do not label a relation estimate “total” if clients interpret it as exact filtered rows. Decide whether counts share the request snapshot with page rows. If not, document that concurrent writes can make them differ. + +## Resource and error policy + +Set and propagate: + +- database/endpoint timeout; +- request abort/cancellation; +- statement/result limits; +- pool wait timeout; +- query complexity caps; +- retry policy only for safe transient failures; +- redaction of SQL/SPARQL and values; +- stable domain/API error categories. + +Map expected failures such as invalid cursor, expired cursor, unsupported field, timeout, unavailable store, serialization retry exhaustion, and rejected query complexity. Preserve the cause in diagnostics without returning raw driver/query text to clients. + +## Integration sequence + +```text +decode query/json/form source + -> validate and normalize public query spec + -> resolve current authorization/tenant authority + -> map public fields/operators through storage registry + -> compile parameterized SQL/SPARQL/search request + -> execute with timeout/cancellation/resource bounds + -> decode rows/bindings and fetch one extra row if using cursor continuation + -> construct signed next/previous cursor from actual boundary rows + -> execute declared count strategy + -> shape stable response metadata +``` + +## Failure cases + +| Signature | Likely defect | Required correction | +|---|---|---| +| Cross-tenant row appears with valid input | Client filter substituted for authority | Server-owned predicate and adversarial fixture | +| Cursor repeats/skips equal timestamps | No stable or direction-consistent tiebreaker | Compound total order and concurrent pagination test | +| Valid cursor works with different filters | Cursor not bound to query context | Filter/resource/version digest | +| Search works but SQL fails for same operator | Shared public model overpromises dialect parity | Per-backend capability registry | +| SPARQL literal breaks query | Raw expression/value concatenation | Inspected serializer/builder and malicious literal tests | +| Exact count differs semantically from page | Predicates or snapshot differ | Shared authority/filter compiler and documented consistency | +| Query compiles but scans entire table | Type/cast/index mismatch | Representative `EXPLAIN` and performance bound | +| Empty allowlist exposes arbitrary columns | Empty interpreted as unrestricted | Explicit deny-all/unrestricted modes | + +## Test matrix + +Test: + +- every field/operator/type pair and forbidden pair; +- null, empty list, large list, malformed date/UUID/number, Unicode, wildcard characters; +- public field names that resemble SQL/SPARQL injection; +- server authority plus adversarial tenant/org filters; +- disabled features and ignored-parameter prevention; +- ascending/descending single and compound sorts; +- duplicate primary sort values and stable tiebreakers; +- insert/delete/update between pages; +- cursor tampering, expiry, key rotation, wrong endpoint/filter/sort/version; +- first/last/empty page, limit+1 continuation, previous-page semantics if supported; +- exact/planned/estimated/no-count response contracts; +- timeout, cancellation, unavailable engine, pool exhaustion; +- generated SQL parameter order and representative query plans; +- generated SPARQL escaping, datatypes, query endpoint behavior, and cross-tenant fixture; +- result decoding for nulls, decimals, dates, IRIs, language tags, and driver-specific values. + +## Executable verification + +Capture generated query text and parameters in tests through a safe test adapter, then run against disposable real engines. Use `EXPLAIN`/`EXPLAIN ANALYZE` only with controlled data and permissions. Replay a frozen dataset through every pagination mode and assert the set of IDs is complete with no duplicates. Execute malicious and cross-tenant inputs and require zero unauthorized results. + +For SPARQL, run queries against the selected QLever/Blazegraph/other engine; parser-level builder tests cannot establish engine extension or term-comparison behavior. + +## Deliberate exclusions + +- Do not require the retained finance query utilities or claim their exact names are published APIs. +- Do not force Zod; Standard Schema or another selected validator can own normalization. +- Do not force Drizzle, `@okikio/sparql`, PostgreSQL, or a graph engine. +- Do not allow a generic query DSL to claim operators a backend cannot implement faithfully. +- Do not use arbitrary public identifiers or raw fragments. +- Do not encode authorization solely in a client-visible cursor or projection. +- Do not promise stable pagination without a total order and context-bound cursor. +- Do not call planner/table estimates exact. + +## Sources and freshness + +Grounded in the retained new/old finance `utils/query` schemas, filtering, sorting, field-selection, cursor pagination, composite query factory, tests/benchmarks, `utils/execution/{db,sparql}.ts`, PopModern SPARQL/search endpoints, and `@okikio/sparql` consumer evidence, reviewed 2026-07-17. Several implementation choices are counterexamples requiring correction or proof, especially empty allowlists, cursor context binding, raw SPARQL composition, and datatype comparisons. Recheck public exports and engine/builder behavior at installed versions. diff --git a/skills/build-data/references/storage-ownership.md b/skills/build-data/references/storage-ownership.md index cc84646..9e5b31a 100644 --- a/skills/build-data/references/storage-ownership.md +++ b/skills/build-data/references/storage-ownership.md @@ -1,37 +1,242 @@ -# Storage ownership +# Storage ownership and authority -| Store | Strong default role | Required boundary | -|---|---|---| -| PostgreSQL | Transactional source of truth | Constraints, transactions, migrations, tenant policy | -| ClickHouse | High-volume analytical events and aggregates | Ordering, partitions, retention, deduplication, mutation cost | -| DuckDB | Local/in-process analytical queries and file transforms | File/schema ownership and memory/disk limits | -| Typesense | Search projection | Rebuild, alias/index replacement, lag and schema | -| QLever/Blazegraph | Graph/query projection | RDF identity, ontology provenance, engine deployment | -| JSONL | Streaming/replay/interchange evidence | Record schema, line safety, compression, manifests | -| Parquet | Typed analytical staging and batches | Schema evolution, row groups, partition layout | +Use this reference before introducing, removing, or integrating a database, search engine, graph store, cache, artifact format, or queue. The goal is not to assign one fashionable product per workload. The goal is to name the authority, guarantees, failure boundary, recovery path, and operational owner for every fact. -These roles are starting points, not universal mandates. Make deviations -explicit from workload evidence. +## Contents -## Authority and projection +- Evidence-first classification +- Authority map +- Store capability model +- Common store roles +- Cross-store consistency +- Tenant and privacy ownership +- Lifecycle and resource ownership +- Migration sequence +- Failure and recovery +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -Name one authoritative source for each fact. Search indexes, analytical tables, -materialized views, graph stores, and caches should have a rebuild or repair path. +## Evidence-first classification -Define the projection contract: +Inventory before deciding: ```text -authoritative commit - -> durable change identity - -> idempotent projector - -> projection checkpoint +manifests and lockfiles + -> runtime imports and client factories + -> schema/migration files + -> deployment manifests and environment bindings + -> writer call sites + -> reader/query call sites + -> worker/backfill/reconciliation code + -> production tasks and health checks +``` + +A README label is a discovery lead, not operational proof. The retained PopModern repository contains active QLever and Typesense infrastructure, PostgreSQL/Supabase migrations, ClickHouse scaffolding, older graph/export artifacts, and documentation from different phases. Determine which components actually start, receive writes, and serve reads. + +For each fact/domain, record: + +| Question | Required answer | +|---|---| +| Who accepts the authoritative write? | Named store/table/object plus transaction boundary | +| What invariant is guaranteed there? | Constraint, isolation, append identity, or documented absence | +| Who serves reads? | Direct authority or named projection/cache | +| What lag is allowed? | Objective and measurement | +| How are corrections/deletes propagated? | Change identity, rebuild, tombstone, mutation, or repair | +| How is state recovered? | Backup/restore, replay, rebuild, reconciliation | +| Who owns schema and lifecycle? | Package/service/team/process | + +## Authority map + +Use one authority per fact, not necessarily one store per domain: + +```yaml +facts: + account-membership: + authority: postgres.auth.member + write-owner: identity-service + projections: + - typesense.account-search + invariants: + - unique organization_id,user_id + - current membership checked on protected reads + raw-provider-event: + authority: object-storage/import-run/raw + write-owner: import-worker + identity: provider + event_id + payload_sha256 + analytics-event: + authority: clickhouse.events + write-owner: event-ingest + correction: versioned replacement with reconciliation +``` + +“PostgreSQL is primary” is too vague. A user row can be authoritative in PostgreSQL while a provider payload is authoritative in immutable raw capture and an analytical event is intentionally authoritative in ClickHouse. State the granularity. + +## Store capability model + +Evaluate the installed system against the actual workload: + +- write model: transactional, append, bulk, streaming, mutation, replacement; +- consistency and isolation; +- uniqueness, constraints, and referential integrity; +- query/access patterns and indexes; +- schema evolution and migration ownership; +- concurrency and conflict behavior; +- partitioning/sharding/tenant isolation; +- backup, restore, replay, and point-in-time recovery; +- retention, deletion, and legal/privacy obligations; +- connection/resource lifetime; +- observability, failure injection, and operator repair; +- local development and CI parity. + +Do not infer capabilities from a familiar method shape. A custom adapter exposing `select().from()` does not imply transactions, relational queries, `RETURNING`, Drizzle Kit migrations, prepared statements, or PostgreSQL semantics. + +## Common store roles + +These are strong starting hypotheses, not mandates: + +| Store | Common role | Must define | +|---|---|---| +| PostgreSQL | Transactional identities, relationships, workflow control state | constraints, isolation, migration, tenant policy, pool lifetime | +| ClickHouse | High-volume events, time-range analytics, aggregates | ordering, partitions, batches, late data, correction convergence, TTL | +| DuckDB | Local/in-process analytics and file transformation | file ownership, concurrency, memory/disk limits, extension policy | +| Typesense/search engine | Rebuildable search projection | document schema, ranking, filters, aliases, lag, deletion, rebuild | +| QLever/graph engine | RDF/SPARQL serving projection | term/ontology identity, complete dataset, index build, update/rebuild policy | +| Object storage | Immutable raw evidence and artifacts | object identity, publication manifest, retention, encryption, delete policy | +| JSONL | Streaming replay/interchange segments | line schema, committed offsets, corruption/compression policy | +| Parquet | Typed analytical staging/batches | explicit schema, row groups, partitions, reader compatibility | +| Queue | Delivery of work, not business state by default | delivery/ordering, visibility/lease, retry/DLQ, deduplication, redrive | + +Avoid the “one database for everything” reflex and the opposite “one product per feature” reflex. Operational cost, recovery complexity, and dual-write risk are first-class selection evidence. + +## Cross-store consistency + +Cross-store writes are not atomic unless an inspected protocol proves they are. Prefer: + +```text +transactional authority commit + -> durable outbox/change record in same commit + -> projector leases change + -> idempotent sink write + -> durable sink receipt/checkpoint -> lag and failure visibility - -> reconciliation/rebuild + -> reconciliation/repair ``` -## Selection evidence +For an immutable artifact source, the final manifest can play the durable handoff role. For an external provider, persist the normalized event identity before starting projections. + +Define what happens in every crash window: + +- authority commit succeeds, change publication fails; +- sink write succeeds, checkpoint fails; +- checkpoint succeeds prematurely; +- projector applies events out of order; +- one of several required sinks fails; +- a privacy deletion reaches some stores but not others. + +Exactly-once is rarely the useful claim. Require at-least-once delivery plus idempotent/version-aware application and reconciliation unless a narrower guarantee is proved end to end. + +## Tenant and privacy ownership + +Tenant authority belongs in server-controlled policy and the authoritative schema: + +- include tenant/organization identifiers on owned rows where appropriate; +- use composite unique/foreign-key shapes when relationships must remain within a tenant; +- apply tenant predicates before user filters; +- verify authorization at the authoritative store, not from a search document alone; +- partition or row-level security only when the operational model owns it correctly; +- propagate tenant moves and membership revocations explicitly; +- inventory every projection and artifact affected by deletion/export requests. + +The retained finance schema uses organization identifiers and composite keys across `auth`, `finance`, and `workflows`. That is useful observed design evidence, but it does not prove every query enforces organization policy. Inspect query call sites and request authorization separately. + +## Lifecycle and resource ownership + +For every runtime client, identify: + +- construction owner (composition root, worker boot, request scope, test); +- underlying pool/socket/file handle; +- maximum connections/concurrency and timeout behavior; +- readiness/health semantics; +- shutdown and drain method; +- cancellation of in-flight requests; +- test replacement/fake boundary. + +The retained finance `createDatabase()` creates a `postgres.js` client and returns only the Drizzle wrapper. That shape can obscure `client.end()` from the composition root. A production design can return `{ db, client, close }`, accept an externally owned client, or otherwise make shutdown reachable. Do not claim graceful shutdown from a wrapper type alone. + +Global logging or environment reads are separate owners. Database construction must not silently configure the application's logger or require environment access at import time. Preserve the consumer's selected owners. + +## Migration sequence + +When changing ownership: + +1. Document current writer/readers and recovery evidence. +2. Define the future authority and invariant contract. +3. Add durable change capture or a named snapshot boundary. +4. Backfill a versioned target. +5. Reconcile identities, content, tenant policy, and domain invariants. +6. Dual-read or shadow-query when it produces useful evidence. +7. Cut over readers with rollback available. +8. Cut over the single authoritative writer. +9. Observe lag/errors and repair gaps. +10. Remove old writes/read paths only after connected consumers and recovery are verified. + +Avoid prolonged dual-authority writes. If both systems accept independent changes, conflict resolution and ownership are unresolved. + +## Failure and recovery + +| Signature | Likely ownership defect | Correction evidence | +|---|---|---| +| Search result grants access after membership revocation | Projection treated as authorization authority | Current membership read plus server-owned source query | +| PostgreSQL and Typesense differ after request success | Direct dual write has no durable handoff | Outbox/checkpoint and reconciliation | +| ClickHouse is expected to reject duplicate business IDs | OLTP invariant assigned to analytical store | Upstream constraint/idempotency plus convergence model | +| README says Blazegraph, deployment starts QLever | Store selected from documentation label | Active manifests, endpoint, query test | +| Drizzle wrapper exists but process never exits | Pool lifetime owner hidden | Reachable close/drain and in-flight shutdown test | +| Restore brings DB back but search is stale | Recovery stops at authority | Projection rebuild/catch-up runbook | +| Queue says delivered but workflow has no durable state | Delivery confused with acceptance | State transaction/outbox and idempotent consumer | +| Two stores both called source of truth | Conflict policy absent | Fact-level authority map | + +## Test matrix + +Test: + +- constraint enforcement under concurrent writes; +- server-owned tenant filters and cross-tenant attack inputs; +- source commit followed by projector crash in each commit window; +- duplicate and out-of-order change delivery; +- full backup/restore followed by projection rebuild/catch-up; +- privacy delete/export across authority, projections, caches, and artifacts; +- schema migration from representative previous versions; +- empty install and seed idempotency; +- pool exhaustion, database restart, timeouts, and graceful shutdown; +- projection lag and alert thresholds; +- target cutover/rollback during active reads; +- reconciliation with missing, extra, and stale-version records; +- explicit behavior of unsupported adapter capabilities. + +## Executable verification + +Build a storage inventory from code and deployment, then run store-native checks. Examples to adapt: + +```bash +rg -n 'postgres\(|drizzle\(|createClient|Typesense|QLever|ClickHouse|Parquet' . +rg -n 'DATABASE_URL|TYPESENSE|QLEVER|CLICKHOUSE' . --glob '!**/*.lock' +``` + +Apply migrations to an empty disposable database and upgrade from a representative snapshot. Execute concurrent invariant tests, issue real projection queries, compare identity/version sets, restart processes, and prove close/drain behavior. A typecheck or client-construction test is not storage verification. + +## Deliberate exclusions + +- Do not mandate a store solely from the table above. +- Do not force PostgreSQL for raw immutable evidence or ClickHouse for small relational workloads without evidence. +- Do not add a queue or outbox when the projection is safely rebuilt from immutable snapshots and the product contract allows that recovery. +- Do not call a cache/search/graph store authoritative by convenience. +- Do not infer transaction or migration support from Drizzle-like syntax. +- Do not force Zod, LogTape, Effect, or any specific configuration/observability framework. +- Do not hide a client's lifecycle behind a wrapper with no close path. +- Do not claim exactly-once, zero-downtime, or point-in-time recovery without executable system evidence. + +## Sources and freshness -Inspect actual deployment and query code. A README can call Blazegraph primary -while the active pipeline serves QLever. A custom adapter can resemble Drizzle -without supporting transactions or Kit migrations. Names are not operational -proof. +Grounded in the retained new/old finance database package, generated Drizzle migration and cross-schema finance/workflow schemas, finance query utilities, PopModern data pipeline, QLever/Typesense/ClickHouse infrastructure, and projection code, reviewed 2026-07-17. The finance and PopModern repositories contain current, legacy, and prototype paths; use them to identify real contracts and counterexamples, not as proof of the consumer's deployment. Recheck database, engine, driver, and adapter guarantees at installed versions. diff --git a/skills/build-devtools/SKILL.md b/skills/build-devtools/SKILL.md index 8b5323c..553419f 100644 --- a/skills/build-devtools/SKILL.md +++ b/skills/build-devtools/SKILL.md @@ -6,9 +6,10 @@ description: Design, integrate, migrate, review, diagnose, or verify developer t # Build developer tools Map the repository's toolchain before changing it. When active, `deno-software` -owns Deno configuration and publication and `build-clis` owns CLI product -behavior. Otherwise preserve those checks locally. This skill owns developer -workflow, generation, packaging automation, and release evidence. +owns Deno configuration and publication, `build-clis` owns CLI product behavior, +and `build-libraries` owns the reusable public API, entrypoint partitioning, and +selective-adoption contract. Otherwise preserve those checks locally. This skill +owns developer workflow, generation, packaging automation, and release evidence. ## Toolchain ownership map @@ -38,6 +39,9 @@ mirrored tasks with different semantics. - [toolchains.md](references/toolchains.md): Mise, Aube, tasks, manifests, lockfiles, CI, editors, and ownership. +- [mise-aube.md](references/mise-aube.md): load for detailed Mise and Aube + configuration, tool/runtime/task ownership, lockfiles, workspaces, security, + lifecycle-build jails, CI, migration, and rollback. - [generated-artifacts.md](references/generated-artifacts.md): check/write, provenance, deterministic generation, drift, and safe formatting. - [packaging.md](references/packaging.md): cross-runtime builds, exports, diff --git a/skills/build-devtools/references/generated-artifacts.md b/skills/build-devtools/references/generated-artifacts.md index 47cba77..df5f5c2 100644 --- a/skills/build-devtools/references/generated-artifacts.md +++ b/skills/build-devtools/references/generated-artifacts.md @@ -1,38 +1,363 @@ # Generated artifacts -## Required generator contract +## Contents -- default read-only check mode; -- explicit `--write` or equivalent mutation mode; -- deterministic output and stable ordering; -- pinned source identity, URL, version, and digest where external data is used; -- explicit network/read/write permissions; -- semantic validation of inputs and outputs; -- atomic or safely replaceable writes; -- stale-output CI gate; -- no unrelated file or Markdown formatting. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Ownership model](#ownership-model) +- [The generator contract](#the-generator-contract) +- [External-source acquisition](#external-source-acquisition) +- [Transforming authored files](#transforming-authored-files) +- [Generated Markdown](#generated-markdown) +- [Atomic writes and crash recovery](#atomic-writes-and-crash-recovery) +- [Review and CI design](#review-and-ci-design) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Executable verification](#executable-verification) +- [Sources and freshness](#sources-and-freshness) -## Mutable upstream sources +## When to load this reference -If an upstream `latest` URL is convenient, resolve it to an immutable version and -compare payload hashes before trusting it. Record both source identities and the -generator revision in generated output or a manifest. +Load this reference when code, schemas, clients, Unicode tables, documentation +sections, manifests, lock-derived files, fixtures, completion scripts, or other +artifacts are produced from another source. Also load it when a task says +"regenerate", "sync", "update the snapshot", or "fix drift". Do not treat a +file as generated merely because it looks repetitive; establish the producer +and source first. -## Source-aware updates +## Outcome -Prefer AST or structured-data edits when updating code/config constants. Preserve -comments and authored layout where they are part of reviewability. Check mode must -exit nonzero on drift without rewriting files; write mode should converge so the -second check is clean. +Produce a generator for which all of the following statements are true: -## Generated ownership +- one named input is authoritative for every output field; +- check mode observes drift without mutating the worktree; +- write mode changes only owned regions and converges in one pass; +- identical source bytes, tool versions, options, and platform policy yield + byte-identical output; +- mutable remote names are resolved to immutable identities and verified; +- malformed, incomplete, or surprising input fails before the destination is + replaced; +- the artifact records enough provenance to reproduce or audit the update; +- CI and reviewers can distinguish authored changes from regeneration noise. -Mark generated files and point to their generator. Decide whether they are -committed, produced during build, or release-only. Verify clean-tree regeneration -and fail when manual edits would be overwritten. +Generation is a small compiler pipeline, not a convenient file-copy command: -## Failure tests +```text +locate source + -> acquire exact bytes + -> authenticate identity and integrity + -> parse into an internal model + -> validate semantics and invariants + -> render deterministically + -> compare with current output + -> check or atomic replace + -> validate the consumer +``` -Exercise unavailable network, digest mismatch, malformed upstream data, unknown -schema version, partial write, permission denial, stale output, and deterministic -repeat generation. +## Ownership model + +Classify every relevant file before writing code. + +| Class | Authority | May humans edit it? | Required check | +|---|---|---:|---| +| Authored source | repository author | yes | normal validation | +| Generated source | named generator input | no, except through input/generator | drift plus consumer tests | +| Mixed authored/generated | author outside bounded markers | only outside owned regions | marker integrity and minimal diff | +| Build output | source plus build configuration | no | clean rebuild/package check | +| Release artifact | immutable revision plus release inputs | no | digest/provenance/consumer check | +| Snapshot/fixture | test owner and update command | only by explicit review | semantic test plus snapshot diff | +| Vendored source | upstream revision plus local patch policy | only through vendor workflow | license, digest, patches, build | + +Record the classification near the task, manifest, or file header. A generated +header should name the generator and source, but must not include a wall-clock +timestamp unless time is part of the product contract; timestamps destroy +reproducibility without proving freshness. + +Mixed ownership is a risk boundary. Prefer a separate generated file imported +by an authored file. If the output must share a file with authored prose, use +unique, non-nesting start/end markers and reject missing, duplicated, reversed, +or overlapping markers. Never replace text between a pair of loose regex +matches without proving that the pair identifies exactly one owned region. + +## The generator contract + +### Check and write are distinct modes + +The safe default is check mode. Mutation requires an explicit `--write`, +`--update`, or equivalent option. Both modes run the same acquisition, parse, +validation, and render functions; only the final action differs. + +```ts +type Mode = "check" | "write"; + +const source = await acquirePinnedSource(signal); +const model = parseAndValidate(source.bytes); +const next = render(model); // stable sort, stable newline, no current time +const current = await readText(output); + +if (next === current) Deno.exit(0); +if (mode === "check") { + reportSemanticDrift(current, next, source.provenance); + Deno.exit(1); +} + +await atomicReplace(output, next); +``` + +Do not implement check mode by running write mode and asking Git whether the +tree changed. That mutates user files, may trigger formatters/watchers, loses +the original failure state, and is unsafe in a dirty worktree. + +Required properties: + +- the mode and target paths are visible in `--help`; +- read, write, network, environment, and subprocess permissions are narrower + in check mode than in write mode; +- every input is explicit: file, environment name, URL, tool version, locale, + timezone, and feature option; +- output uses a fixed encoding, newline policy, ordering, numeric rendering, + and path normalization; +- the second check after a write is clean; +- a second write produces no byte change; +- failures leave the old output usable. + +### Semantic validation precedes rendering + +Parsing successfully is not enough. Check domain invariants such as unique +identifiers, sorted non-overlapping ranges, referential integrity, version +compatibility, required exports, bounded sizes, and expected record counts. +Prefer a typed intermediate model that cannot represent an unchecked row. + +For data updates, compare semantic sets as well as rendered text. A different +range compression or property order may be textually different but semantically +equal; an equal record count may still hide a replacement. Reports should show +missing/extra keys and representative samples, not only "generated file differs". + +The attached Undent Unicode generator demonstrates the stronger pattern: it +resolves the Unicode version from `latest/ucd/ReadMe.txt`, downloads both the +mutable latest and matching versioned `EastAsianWidth.txt`, compares SHA-256, +parses only relevant properties, checks range drift, validates the target module, +and separates check from `--write`. Reuse the method, not its package-specific +constants. + +## External-source acquisition + +Treat every network source as untrusted input and every mutable alias as a +discovery pointer. + +1. Fetch the small identity document or registry metadata with a timeout and + bounded response size. +2. Parse and validate the advertised version/ref. +3. Construct or discover an immutable URL: exact version, tag, commit, digest, + or registry integrity. +4. Fetch the immutable object. +5. If a mutable and immutable object are both available, compare their bytes + or digests before trusting the mutable response. +6. Store identity, URL, digest, generator revision, schema version, and relevant + options in a manifest or generated comment. +7. Parse from bytes already verified; do not refetch during rendering. + +`latest`, a branch name, a floating template registry entry, or a package range +is not provenance. For Git/template acquisition with Giget 3.3.0, prefer an +exact tag or commit and record the resolved provider, source, subdirectory, and +archive digest. `--offline` proves cache availability, not that the cache is the +requested revision. `--force-clean` is destructive and requires destination +classification and authorization before use. Authentication headers must never +be emitted in generated manifests or error logs. + +Support failure injection for timeout, non-2xx response, oversized response, +redirect to an unexpected host, digest mismatch, unsupported upstream version, +malformed rows, duplicated keys, empty source, and truncated transfer. A stale +cache may be used only when the product defines a stale/offline policy and the +output records which source was used. + +## Transforming authored files + +Use the narrowest transformation mechanism that preserves ownership. + +| Input | Preferred mutation | Avoid | +|---|---|---| +| JSON with no comments | parse, validate, update owned keys, stable serialize | global text replacement | +| JSONC/TOML/YAML/package manifest | format-aware editor or bounded structured patch | dropping comments/unknown keys | +| static-ish JS/TS config | Magicast or a precise AST transformation with fallback | evaluating config to recover syntax | +| arbitrary program | codemod with syntax/type guards and fixtures | pretending it is JSON | +| Markdown | explicit markers with a pure section generator | repository-wide formatting | +| binary/media | content-addressed replace plus metadata validation | in-place partial mutation | + +Magicast 0.5.3 provides `loadFile`, `writeFile`, proxy-like access to imports, +exports, object literals, and function-call arguments, and a `core` entrypoint +without filesystem helpers. It is designed for static-ish JavaScript; dynamic +spread expressions, computed values, branches, and arbitrary calls can throw or +be outside its model. Detect supported shapes, preserve the original source, +and return a manual patch instruction when the shape is unsupported. + +```ts +import { loadFile, writeFile } from "magicast"; + +const module = await loadFile("build.config.ts"); +const exported = module.exports.default; +const config = exported.$type === "function-call" + ? exported.$args[0] + : exported; + +if (!config || typeof config !== "object") { + throw new Error("Unsupported build config shape; update entries manually"); +} + +config.entries ??= []; +if (!config.entries.includes("./src/worker")) { + config.entries.push("./src/worker"); +} +await writeFile(module, "build.config.ts"); +``` + +Never load an executable JS/TS configuration simply to edit it. Runtime loading +executes user code and converts syntax/comments into values that cannot be +round-tripped. An AST tool is not automatically safe either: require a bounded +input grammar and test comments, quote style, wrappers such as `defineConfig`, +imports, spreads, unsupported expressions, and idempotence. + +## Generated Markdown + +Generated Markdown must remain reviewable. Automd 0.4.3 recognizes bounded +comment directives and supports built-in/custom generators, pure transformation, +output, and watch workflows. It is appropriate for badges, contributor lists, +typed references, fetched snippets, or other explicitly owned sections. It is +not authority for prose around those markers, and it does not justify formatting +the entire file. + +```md + + +The generated section lives here. + + +``` + +Pin fetched content. Run an observation-only diff/check in CI. Inspect the +actual 0.4.3 CLI/help and configured generators before copying commands because +the marker language and custom-generator API are versioned surfaces. A custom +generator should accept parsed options and repository context, return text, and +have tests independent of file writing. + +Do not: + +- generate prose that encodes decisions no machine source owns; +- nest marker regions; +- permit a remote snippet to escape its marker; +- rewrite code fences, lists, tables, or wrapping outside the owned region; +- silently regenerate during install, tests, or documentation viewing; +- accept a generated table because its row count stayed constant. + +## Atomic writes and crash recovery + +Render and validate the complete next artifact before touching the destination. +For a single file, write a temporary sibling, flush when durability matters, +apply intended permissions, then rename on the same filesystem. For a generated +directory, build in a fresh staging directory, validate its manifest and +consumers, then swap or replace according to a documented recovery protocol. + +Atomic rename prevents partial bytes but does not make a multi-file update +transactional. For several outputs, use a manifest-last protocol: + +1. write content-addressed or versioned files; +2. verify all of them; +3. atomically replace the manifest/pointer last; +4. garbage-collect old generations separately. + +If files must be replaced in place, record a journal and test interruption after +every step. Never delete the old generated directory before the new one passes +validation; the Undent npm build can safely clear its ignored `npm/` directory +because it is disposable build output, while committed source generation needs +stronger preservation. + +## Review and CI design + +A reviewable generator change normally contains: + +- the source/model change; +- generator implementation change if required; +- generated diff; +- provenance update; +- generator unit/property/failure tests; +- target-consumer verification; +- an explanation of surprising additions, removals, or reorderings. + +CI should run the repository's true check command with only required permissions. +For a committed artifact: + +```sh +deno task generate:check +deno task test:generated-consumer +git diff --exit-code -- path/to/generated +``` + +The last command is a defense in depth, not the implementation of check mode. +Do not run `deno fmt` or Prettier across Markdown merely to make a generated +section pass. Scope code formatting to generated code or render already-formatted +bytes. In dirty worktrees, compare only owned paths and refuse write mode when +an output contains unrelated user edits unless an explicit merge is supported. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Check rewrites files | write path reused for observation | mode branch and permissions | +| Second write changes output | nondeterministic order, time, locale, random ID, absolute path | byte diff and complete input inventory | +| CI passes but clone fails | undeclared local tool/cache/input | clean clone with empty caches | +| Latest and versioned digests differ | alias advanced mid-run, mirror error, compromised response | identity fetch and immutable URL | +| Comments disappear from config | value serialization replaced syntax | structured/AST editor boundary | +| Generated section consumes following prose | marker missing/duplicated or greedy parser | marker cardinality and span tests | +| Partially written source after failure | direct destination write | staging/rename protocol | +| Generator reports no drift but consumer fails | textual comparison without semantic validation | output schema and real consumer test | +| Huge unrelated Markdown diff | global formatter or whole-file renderer | owned-region diff and formatter scope | +| Offline run uses wrong template | cache key omitted revision or subdir | cache identity and recorded provenance | + +## Deliberate exclusions + +- Do not add a generator when a small stable file is clearer to author and test. +- Do not fetch mutable remote content during ordinary library import, package + install, help generation, or completion. +- Do not make the generated file a second editable source of truth. +- Do not use a snapshot update to approve a behavioral change without reading + the semantic diff. +- Do not call an output reproducible if the toolchain, locale, platform inputs, + or remote bytes are unrecorded. +- Do not adopt Automd for authored prose, Magicast for arbitrary dynamic code, + or Giget when a registry/package artifact already supplies stronger integrity. + +## Executable verification + +Run the applicable subset and record exact commands/results: + +1. check mode against the committed state; +2. copy the repository or fixture, run write, then check; +3. hash output, run write again, and compare the hash; +4. generate under a different temporary absolute path and compare bytes; +5. randomize source enumeration order and assert stable output; +6. deny write permission in check mode and confirm it still works; +7. inject malformed input, timeout, digest mismatch, and interrupted replacement; +8. prove unsupported AST/config shapes remain unchanged with an actionable error; +9. run the compiler/importer/package/docs consumer of the generated artifact; +10. inspect `git diff --word-diff` or a semantic report for owned paths only. + +For a remote source, retain the immutable input or at least its digest and +identity metadata. For release generation, also test from a clean checkout of +the tagged commit so untracked local files cannot satisfy the generator. + +## Sources and freshness + +- Attached `undent.zip`, observed source: `scripts/sync_unicode_east_asian_width.ts` + and `scripts/build_npm.ts`; verified in the retained archive on 2026-07-17. +- Automd 0.4.3 published README, declarations, and package manifest; current + source record `automd-0-4-3`, verified 2026-07-17. +- Magicast 0.5.3 published README, exports, and package manifest; current source + record `magicast-0-5-3`, verified 2026-07-17. High-level helpers are explicitly + experimental and require source/test inspection at the installed version. +- Giget 3.3.0 published README and package manifest; source record + `giget-3-3-0`, verified 2026-07-17. +- Attached production CLI guidebook v1.1, normative generator, generated-doc, + packaging, permission, and release requirements; verified 2026-07-13. + +Recheck installed exports and CLI help before copying an API. The versioned +records above describe the pinned artifacts, not all future releases. diff --git a/skills/build-devtools/references/hygiene.md b/skills/build-devtools/references/hygiene.md index 6e01f1f..b3ba64c 100644 --- a/skills/build-devtools/references/hygiene.md +++ b/skills/build-devtools/references/hygiene.md @@ -1,36 +1,280 @@ -# Repository hygiene +# Repository hygiene and retained artifacts -## Classify before deleting +## Contents -Classify suspicious files as authored source, generated source, fixture, vendored -dependency, cache/download, build output, release artifact, or unknown. Check -repository instructions and consumers before removal. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Classify before changing](#classify-before-changing) +- [Dirty worktree protocol](#dirty-worktree-protocol) +- [Caches and build outputs](#caches-and-build-outputs) +- [Binaries and vendored dependencies](#binaries-and-vendored-dependencies) +- [Generated and archived material](#generated-and-archived-material) +- [Secrets, personal data, and databases](#secrets-personal-data-and-databases) +- [Workspace and task integrity](#workspace-and-task-integrity) +- [Reviewability and Markdown](#reviewability-and-markdown) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Executable verification](#executable-verification) +- [Sources and freshness](#sources-and-freshness) -Common local-only artifacts include editor-managed tool binaries, package caches, -coverage, temporary databases, generated preview images, and framework build -directories. Intentional vendoring requires version, source, license, integrity, -update, and platform policy. +## When to load this reference -## Dirty worktrees +Load this reference for repository cleanup, large-file review, cache/vendor/build +classification, generated outputs, accidental binaries, archive creation, dirty +worktrees, secret scanning, missing tasks, or "remove cruft" requests. A review +or diagnosis authorizes inspection, not deletion. -Preserve user changes. Inspect status and diffs before generation, formatting, -or cleanup. Never overwrite a modified generated file without determining -whether it is an intentional source change or stale output. +## Outcome -## Reviewability +Leave a repository whose tracked and distributed contents have explicit owners, +reproducible acquisition/build/update paths, appropriate licensing/integrity, +and no accidental local state. Preserve user work and review history. Report +unknowns instead of deleting files based on name or size. -Do not run repository-wide formatters over Markdown. Scope code formatters by -extension/path or configure a Markdown exclusion. After changes, inspect -Markdown numstat and diffs for unrelated wrapping, table, or code-block churn. +Hygiene is not a small repository at any cost. Fixtures, corpora, snapshots, +generated tables, native binaries, and vendored code can be intentional. The +goal is explained ownership and safe lifecycle. -## Hygiene checks +## Classify before changing -- ignored and untracked large files; -- secrets and environment files; -- executable/binary provenance; -- generated files without owners; -- stale tasks and permission sets; -- duplicate dependency versions; -- workspace globs pointing to missing directories; -- docs claiming scripts or artifacts that do not exist; -- clean-clone setup and clean-tree regeneration. +For every suspicious path classify: + +| Class | Evidence | Normal action | +|---|---|---| +| Authored source/docs | imports, manifest, history, instructions | preserve; normal review | +| Generated committed source | generator/header/task, consumer | preserve; add drift/update contract | +| Ignored build output | build task and clean reproduction | remove from deliverable; rebuild as needed | +| Cache/download | cache owner and invalidation key | ignore/remove locally; never package | +| Fixture/corpus/snapshot | named tests and update policy | preserve if needed; size/license review | +| Vendored dependency | upstream revision, license, patches | preserve only with vendor policy | +| Release artifact | release identity/digest/retention | store in release system, not casually in Git | +| Local tool binary | editor/tool installer, platform-specific | ignore/remove from artifact unless vendoring intentional | +| Secret/personal/runtime data | content/schema/location | stop exposure; remediate according to policy | +| Unknown | insufficient evidence | quarantine/report; do not delete | + +Inspect repository instructions, Git status/history, ignore rules, manifests, +imports, task references, workflows, package `files`, Docker contexts, deployment +config, and release scripts. Use content/magic, size, mode, and digest—not only +extension. A file named `data.db` may be a required test fixture or leaked +production-like state; a file named `tool` may be source or an 84 MB executable. + +Before calling two archives different versions, compare normalized relative +paths and hashes. The retained `old-finance` and `new-finance` archives contain +identical source content despite different archive identities; inventing an +evolution story would be a provenance error. + +## Dirty worktree protocol + +1. Record `git status --short`, branch/revision, staged/unstaged/untracked paths. +2. Read diffs for files in scope and distinguish user changes from the requested + work where possible. +3. Identify commands that can mutate broadly: formatters, generators, installers, + package managers, migrations, build tools, and cleanup scripts. +4. Scope commands to owned paths or use isolated copies/worktrees for destructive + observations. +5. Never overwrite a modified generated file before determining whether the + input/generator or output was intentionally edited. +6. After each mutation, recheck status and attribute new changes. + +Do not stash, reset, checkout, clean, or delete user changes without explicit +authorization. `git clean`, recursive deletion, builder `--force-clean`, and +package-manager prune commands can remove untracked work. Prefer listing/dry-run +and targeted paths. If clean-state evidence is required, copy/archive tracked +files or use an authorized temporary worktree rather than normalizing the user's +tree. + +## Caches and build outputs + +Common local-only candidates include: + +- `node_modules`, Deno/npm/pnpm/Aube caches and global stores; +- framework/build directories, coverage, temporary declarations/maps; +- `.DS_Store`, editor state, preview screenshots, local logs; +- temporary SQLite/ClickHouse/Postgres volumes and test run artifacts; +- compiled binaries downloaded by editors/tool managers; +- benchmark scratch reports not selected for the permanent study record. + +Classify by owner/invalidation. A cache key should include every input that makes +entries valid: platform/architecture, runtime/tool version, lock digest, build +flags, source digest, and sometimes environment. A cache restore is an +optimization, never authority. CI must be able to rebuild with empty caches and +must not publish cache contents accidentally. + +Ignore rules should be specific enough not to hide authored source. Test them: + +```sh +git check-ignore -v path/to/suspect +git ls-files --error-unmatch path/to/suspect +``` + +Review Docker/package/archive contexts separately from Git tracking. A Gitignored +secret or cache can still enter an image when `.dockerignore` is wrong, or a ZIP +when the archive command takes the filesystem rather than tracked files. Prefer +`git archive` or an explicit staging manifest for source deliverables. + +## Binaries and vendored dependencies + +Intentional binary/vendored material needs: + +- exact upstream project/version/revision and download URL; +- cryptographic digest/signature and verification procedure; +- license, notices, source-offer obligations, and security owner; +- platform/architecture/libc/runtime matrix; +- update, vulnerability response, rollback, and end-of-life policy; +- reason package-manager/system acquisition is insufficient; +- tests that load/execute the shipped artifact on supported targets. + +Do not commit a downloaded tool merely to make one editor or sandbox work. Use +Mise/Aube/runtime setup or a documented bootstrap with integrity. If offline or +supply-chain constraints justify vendoring, store a manifest and scripts rather +than an unexplained binary. Never execute an unverified binary during inspection. + +Git LFS can store large tracked objects but does not supply provenance, license, +platform coverage, or reproducible update policy. Submodules preserve a Git +revision but still need trust, availability, recursive checkout, and packaging +rules. Vendored generated code may require both upstream and generator identity. + +## Generated and archived material + +Every committed generated file must name or be mapped to: + +- source inputs and their version/digest; +- generator command/tool version; +- check/write policy; +- semantic and consumer verification; +- whether formatting is owned by the generator; +- update and review procedure. + +Archive/package creation should begin from an explicit manifest or tracked +revision, not `zip -r .`. Exclude `.git`, caches, `node_modules`, credentials, +local databases, test temp, editor binaries, prior archives, SkillOpt run data, +and internal evidence unless intentionally part of the deliverable. Verify ZIP +integrity and list its contents/large files before handoff. + +Generated experiment artifacts are evidence only if tied to source/harness and +the protocol. Retain raw samples used by conclusions; remove unowned exploratory +scratch. Do not delete a rejected-candidate report that prevents repeated work. + +## Secrets, personal data, and databases + +Stop and scope remediation when discovering credentials or sensitive data. + +- Do not print full values while diagnosing; report path, key name, and bounded + fingerprint when useful. +- Determine whether it is tracked, committed in history, packaged, published, + or only local. +- Rotate/revoke exposed credentials through authorized systems; deleting the + file alone is not remediation. +- Preserve evidence needed for incident response without duplicating secrets. +- Add `.env.example` with names and safe placeholders, not real values. +- Treat database files, request traces, benchmark corpora, screenshots, and logs + as potentially personal/confidential even if no key pattern matches. + +History rewriting and remote deletion are destructive, coordinated actions; +never infer authorization from a cleanup request. A secret scanner pass lowers +risk but does not prove absence. Review generator/release logs and artifact +contents too. + +## Workspace and task integrity + +Hygiene includes connected contracts: + +- workspace globs resolve to intended packages and exclude output/evidence; +- each package has one dependency/version owner and compatible lock graph; +- scripts/tasks referenced by README, CI, prepack, release, or editor config + exist and use current options; +- permission lists cover actual imports/files/hosts/subprocesses without blanket + access; +- package export/files maps contain required assets and exclude internals; +- generated files have producers; generators have consumers; +- duplicate dependency versions are either intentional compatibility boundaries + or candidates for alignment; +- dead config is proven unused across local, CI, package, container, deploy, + docs, and developer environments before removal. + +Use clean-clone setup and a complete lifecycle (`install -> check -> build -> +pack -> consumer`) to find undeclared local dependencies. An unused search hit is +not proof a file is dead when a framework, manifest, dynamic loader, or release +workflow discovers it conventionally. + +## Reviewability and Markdown + +Do not run repository-wide Markdown formatting during code, generator, cleanup, +or skill-document work unless the user explicitly requests it. Wrapping, table +alignment, list normalization, heading changes, and code-fence formatting hide +substantive documentation edits and can damage marker directives. + +Scope code formatting by extension/path or configure Markdown exclusions. After +editing: + +- inspect Markdown `git diff --stat`/`--numstat` and `git diff -w`; +- review every changed Markdown hunk; +- confirm generated markers/code fences are balanced; +- check links without rewriting prose; +- ensure no unrelated newline/wrapping-only churn. + +Do not normalize line endings repository-wide. Preserve an authored file's +style unless the task owns that migration and reviewers can isolate it. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| ZIP unexpectedly huge | cache, binary, database, nested archive | sorted archive size listing | +| Clean clone cannot build | untracked input/global tool/cache | task/input and toolchain inventory | +| Deleted file returns after build | generated output treated as cruft | producer/consumer ownership | +| Gitignored file appears in container/package | separate context/include rules | Docker/package/archive manifest | +| Binary works only on one host | platform artifact committed without matrix | file identity and target policy | +| Cleanup destroys user work | destructive command in dirty tree | status/dry-run/authorization protocol | +| Markdown diff dwarfs logic change | broad formatter or generator scope | `git diff -w` and task configuration | +| Two "versions" have identical hashes | names mistaken for content evolution | normalized archive comparison | +| Secret deleted but still usable | no rotation/history/publication response | credential and incident state | +| Dead task removal breaks release | connected workflow not searched/tested | CI/prepack/release/editor consumers | + +## Deliberate exclusions + +- Do not delete unknown, untracked, generated, or large files based only on a + pattern match. +- Do not execute unknown binaries to identify them. +- Do not run `git clean`, resets, stashes, broad formatters, package upgrades, + or lockfile regeneration as routine hygiene. +- Do not move large files to LFS and call provenance solved. +- Do not include uploaded evidence/codebases in a public artifact unless the + user explicitly requests redistribution and licensing permits it. +- Do not rewrite history or rotate credentials without the required authority. +- Do not confuse a clean Git status with a reproducible repository. + +## Executable verification + +1. Capture status/revision and a sorted inventory of tracked, untracked, + ignored, executable, binary, and large files. +2. Map generated/vendor/build/cache candidates to producers and consumers. +3. Validate ignore rules plus package/Docker/archive include rules. +4. Run secret and license/provenance review with redacted reporting. +5. Execute setup/check/build/pack/consumer from a clean isolated copy with empty + relevant caches. +6. Verify generated drift and vendor digests without mutation. +7. Create the deliverable from an explicit manifest/tracked revision; list and + integrity-test it; assert forbidden paths and oversized surprises are absent. +8. Inspect all Markdown changes and compare whitespace-insensitive stats; do not + format Markdown. +9. Recheck worktree status and attribute each resulting path. +10. Report preserved unknowns and blocked remediation rather than presenting + partial cleanup as complete. + +## Sources and freshness + +- All retained uploaded archives, observed repository roots, manifests, ignore + files, workflows, fixtures, generated outputs, binaries, and duplicate archive + identity during the July 2026 evidence audit. +- Attached `undent.zip`, observed generator/build/release ownership and ignored + npm output; verified 2026-07-17. +- Attached `wikitext.zip`, observed retained experimental artifacts and explicit + incomplete-matrix reporting; verified 2026-07-17. +- Attached production CLI guidebook v1.1, normative least-privilege tooling, + packed artifact, cache/state path, install/uninstall, and release checks; + verified 2026-07-13. + +Re-evaluate repository-specific ignore, retention, licensing, and incident rules +at the current revision. Generic cache lists are discovery hints, not deletion +authorization. diff --git a/skills/build-devtools/references/mise-aube.md b/skills/build-devtools/references/mise-aube.md new file mode 100644 index 0000000..7864b54 --- /dev/null +++ b/skills/build-devtools/references/mise-aube.md @@ -0,0 +1,422 @@ +# Mise and Aube toolchain architecture + +## Contents + +- Status and ownership +- Mise configuration and trust +- Mise tools, backends, and lockfiles +- Mise tasks and freshness +- Mise environments and secrets +- Aube package-manager model +- Existing lockfiles and migration +- Aube workspaces, catalogs, and deploys +- Dependency builds and the jail +- Runtime and Mise integration +- CI, offline, and proxy operation +- Adoption and rollback +- Failure signatures +- Verification +- Sources and freshness + +## Status and ownership + +Mise and Aube are complementary tools maintained in the jdx ecosystem. They do +not own the same state. + +| Concern | Mise owner | Aube owner | Repository owner that remains authoritative | +|---|---|---|---| +| developer tool versions | `[tools]`, backends, `mise.lock` | may consume a Node runtime policy | runtime/package manifests when another tool already owns them | +| shell environment | `[env]`, environment overlays, activation | npm-compatible environment during scripts | secret manager and deployment platform | +| repository tasks | `[tasks]` or file tasks | `package.json` scripts through `aube run`/`aubr` | one canonical task graph selected by the repository | +| JS dependency graph | no | package manifest, supported lockfile, store, linker | `package.json` and the selected lockfile | +| dependency lifecycle scripts | no | explicit build approval and optional jail | reviewed policy in workspace config | +| workspace selection | Mise task graph and monorepo roots | package graph filters and catalogs | workspace manifests and package declarations | +| publication payload | may invoke/check tasks | pack, publish, deploy | package export/files policy and release workflow | + +Do not create Mise tasks that secretly differ from package scripts or Deno tasks. +A Mise task can be the canonical owner or a thin adapter, but the choice must be +visible in help, CI, and documentation. + +Do not introduce Aube merely because it can read an existing lockfile. First +classify the current package-manager owner, CI/runtime constraints, lifecycle +scripts, native dependencies, linker assumptions, workspace filters, Corepack or +`packageManager` policy, and rollback requirement. + +## Mise configuration and trust + +Mise configuration is hierarchical. A nearer `mise.toml` can override values +from a parent, user, or system scope. Environment-specific configuration and +idiomatic version files add more inputs. Before editing, inspect the resolved +configuration, not one file in isolation: + +```sh +mise config ls +mise config get +mise ls --current +mise doctor +``` + +Record which file owns each tool, environment value, setting, and task. In a +monorepo, state whether subprojects inherit root tools and whether the root is +marked as the monorepo root. + +Set a minimum Mise version when configuration uses newer semantics: + +```toml +min_version = { hard = "2026.7.0", soft = "2026.6.0" } +``` + +Choose the real compatibility boundary rather than copying this version. + +Mise requires trust before executing configuration in an untrusted project. +Treat this as a code-execution boundary: configuration can install tools, render +templates, load environment values, and run tasks. Do not globally trust an +arbitrary checkout merely to make automation green. CI should install from the +reviewed revision and use an explicit trust policy. + +Idiomatic version files such as `.node-version`, `.python-version`, `.nvmrc`, +`rust-toolchain.toml`, and `package.json` can preserve interoperability with +other tools. Current Mise disables general idiomatic-file discovery by default; +enable only the required tool names. Decide whether an idiomatic file or +`mise.toml` owns each version. Two active owners can resolve differently. + +## Mise tools, backends, and lockfiles + +Example: + +```toml +[tools] +deno = "2.9" +node = "24" +"aqua:astral-sh/uv" = "0.8" +aube = "1.27" + +[settings] +lockfile = true +``` + +Mise supports core tools and multiple backends. Current registry policy prefers +direct, security-aware backends such as Aqua or GitHub for many standalone +tools. A registry shorthand can change backend over time; use an explicit +backend when that identity is part of the reproducibility or security contract. +Language-package backends such as npm/pipx/gem can bind the installed tool to +whatever language runtime was active. Make that dependency explicit. + +Custom plugin URLs are executable supply-chain inputs. Pin a Git reference where +supported, verify the plugin type, and review hook behavior. Do not treat an asdf +or vfox plugin as equivalent to a signed standalone release solely because both +install a command with the same name. + +`mise.lock` can pin resolved URLs and checksums. Current Mise can also record +verified provenance for supported backends. Run lock generation on every target +platform needed by CI or release, because cross-platform entries may be derived +from metadata without downloading and verifying that platform artifact. + +```sh +mise lock +mise install --locked +mise ls --current +``` + +A loose declaration such as `node = "24"` is not reproducible by itself. If the +project promises identical versions, commit and enforce the lockfile. If it +intentionally follows latest compatible releases, state that update policy and +test upgrades rather than claiming a pin. + +## Mise tasks and freshness + +Tasks can live in `mise.toml` or executable files under configured task +directories. File tasks retain language-aware editor support and can work for +non-Mise users. Their `#MISE` or `# [MISE]` directives are syntax: a formatter +that changes them may silently remove task metadata. + +```toml +[tasks.check] +description = "Run the repository quality gate" +depends = ["generate:check"] +run = [ + { task = "lint" }, + { tasks = ["typecheck", "test"] }, +] + +[tasks."generate:check"] +run = "deno task generate --check" +sources = ["schemas/**/*.ts", "scripts/generate.ts"] +outputs = ["generated/manifest.json"] +tools.deno = "2.9" +``` + +Important task properties include: + +- `run`, `run_windows`, `file`, and `shell` for execution; +- `depends`, `depends_post`, and `wait_for` for graph order; +- structured task references with arguments and environment overrides; +- `tools`, `env`, `dir`, and `usage` for an execution boundary; +- `sources` and `outputs` for freshness and watch behavior; +- `timeout`, `confirm`, `hide`, `quiet`, `silent`, `raw`, and `interactive` for + control and output behavior. + +`depends` can run concurrently. Do not put order-dependent mutations in sibling +dependencies. `confirm` protects the task's own run body, not dependencies that +execute first. Put authorization before side effects, or represent the guarded +action as an explicit task call after confirmation. + +`sources`/`outputs` use freshness rules, not semantic content equality. Generated +artifact checks still need deterministic check mode and a content comparison. +Glob breadth affects scan time; exclusions and downstream invalidation must be +tested. An automatically tracked output stamp proves the task ran, not that a +declared artifact exists or is valid. + +`raw` and `interactive` alter I/O scheduling. Prefer `interactive` for exclusive +terminal ownership. Raw output can break parallel presentation and bypass +redaction. Stable machine-readable output should normally come from the called +tool, not Mise's progress UI. + +## Mise environments and secrets + +Mise can construct environment variables from config, dotenv files, templates, +paths, and environment-specific overlays. Classify values as public defaults, +developer-local values, or secrets. Do not commit credentials because Mise can +load them. + +Use redaction declarations for task output, then verify the CI system also masks +the values. Mise redaction is line-oriented and does not apply to raw task I/O. +Secrets can still leak through child tools, command arguments, debug logs, +artifacts, or structured output. + +Activation mutates the interactive shell. `mise exec -- command` or task-local +tools often produce a clearer CI boundary than relying on shell startup files. +Shims do not provide every activation feature. Test the selected mode in clean +shells on supported platforms. + +## Aube package-manager model + +Aube is a versioned Node.js package manager, not a generic task helper. As of the +2026-07-17 source review, the public npm package is `@endevco/aube` and the +current release is 1.27.0. Commands include `aube`, `aubr` (`aube run`), and +`aubx` (`aube dlx`). Verify the current release before pinning. + +Aube uses an isolated `node_modules` layout by default, a content-addressed +global store, and `node_modules/.aube/` for its virtual store. Only declared +dependencies should be relied on; a hoisted linker can hide undeclared phantom +dependencies. Test packages under the selected linker. + +Routine commands check install freshness before running: + +```sh +aubr build +aube test +aube exec vitest +aubx cowsay hello +``` + +This auto-install behavior is a contract, not always an advantage. In hermetic +CI, long-lived development processes, offline operation, or commands that must +never mutate the dependency tree, choose the documented frozen/no-install or +verification policy deliberately. `aubeNoAutoInstall` and +`verifyDepsBeforeRun` change behavior; inspect the current settings reference +before using them. + +## Existing lockfiles and migration + +Aube can read and update supported pnpm, npm, Yarn, and Bun lockfiles in place. +That enables a reversible trial without a lockfile conversion. It does not prove +that resolution, peer behavior, patches, lifecycle scripts, linker layout, and +platform filtering match the current package manager. + +Safe adoption order: + +1. pin and install Aube without deleting the existing package manager; +2. run an install using the existing committed lockfile; +3. compare lockfile diff and resolved graph; +4. test scripts, binaries, native builds, patches, peers, workspace filters, + package packing, deployment, and clean consumers; +5. keep the old command working during the trial; +6. convert to `aube-lock.yaml` only as a separate explicit migration; +7. remove the old lockfile only after the new owner passes CI and rollback is + documented. + +Never keep two writable lockfiles for one dependency graph. During coexistence, +one committed lockfile remains authoritative and both tools must operate on it. + +CI should use frozen behavior and fail on manifest/lock drift. Docker builds +should separate the dependency layer without assuming that a later `aubr` +command may rewrite the lockfile or contact the registry. + +## Aube workspaces, catalogs, and deploys + +Aube discovers workspaces from `aube-workspace.yaml` and can consume an existing +`pnpm-workspace.yaml`. The file is an ownership boundary for package globs, +catalogs, lifecycle-build policy, and several settings. + +```yaml +packages: + - "packages/*" + - "apps/*" + +catalog: + zod: ^4.0.0 + +catalogs: + test: + vitest: ^3.0.0 +``` + +Workspace protocol declarations remain in each consumer manifest. Publishing +or deploy flows rewrite them to concrete versions. Verify the packed manifest, +not only the source declaration. + +Filters can address exact names, globs, paths, dependency/dependent graphs, Git +changes, and exclusions. Validate a filter with a dry list before running a +mutation or publication. Recursive execution must respect dependency order, +failure propagation, concurrency, and output clarity. + +`aube deploy` creates a deployable package tree and installs its dependencies. +Its file selection is tied to pack semantics unless `deployAllFiles` is chosen. +Do not use `deployAllFiles` to compensate silently for an incorrect package +payload; determine whether the missing file belongs in publication, deployment +only, runtime generation, or an external artifact. + +Catalog modes change how `aube add` writes manifest ranges. A strict catalog can +prevent drift, but named catalogs are not automatically selected. Verify every +workspace consumer and the resulting lockfile. + +## Dependency builds and the jail + +Aube does not run dependency lifecycle scripts unless they are approved. Root +project scripts are a separate policy. Current approval data uses `allowBuilds`; +legacy pnpm/Bun allow and deny inputs can also affect the result. Review the +resolved policy rather than assuming one key is the only owner. + +```yaml +allowBuilds: + esbuild: true + sharp: true + unreviewed-native-package: false + +jailBuilds: true +jailBuildPermissions: + sharp: + env: [SHARP_DIST_BASE_URL] + write: ["~/.cache/sharp"] +``` + +Build approval answers whether a script may run. The jail limits what an +approved script can access. As of this review, jail behavior is platform +dependent, `jailBuilds` is not yet the default, and Windows isolation differs +from Linux/macOS. Grant the narrowest package-specific permissions and test the +actual native build. Do not use a broad exclusion merely to make install pass. + +Security checks, minimum release age, trust policy, registry authentication, +proxies, certificates, offline caches, and advisory behavior are connected. +Treat registry tokens and custom CAs as secrets; do not embed them in shared +workspace files or diagnostic output. + +## Runtime and Mise integration + +Aube can use project runtime declarations and can delegate runtime installation +to Mise. Mise can also install Aube itself. Avoid an infinite or ambiguous owner +loop: + +```text +Mise installs the pinned Aube executable + -> Aube validates project packageManager/devEngines policy + -> one selected owner installs Node + -> Aube resolves and links JS dependencies + -> Mise or package scripts invoke canonical tasks +``` + +Choose whether Node is installed by Mise or Aube and set fallback behavior. An +air-gapped CI environment should error rather than download an unexpected +runtime. Aube's package-manager version switching and Mise's tool pin must agree; +otherwise the first Aube process can re-exec another version. + +## CI, offline, and proxy operation + +Verify at least: + +- clean install with empty project store and cache; +- warm install with the expected store reuse; +- frozen install with no manifest or lockfile mutation; +- offline install from a deliberately prepared cache/store; +- authenticated registry and scoped registry behavior; +- HTTP(S) proxy and custom CA behavior if supported; +- all target OS/architecture/libc optional dependencies; +- approved native builds under the actual jail policy; +- workspace filters and recursive task failure; +- packed and deployed clean consumers; +- uninstall/rollback to the prior package manager. + +Do not quote benchmark numbers from another machine as a project result. Protect +correctness, install security, disk use, cold and warm behavior, and routine +script latency as separate metrics. + +## Adoption and rollback + +Use a staged decision: + +1. inventory versions, config scopes, manifests, lockfiles, scripts, workspaces, + registries, native builds, CI, editors, and deployment; +2. assign one owner per concern; +3. pin Mise/Aube and record supported platforms; +4. add a read-only or existing-lockfile trial; +5. compare graphs, artifacts, and real workflows; +6. enable stricter security/frozen policies explicitly; +7. migrate task or lock ownership separately; +8. retain a rollback command until a clean clone and release pass; +9. remove redundant wrappers and stale configuration only after reachability + searches and CI verification. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| local task passes but CI fails | different config scope, shell activation, tool version, working directory, or auto-install policy | `mise config ls`, task trace, clean CI shell, Aube frozen logs | +| lockfile changes on routine test | auto-install found manifest drift or two writers own the graph | Aube state, package manifest diff, lock owner, `verifyDepsBeforeRun` policy | +| package works only with hoisting | undeclared dependency or peer/linker assumption | isolated linker, `aube why`, manifest and peer declarations | +| native package is present but unusable | build script was unapproved, jailed without required permission, or incompatible platform artifact selected | ignored/approved builds, jail diagnostics, package build output | +| secrets appear in task output | raw I/O or child tool bypassed Mise redaction | task output mode, CI masking, process arguments and diagnostic sink | +| task skips despite invalid output | timestamp freshness or auto stamp was treated as semantic validation | declared outputs, generator check mode, artifact validator | +| subproject resolves wrong runtime | hierarchical config or idiomatic file introduced a second owner | resolved Mise config and version-source inventory | +| Aube re-execs unexpectedly | `packageManager`/`devEngines` pin disagrees with Mise-installed Aube | package manager version policy and runtime installer setting | +| deploy lacks runtime file | pack selection excludes it or file belongs to another artifact owner | `aube pack`/deploy contents and package `files`/ignore rules | +| offline claim fails | cache/store was not fully prepared or tool/runtime installer still needs network | cold trace, registry requests, Mise/Aube download policy | + +## Verification + +Run commands appropriate to the selected ownership; do not copy all commands +blindly: + +```sh +mise doctor +mise config ls +mise ls --current +mise tasks +mise run check +mise lock +mise install --locked + +aube install +aube list --depth 0 +aube why +aube ignored-builds +aube pack +``` + +Then perform clean-clone, frozen, offline, native-build, workspace-filter, +package-consumer, deploy, and rollback scenarios. Record exact versions, +platform, lockfile before/after digest, store/cache state, commands, exit codes, +and artifact contents. + +## Sources and freshness + +- Mise configuration, tools/backends, lockfile/provenance, tasks, environment, + and security documentation at https://mise.jdx.dev/, verified 2026-07-17. +- Aube getting-started, configuration, workspaces, lifecycle scripts, security, + and settings documentation at https://aube.jdx.dev/, verified 2026-07-17. +- `@endevco/aube` npm metadata, version 1.27.0 observed 2026-07-17. + +Both tools evolve quickly. Recheck exact option names, defaults, backend policy, +lockfile support, jail behavior, and installed versions before applying this +manual. Examples illustrate current ownership and failure contracts; the +repository's installed version and generated help remain authoritative. diff --git a/skills/build-devtools/references/packaging.md b/skills/build-devtools/references/packaging.md index 3b5bdc3..3872166 100644 --- a/skills/build-devtools/references/packaging.md +++ b/skills/build-devtools/references/packaging.md @@ -1,41 +1,364 @@ # Cross-runtime packaging -## One source, explicit outputs +## Contents -Keep one authoritative source tree where possible. Generate runtime-specific -packages deterministically rather than hand-maintaining divergent dependency -graphs. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Start from consumers](#start-from-consumers) +- [Authority and artifact graph](#authority-and-artifact-graph) +- [Exports and entrypoints](#exports-and-entrypoints) +- [Build strategies](#build-strategies) +- [Types, source maps, and runtime semantics](#types-source-maps-and-runtime-semantics) +- [Package contents and assets](#package-contents-and-assets) +- [Executables and platform artifacts](#executables-and-platform-artifacts) +- [Clean-consumer matrix](#clean-consumer-matrix) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Executable verification](#executable-verification) +- [Sources and freshness](#sources-and-freshness) -For each target prove: +## When to load this reference -- exact root and subpath exports; -- type declarations and source maps; -- runtime files and assets; -- platform/engine requirements; -- side-effect declaration; -- license, README, and changelog; -- package allowlist/contents; -- clean consumer import and representative behavior. +Load this reference for npm/JSR publication, Deno-to-Node builds, library export +changes, workspace publication, binaries, templates, generated declaration +files, or bugs that appear only after packing/installing. It is also required +when source tests pass but consumers report missing modules, assets, types, +executables, or incompatible runtime behavior. -## Deno and Node +## Outcome -A Deno-first source can use a build system such as dnt for Node output. Keep -authoritative Deno tests and add generated Node type/runtime tests. Excluding a -Deno-only test from the Node build is acceptable when the corresponding public -contract is verified elsewhere and the exclusion is explicit. +Create an artifact whose public contract is explicit and proven outside the +source workspace. A complete package decision covers: -Version must come from one release source such as a validated release environment -value or tag, then propagate into every manifest and artifact. +- the authoritative source and version; +- every public root/subpath/condition and its runtime/type target; +- which files are bundled, transpiled file-to-file, copied, or generated; +- dependency externalization and runtime requirements; +- assets, licenses, notices, source maps, and executable permissions; +- supported runtimes, module systems, platforms, and package managers; +- the actual archive contents and unpacked size; +- installation, import, behavior, upgrade, and removal in clean consumers. -## Clean generation +"The build passed" is not a packaging result. The package archive, registry +metadata, loader, and consumer form a connected contract. -Remove or recreate the output directory before building. Never package stale -files left by a previous build. Compare package contents to an allowlist and run -consumer tests outside the repository workspace so local resolution cannot hide -missing files or dependencies. +## Start from consumers -## Lifecycle +Inventory intended consumers before selecting a builder. -Verify install, upgrade, public imports, executable behavior where present, -uninstall, and supported runtimes. Keep source-runtime checks distinct from -generated-package checks in reports. +| Consumer | Questions that change the artifact | +|---|---| +| Deno/JSR | Are source TypeScript and `jsr:` dependencies valid? Are exports declared in `deno.json`? | +| Node ESM | Do export conditions reach ESM files and declarations? What Node version is supported? | +| Node CommonJS | Is CJS truly required? Are dual-package state and default/named interop tested? | +| Bundler/browser | Are Node built-ins absent/conditional? Are side effects and browser assets correct? | +| Worker/edge | Are dynamic code loading, filesystem, sockets, and Node compatibility excluded or adapted? | +| CLI user | Does `bin` point to a published executable with a shebang and correct mode? | +| Framework adapter | Are peer dependencies and renderer/runtime versions compatible? | +| Type-only consumer | Do resolution modes find the same public graph as runtime resolution? | + +Do not promise a runtime because the language transpiles. Search public source +for `Deno.*`, `node:` imports, native dependencies, dynamic `require`, filesystem +layout assumptions, environment reads, subprocesses, and bundler transforms. +Separate portable core from host adapters when the capability boundary is real. + +## Authority and artifact graph + +Prefer one authoritative implementation and generate only the distributions +that differ mechanically. Record the graph: + +```text +source modules + public export inventory + version + -> source-runtime package (for example JSR) + -> Node package build + -> ESM/CJS runtime files + -> declarations and maps + -> copied license/readme/changelog/assets + -> package.json generated from explicit policy + -> packed archives + -> registry publications +``` + +One source does not mean one artifact can serve every host. It means behavioral +changes are authored once and transformation boundaries are deterministic. Each +artifact still needs independent verification. + +Define one version source for a release: exact tag, validated release input, or +manifest. Propagate it into all generated manifests and `--version` output. Do +not read `latest` from a registry while building a release. Reject a mismatch +between tag, source manifest, generated package, and release notes before +publication. + +Workspace ownership matters. Identify the nearest package manifest, workspace +root, catalog/override policy, internal dependency protocol, and publish order. +Internal packages may need concrete published versions in their packed manifests; +verify the archive rather than assuming the package manager rewrites them. + +## Exports and entrypoints + +Build the public export inventory before writing a build config. + +```text +. +./unicode +./browser +./package.json # only if intentionally public +./styles.css # asset contract, if applicable +bin: tool +``` + +For every entry state: + +- supported import spelling; +- runtime conditions (`import`, `require`, `browser`, `node`, `default`, or a + project-specific condition); +- declaration target and resolution modes tested; +- source and output file; +- side effects and initialization behavior; +- required peers, assets, permissions, and platform restrictions; +- whether deep imports outside the map are deliberately blocked. + +Example ESM-only package surface: + +```json +{ + "type": "module", + "exports": { + ".": { + "types": "./dist/index.d.mts", + "import": "./dist/index.mjs" + }, + "./unicode": { + "types": "./dist/unicode.d.mts", + "import": "./dist/unicode.mjs" + } + }, + "files": ["dist", "license", "readme.md"], + "sideEffects": false, + "engines": { "node": ">=20" } +} +``` + +This is a pattern, not a manifest to copy blindly. Export condition ordering, +extension, and declaration choices must match actual emitted files. Keep legacy +`main`, `module`, and `types` only when supported consumers need them and ensure +they agree with `exports`. `sideEffects: false` is a correctness assertion: do +not use it if importing a module registers behavior, CSS, polyfills, or globals. + +Do not export an undocumented implementation file accidentally. Conversely, +documentation is not proof of an export. The attached Wikitext archive is a +counterexample: a README may mention a public capability that the root export +and implementation do not supply. Compare docs, export map, source entrypoint, +generated output, and packed archive. + +## Build strategies + +Select a builder by output semantics. + +### Direct/source publication + +Use source publication when the registry/runtime accepts the source graph and +the public code is already portable. Verify versioned dependencies, export maps, +publish includes/excludes, documentation examples, and registry dry-run. Source +publication does not remove the need for consumer tests. + +### Deno-to-Node with dnt + +`@deno/dnt` can transform a Deno source graph into a Node npm package, generate +entrypoint exports, rewrite dependencies, create declarations, add shims, and +run Node-oriented checks. The attached Undent `scripts/build_npm.ts` shows: + +- explicit root and `./unicode` export entrypoints; +- clearing ignored output before building; +- no Deno shim because public source uses no Deno globals; +- both source and output typechecking; +- exclusion of Deno-native test dependencies from the Node graph, with Deno + tests remaining authoritative and Node package behavior tested separately; +- generated package metadata, `sideEffects: false`, Node engine policy, and + post-build copying of license/readme/changelog; +- release-version validation with semantic-version parsing. + +An excluded test is a documented coverage transfer, not a free omission. State +which equivalent consumer/behavior check covers the generated artifact. + +### unbuild and mkdist + +Unbuild 3.6.1 supports inferred or explicit entries, Rollup-based bundles, +TypeScript declarations, multiple configs, sourcemaps, dependency checks, +development stubs, and `mkdist` file-to-file output. Choose intentionally: + +- bundle when a compact runtime unit is desired and dependency boundaries are + understood; +- `mkdist` when preserving module/subpath structure and per-file tree shaking + matters; +- multiple configs only when they produce separately named/tested artifacts; +- `--stub` for local development, never as a release artifact. + +```ts +import { defineBuildConfig } from "unbuild"; + +export default defineBuildConfig({ + entries: [ + "./src/index", + { builder: "mkdist", input: "./src/runtime", outDir: "./dist/runtime" }, + ], + outDir: "dist", + declaration: "compatible", + sourcemap: true, +}); +``` + +At 3.6.1, unbuild's Rollup path, mkdist path, declaration modes, externals, and +hooks have different semantics. Inspect the pinned type declarations/config +resolution before using an option. Its README also marks `obuild` as an +experimental successor; do not migrate release infrastructure for speed alone. + +## Types, source maps, and runtime semantics + +Test declarations as a consumer under every supported resolver, not only by +typechecking source. Include NodeNext/Node16/bundler modes where promised. +Confirm: + +- declarations reference published paths only; +- `.d.mts`, `.d.cts`, and `.d.ts` agree with runtime conditions; +- type-only exports are reachable and runtime exports are not phantom types; +- declaration maps/source maps point to included sources or intentionally + omit sources; +- CJS default/named interop matches runtime behavior; +- generated code preserves error causes, URL/path behavior, Unicode, async + cancellation, and other public semantics. + +Bundling can duplicate singleton state or hide peer dependencies. Externalizing +everything can leave a consumer without a required runtime dependency. Classify +each dependency as bundled, runtime dependency, optional dependency, peer, or +development-only and test absence/presence behavior. Native packages require +platform/architecture/libc/ABI coverage and lifecycle-script policy. + +## Package contents and assets + +Use an allowlist (`files` or registry equivalent) and inspect the pack result. +Expected content often includes runtime output, declarations, maps if promised, +license/notices, README, changelog, templates, schemas, WASM/native assets, CSS, +and executable files. Reject: + +- source secrets, `.env`, tokens, caches, databases, benchmark corpora, coverage, + editor binaries, internal fixtures, release credentials, and unrelated docs; +- missing runtime-loaded templates, migrations, workers, WASM, native binaries, + fonts, CSS, or schema files; +- absolute build paths and nondeterministic timestamps where reproducibility is + claimed; +- output retained from a previous build. + +Generate in a fresh directory. Record a sorted package-content manifest with +path, size, mode where relevant, and digest for release artifacts. Inspect both +compressed archive and unpacked installed form. Licenses of bundled/vendored +dependencies may require notices even when those packages are not visible as +runtime dependencies. + +## Executables and platform artifacts + +For npm `bin`, verify the target is included, starts with a portable shebang, +has executable permission in the tarball, resolves runtime dependencies, and +does not import a development stub. Run `--help`, `--version`, an error path, +stable stdout, cancellation, and install/uninstall through the packed artifact. + +For `deno compile`, Node SEA, or native launchers, record runtime/tool version, +compile flags, embedded assets, environment assumptions, target triples, signing, +and checksums. Cross-compilation success does not prove target execution. Run on +each supported OS/architecture or narrow the support claim. A standalone binary +changes update, vulnerability, license, certificate, and removal responsibilities. + +## Clean-consumer matrix + +Create consumers outside the workspace with no link protocol, source alias, +root `node_modules`, or repository TypeScript configuration. Install the exact +tarball or registry version. + +| Axis | Minimum proof | +|---|---| +| Runtime | each promised Deno/Node/browser/worker version boundary | +| Module | ESM and CJS only if each is promised | +| Resolver | relevant TypeScript and runtime resolution modes | +| Entry | every root/subpath/bin/asset import | +| Package manager | supported managers or one explicit authority | +| Lifecycle | fresh install, upgrade from previous supported version, uninstall | +| Behavior | representative public call and representative error | +| Host | framework/bundler adapter build when exported | + +Example packed npm check: + +```sh +npm pack --json +mkdir -p "$TMPDIR/package-consumer" +cd "$TMPDIR/package-consumer" +npm init -y +npm install /absolute/path/to/package-1.2.3.tgz +node --input-type=module -e 'import("package").then(m => console.log(Object.keys(m)))' +npm uninstall package +``` + +Use the repository's chosen package-manager adapter when testing manager-neutral +tooling, but preserve the consumer lockfile and exact commands as evidence. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Source tests pass, import fails after install | export target or file omitted | packed manifest and clean consumer | +| Types resolve but runtime fails | declaration/runtime graph mismatch | export conditions and emitted extensions | +| Runtime works, types fail under NodeNext | wrong declaration extension/path | resolver trace in clean TS consumer | +| One subpath contains stale code | output directory not cleared or unowned entry | clean build and output manifest | +| Package works only in monorepo | workspace alias/hoist/undeclared dependency | isolated install with empty cache | +| Browser bundle imports `node:` | host boundary leaked into public core | public graph and conditional export | +| CJS and ESM have different singleton state | dual package instantiated twice | conditional export design and tests | +| CLI installs but cannot execute | missing bin, shebang, mode, or runtime dependency | tar metadata and installed bin link | +| Registry packages have different behavior | independent generation/version drift | source/version authority and archive diff | +| Native install passes one runner only | missing platform/ABI coverage | artifact matrix and lifecycle logs | + +## Deliberate exclusions + +- Do not generate CommonJS merely because a builder supports it. +- Do not include source/tests/config by default to mask a missing runtime file. +- Do not bundle peers or native dependencies without an explicit ownership and + license decision. +- Do not claim browser/edge support based only on types. +- Do not test through workspace symlinks as package verification. +- Do not publish first and inspect the package later; dry-run/pack locally and + preserve the exact inspected artifact. +- Do not retain obsolete legacy fields or deep imports unless a measured + compatibility commitment requires them. + +## Executable verification + +1. Build twice from separate clean directories and compare normalized manifests + and artifact hashes. +2. Run registry dry-run/package pack and compare contents to an allowlist. +3. Assert version agreement across tag/input, source manifest, output manifests, + executable `--version`, changelog, and archive name. +4. Import every public entry at runtime and typecheck it in clean consumers. +5. Execute representative behavior and error paths, not only `Object.keys`. +6. Test declared runtime/module/platform matrix and deliberate unsupported cases. +7. Scan archive for secrets, absolute paths, caches, internal fixtures, and + unexpected executable/binary files. +8. Test install, upgrade, and uninstall without workspace links. +9. For dual registries, install each artifact and compare public behavior while + respecting intended runtime differences. +10. Re-run from the exact release commit/tag with frozen dependencies. + +## Sources and freshness + +- Attached `undent.zip`, observed `scripts/build_npm.ts`, `deno.json`, CI and + publish workflows; verified 2026-07-17. +- Attached `wikitext.zip`, observed export/documentation discrepancy used as a + counterexample; verified 2026-07-17. +- Unbuild 3.6.1 published README, declarations, and package manifest; source + record `unbuild-3-6-1`, verified 2026-07-17. +- Pkg-types 2.3.1 published declarations/implementation for package discovery, + normalization, exports, workspaces, and cache behavior; source record + `pkg-types-2-3-1`, verified 2026-07-17. +- Attached production CLI guidebook v1.1, normative package layout, Deno/Node, + executable, and packed-consumer requirements; verified 2026-07-13. + +Inspect the installed builder and package-manager versions before copying config. +The examples above are decision patterns, not evidence that a future version +emits the same filenames or supports the same options. diff --git a/skills/build-devtools/references/performance.md b/skills/build-devtools/references/performance.md index 9500b85..e817c98 100644 --- a/skills/build-devtools/references/performance.md +++ b/skills/build-devtools/references/performance.md @@ -1,31 +1,330 @@ # Performance experiments -## Protocol before result +## Contents -Write the hypothesis, target workflow, protected workflows, metrics, dataset, -runtime/tool versions, warmup, process isolation, ordering, sample count, -statistics, acceptance threshold, and rollback before measuring. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Question, hypothesis, and acceptance rule](#question-hypothesis-and-acceptance-rule) +- [Workload and protected-workflow design](#workload-and-protected-workflow-design) +- [Candidate isolation and provenance](#candidate-isolation-and-provenance) +- [Timing protocol](#timing-protocol) +- [Memory and resource protocol](#memory-and-resource-protocol) +- [Statistics and decisions](#statistics-and-decisions) +- [Stress and scale lanes](#stress-and-scale-lanes) +- [Correctness and operational gates](#correctness-and-operational-gates) +- [Artifacts and reporting](#artifacts-and-reporting) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Executable verification](#executable-verification) +- [Sources and freshness](#sources-and-freshness) -Baseline and candidate should carry their own source and harness snapshot. Run -timing and retained-memory measurements in fresh processes where cross-run state -would bias results. Preserve raw samples and use deterministic round-robin order. +## When to load this reference -## Decision gates +Load this reference for optimization proposals, benchmark additions, throughput, +latency, memory, allocation, startup, binary size, build speed, bundle size, or +claims that one implementation is faster. Also load it when a microbenchmark +improves while users report regressions, or when benchmark results are being +used to accept/reject architecture. -Use effect size and uncertainty, not only the fastest sample. The attached -wikitext experiment protocol uses a useful example policy: require a meaningful -median target improvement, reject statistically supported critical regressions, -and adjust multiple comparisons. Adopt thresholds appropriate to the project and -write them before seeing results. +## Outcome -## Protected workflows +Produce an experiment that another maintainer can rerun, audit, and reject. It +must preserve raw samples and candidate source, precommit its decision rule, +measure representative target and protected workflows, run correctness gates, +model uncertainty, and distinguish observed results from extrapolation. -A tokenizer microbenchmark win does not justify slower parse, session, memory, -or real consumer behavior. Test representative end-to-end workloads, correctness, -allocation/retention, and cold/warm behavior. +Performance is multi-dimensional. A candidate is not "faster" without naming: -## Reporting +- operation and input distribution; +- cold/warm/cache/process state; +- latency statistic or throughput unit; +- memory definition (peak, retained, allocation, RSS, heap, external); +- runtime/tool/hardware/OS/power conditions; +- error/cancellation/backpressure behavior; +- protected workflows and permitted regressions; +- scope: this machine/run/version versus broader claim. -Separate hypothesis, methodology, observations, limitations, and conclusion. -Report environment and raw result location. Do not claim a universal speedup from -one fixture or an unexecuted benchmark. +## Question, hypothesis, and acceptance rule + +Write these before collecting candidate data: + +```text +Question: Does flattening parser event fields improve parse workloads? +Mechanism: fewer nested allocations and property reads. +Target workflows: tokenizer events(), parse(), parseWithDiagnostics(), session. +Protected workflows: malformed recovery, Unicode offsets, streaming, memory. +Primary metric: median relative change across named timing cases. +Gate: >=5% target median improvement; no critical timing regression >3% + with adjusted significance; no overall retained-memory regression. +Rollback: retain baseline implementation; reject candidate snapshot. +``` + +Choose thresholds from product value, measurement resolution, and risk—not +after seeing results. Define a smallest effect worth acting on. A statistically +detectable 0.5% change may be operationally irrelevant; a 2% tail-latency +regression may be critical in a service even when average throughput improves. + +Specify primary, secondary, diagnostic, and guardrail metrics. Multiple primary +metrics without a rule invite cherry-picking. State how missing, failed, timed +out, or out-of-memory samples affect the decision; never drop them silently. + +## Workload and protected-workflow design + +Build a workload ledger that represents real use and pathological boundaries. + +| Lane | Examples | Purpose | +|---|---|---| +| Micro | one tokenizer primitive, one serialization step | explain mechanism | +| Component | full tokenizer/parser/filter | measure subsystem | +| End-to-end | public parse/session/CLI/API workflow | decide user impact | +| Error | malformed input, retry, cancellation | protect failure behavior | +| Scale | large input/cardinality/concurrency | expose asymptotics and limits | +| Cold | startup/import/first request/empty cache | deployment and CLI behavior | +| Warm | steady-state repeated workload | long-running behavior | + +Use production distributions or documented fixtures where possible. Include +small, typical, large, Unicode, adversarial, highly compressible/incompressible, +sparse/dense, and hit/miss cases as relevant. Record fixture generation seeds +and digests. Do not let candidate code select easier fixtures. + +Protected workflows prevent local optimization from moving cost elsewhere. A +tokenizer win must not slow complete parse, tree building, diagnostics, session, +or real consumer behavior. A database query win must not increase write cost, +merge backlog, recovery time, or result errors. A build-speed win must not lose +types, source maps, or package checks. + +Verify the benchmark reaches the intended path. Count operations/output, assert +result digests, instrument branch/hit rates if necessary, and detect dead-code +elimination or cached precomputation. A benchmark that returns the wrong result +quickly is a correctness failure. + +## Candidate isolation and provenance + +Keep baseline and candidate source/harness snapshots together. The attached +Wikitext experiment stores approach-local `code/`, artifacts, and README notes; +this prevents later source edits from silently changing what an old report +means. + +Record: + +- source revision and candidate patch/digest; +- benchmark and fixture revision/digests; +- exact command, options, seed, rounds, and ordering; +- runtime/compiler/GC/tool versions and flags; +- OS/kernel, CPU model/count/governor, memory, architecture; +- container/VM, power/thermal, background load, affinity, and CI runner class; +- environment variables, locale/timezone, caches, and dependency lock digest; +- raw stdout/stderr and exit status for failed samples. + +Baseline and candidate must use the same harness and meaningful environment. +If a candidate requires a harness change, first run old code under old and new +harnesses to measure the harness effect. Freeze or version the harness; do not +copy current source into an old approach directory after results exist. + +## Timing protocol + +1. Establish environment preflight and correctness. +2. Choose process isolation. Use fresh processes when JIT/GC/global caches or + module state can leak between variants. +3. Define warmup separately from measured samples; record it. +4. Interleave baseline and candidate in a deterministic randomized or balanced + round-robin order to reduce thermal/time drift. +5. Use enough independent process-level samples for the chosen uncertainty + method; many inner-loop iterations are not independent samples. +6. Preserve raw per-case samples and failures. +7. Monitor clock resolution, CPU throttling, outliers, and environment drift. +8. Re-run a surprising result in a fresh session/machine where material. + +Example schedule artifact: + +```json +{ + "seed": 4217, + "rounds": [ + ["baseline", "candidate"], + ["candidate", "baseline"], + ["baseline", "candidate"] + ], + "runsPerVariant": 10, + "warmup": 3 +} +``` + +Use monotonic high-resolution timing. Avoid timing setup unrelated to the +question unless startup is the target. Conversely, do not exclude parsing, +allocation, I/O, or cleanup that real users pay for. State boundaries exactly. + +For asynchronous/concurrent work, control input arrival, concurrency, queue +depth, backpressure, and completion. Throughput at unbounded queue growth is not +sustainable performance. Report latency distributions and errors under a fixed +load or throughput under a latency/error service objective. + +## Memory and resource protocol + +Name the memory measure. Common measures answer different questions: + +- allocated bytes: churn/GC pressure; +- live heap after controlled GC: retained object graph; +- peak RSS: process/container capacity including non-heap/native/code; +- external/array-buffer memory: data not captured by heap alone; +- per-operation retained state: session/cache/leak risk. + +Run memory harnesses in fresh processes and hold the intended result alive. If +forcing GC, record runtime flags and collect after a consistent sequence; forced +GC changes execution and does not represent peak memory. Use both peak and +retained measures when resource limits matter. + +Detect leaks with repeated lifecycle cycles and plateau expectations, not one +before/after subtraction. Exercise cancellation, errors, subscription cleanup, +worker termination, cache eviction, and disposal. Also track CPU utilization, +I/O bytes/operations, network, file descriptors, event-loop delay, binary size, +and build artifact size when a candidate can shift cost. + +## Statistics and decisions + +Prefer relative paired comparisons from interleaved runs. Report median and a +tail statistic for latency, plus uncertainty. Preserve the full distribution; +do not report only the fastest sample or operations/second summary. + +Bootstrap confidence intervals are useful when distributions are non-normal. +When testing many cases, control family-wise error or false discovery according +to the predeclared plan. The Wikitext study uses bootstrap confidence intervals, +bootstrap p-values, and Holm adjustment at alpha 0.05, with a target-median and +critical-case gate. This is a project-specific policy, not a universal formula. + +Decision table: + +| Observation | Decision | +|---|---| +| Effect clears practical threshold and uncertainty/guardrails | eligible to keep | +| Direction positive but below meaningful threshold | reject or defer; do not call a win | +| One microcase wins, protected E2E regresses | reject or redesign | +| Confidence inconclusive | collect preplanned additional samples or report inconclusive | +| Memory improves but timing gate fails | reject under timing-primary policy; retain result | +| Candidate fails correctness or crashes | reject regardless of speed | + +Do not use "not statistically significant" to prove equality. If equivalence +matters, specify equivalence margins and use an appropriate test/design. Do not +average ratios across incomparable workloads without a declared weighting. + +## Stress and scale lanes + +Stress tests answer survivability/asymptotic questions and should not distort +the standard comparison set. The Wikitext archive separates standard reports +from 16 MiB and 1 GiB stress artifacts, defines ten scenario families, and uses +a streaming-only default at 1 GiB so that lane measures event-path survivability. +Full materialization gets a separately named artifact. + +For stress lanes define: + +- exact size/cardinality/concurrency and generation algorithm; +- whether data is streamed or fully materialized; +- time/memory/resource limit and failure recording; +- repetition policy (often fewer expensive runs, with weaker claims); +- scenario-specific output/correctness checks; +- artifact naming that cannot overwrite another profile; +- preflight inventory proving the lane actually exists. + +A successful 1 GiB stream does not prove a full-tree parser handles 1 GiB. A +failed case must be recorded in the JSON artifact rather than causing the entire +report to disappear. Extrapolation beyond executed scenarios must be labelled. + +## Correctness and operational gates + +Run before and after benchmarks: + +- unit/property/fuzz/fixture tests for unchanged semantics; +- stable output/order/offset/precision contracts; +- errors, retries, cancellation, timeout, and cleanup; +- supported runtime/platform/compiler modes; +- end-to-end consumer workflows; +- security/permission boundaries; +- package/build checks if optimization affects output. + +Optimization may intentionally change output (compression, approximate query, +cache policy). Define permitted error and validate it on an independent data set. +Benchmark fixtures used to tune the candidate cannot be the only accuracy set. + +## Artifacts and reporting + +Retain machine-readable artifacts: + +```text +experiment/ + protocol.md # frozen question and decision rule + schedule.json # exact order/seeds + baseline/code + metadata + candidate/code + metadata + reports/*.json # raw samples and environment + comparisons/*.json # derived statistics + commands.txt + results.md # human interpretation and limits +``` + +Human notes must separate hypothesis, methodology, observations, limitations, +and conclusion. Link every conclusion to artifacts. Record rejected candidates; +negative results stop repeated bad ideas. The attached Wikitext results retain +several rejected event-shape approaches, including a small memory improvement +that missed the timing gate and lazy-property designs with severe timing loss. + +Do not claim a matrix is complete because tooling exists. The Wikitext ledger +explicitly distinguishes implemented runners from collected artifacts and notes +that its broader stress matrix, session stress, and multi-machine evidence are +incomplete. Use the same honesty for blocked hardware, missing credentials, or +unavailable runtimes. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Huge win disappears in end-to-end test | microbenchmark excludes shifted cost | protected workflow ledger | +| Candidate always runs second and loses | thermal/time/order bias | interleaved schedule | +| Many inner iterations, tiny uncertainty | pseudoreplication | independent process/sample unit | +| Memory result changes after timing run | shared process/cache/GC state | fresh-process harness | +| Fastest sample reported | cherry-picked statistic | raw samples and predeclared metric | +| Significant 0.4% win accepted | practical threshold absent | smallest useful effect | +| No errors in report but process crashed | failed samples dropped | collector exit/error recording | +| 1 GiB claim from 16 MiB artifact | tooling/plan confused with evidence | artifact inventory and exact size | +| Old report changes after refactor | candidate source not snapshotted | provenance/digests | +| Speedup changes output | correctness digest/invariant missing | output and independent accuracy tests | + +## Deliberate exclusions + +- Do not run benchmarks before defining the acceptance rule when the result will + decide implementation. +- Do not use one machine to make universal absolute-speed claims. +- Do not treat a benchmark framework's summary as the experiment record. +- Do not optimize only mean/median while ignoring tail, error, memory, or + protected workflows relevant to users. +- Do not remove failed samples as outliers without a predeclared mechanical rule. +- Do not keep a candidate because substantial implementation work was invested. +- Do not infer a completed stress matrix from available scripts. + +## Executable verification + +1. Validate the experiment inventory, source/fixture/report digests, and exact + commands before collection. +2. Run correctness gates for baseline and candidate. +3. Execute balanced/interleaved fresh-process timing and memory schedules. +4. Confirm collectors preserve raw samples, failures, environment, and counts. +5. Recompute comparison statistics from raw reports independently. +6. Verify multiple-comparison adjustment and practical thresholds against known + synthetic inputs/tests. +7. Run protected end-to-end and error workflows. +8. Run named stress lanes within resource authorization and record incomplete + lanes as incomplete. +9. Re-run a representative result in a clean environment or second host before + broad claims. +10. Confirm the conclusion follows the predeclared gate; retain rejection notes. + +## Sources and freshness + +- Attached `wikitext.zip`, observed `experiments/event-shape-study/protocol.md`, + `methods.md`, tools, approach-local snapshots, schedules, raw reports, + comparisons, stress artifacts, and `results.md`; verified 2026-07-17. +- Attached `undent.zip`, normative benchmarking/testing instructions plus + executable timing and memory suites; verified 2026-07-17. + +The thresholds and Deno commands in Wikitext belong to that experiment. Reuse +the experimental controls and evidence discipline; choose product-specific +metrics, effects, runtimes, and gates before collecting new data. diff --git a/skills/build-devtools/references/releases.md b/skills/build-devtools/references/releases.md index e7cca4b..41b1df5 100644 --- a/skills/build-devtools/references/releases.md +++ b/skills/build-devtools/references/releases.md @@ -1,32 +1,350 @@ -# Releases and versioning +# Releases, versioning, and recovery -## Release source +## Contents -Release from a clean immutable commit or tag. Define the one version source and -validate semantic version, prerelease policy, changelog, generated files, and -manifest propagation before publication. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Authority and release state](#authority-and-release-state) +- [Semver is a compatibility decision](#semver-is-a-compatibility-decision) +- [Release notes and changelog generation](#release-notes-and-changelog-generation) +- [Preflight and immutable inputs](#preflight-and-immutable-inputs) +- [Build, attest, and publish](#build-attest-and-publish) +- [Multiple packages and registries](#multiple-packages-and-registries) +- [Post-publication verification](#post-publication-verification) +- [Partial failure, rollback, and repair](#partial-failure-rollback-and-repair) +- [Security and authorization](#security-and-authorization) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Executable verification](#executable-verification) +- [Sources and freshness](#sources-and-freshness) -## Gate sequence +## When to load this reference -1. working tree and revision check; -2. dependency/lock and generated-drift check; -3. format/lint/type/test without broad Markdown formatting; -4. source and generated package builds; -5. package-content and clean-consumer tests; -6. version/changelog/provenance verification; -7. dry run or staging publish where supported; -8. immutable publish and tag/release assets; -9. post-publish clean install and smoke; -10. rollback/yank/deprecation response if post-publish verification fails. +Load this reference for version changes, changelogs, tags, GitHub releases, +registry publication, provenance, release automation, prereleases, yanks, or a +failed/partial release. Planning or reviewing a release does not authorize a +publish, tag, push, registry change, deprecation, or deletion. -## Provenance +## Outcome -Record revision, tool versions, lockfile, build command, package digests, -publisher identity, and CI run. Sign or attest artifacts where the ecosystem and -threat model justify it. +A release is a recoverable state transition from one immutable source revision +to one or more immutable consumer artifacts. Completion requires evidence that: -## Multi-registry publication +- compatibility was classified before choosing the version; +- one version/revision owns every artifact; +- the source tree, lockfiles, generated artifacts, package contents, and release + notes were checked before publication; +- artifacts were built in a controlled environment and their digests/provenance + retained; +- each registry/asset was verified by a clean consumer after publication; +- retries cannot publish different bytes under the same version; +- partial publication and post-release defects have an explicit response; +- credentials and publishing authority are narrow, auditable, and not present + in artifacts. -Generate both artifacts from one source and version. Verify exports, dependencies, -contents, provenance, and clean consumers separately for each registry. Do not -call the release complete because the first publication succeeded. +Treat release creation, registry publication, and deployment as separate state +machines even if one workflow coordinates them. + +## Authority and release state + +Define the canonical tuple: + +```text +release identity = package/product + semantic version + source revision +artifact identity = release identity + target + sha256/integrity +publication identity = artifact identity + registry/channel + timestamp/run +``` + +Choose one version source: a validated release input/tag, a package manifest, or +a release manifest. Other manifests are projections. Reject ambiguity rather +than selecting the highest value. A dirty checkout, untracked generator input, +floating dependency resolution, or uncommitted changelog means the source +revision does not explain the artifact. + +Useful lifecycle states: + +```text +planned -> prepared -> tagged/release-created -> built -> published-per-target + -> post-publish-verified -> complete + \-> failed-partial -> repair/deprecate/supersede +``` + +Persist target-specific state. A GitHub release existing does not mean npm, JSR, +container, binary, documentation, or deployment succeeded. Never infer completion +from one green job when another target was skipped or conditionally false. + +## Semver is a compatibility decision + +Inventory every public contract, not only exported TypeScript names: + +- runtime values, types, subpaths, conditions, bin names, exit codes, stdout; +- configuration keys/defaults/merge order and environment variables; +- persistence schemas, migrations, wire formats, events, cursors, URLs; +- required runtime/OS/architecture/peer/dependency versions; +- CSS/classes/assets/templates and plugin/adapter hooks; +- error classes/codes/retry behavior, performance/resource limits, and security + policy where consumers rely on them. + +Decision model: + +| Change | Normal classification | Required evidence | +|---|---|---| +| Compatible bug fix/internal correction | patch | regression test plus unchanged public contract | +| Backward-compatible capability | minor | old consumer plus new behavior tests | +| Removed/renamed/stricter public behavior | major | migration guide and compatibility fixtures | +| Security response | impact-dependent | advisory, supported-line policy, disclosure plan | +| Prerelease | channel-specific | explicit identifier/sequence and upgrade semantics | + +Zero-major projects often adopt special rules, but tools disagree about whether +`0.x` or `0.0.x` components behave as "major". Changelogen 0.6.2 documents its +own special handling. Encode project policy in tests/config and do not assume a +library's default is the product's contract. + +A dependency update can be breaking when it changes emitted code, supported +engines, peer ranges, generated schemas, native ABI, or transitive types. A +performance "optimization" can be breaking when it changes ordering, memory +limits, precision, error timing, or concurrency. Run compatibility fixtures. + +## Release notes and changelog generation + +Commit history is evidence, not a complete product impact model. Generate a +candidate changelog, then review it against the actual public diff and migration +requirements. Notes should distinguish: + +- user-facing additions, fixes, removals, and security changes; +- upgrade actions, version/runtime boundaries, and deprecations; +- known limitations and deliberately unchanged behavior; +- contributors and commit links when policy permits; +- artifact/checksum/install information when users need it. + +Changelogen 0.6.2 can parse Conventional Commits, select `--from`/`--to`, infer +a bump, update a changelog/version, create commits/tags, publish npm packages, +create canaries, and synchronize GitHub releases. Its breadth increases the +authorization risk: preview/no-output generation is suitable for review, while +`--release`, `--push`, `--publish`, and GitHub release sync are mutations and +must be separately authorized. + +```sh +# Candidate notes from an explicit immutable range; review only. +changelogen --from v1.2.2 --to 4e1c0f7 --no-output + +# Do not run these merely to preview: +# changelogen --release --push +# changelogen --publish +``` + +Pin the tool version. Validate its c12-loaded config and repository field. +Treat author emails/tokens as sensitive. Generated notes do not define migration +policy; schemas, compatibility tests, and maintainers do. + +## Preflight and immutable inputs + +Before creating a tag or release: + +1. confirm requested action and publication targets; +2. require the expected branch/revision and an intentional clean tree; +3. fetch tags/registry state to detect an existing version; +4. restore frozen/locked dependencies without changing the lockfile; +5. validate version syntax, monotonicity, tag prefix, prerelease/channel policy, + manifest propagation, and changelog heading; +6. run scoped formatting (never broad Markdown formatting), lint, typecheck, + unit/integration/compatibility/security tests; +7. run generated-drift checks; +8. build all source/generated packages from clean output directories; +9. inspect packed contents, exports, licenses, engine/peer/dependency metadata; +10. run clean consumers and representative executables; +11. create an artifact manifest with revision, versions, build command, inputs, + target, size, digest, and expected publication name. + +If the version already exists, download its published artifact and compare the +intended bytes/metadata. Most registries are immutable; never try to overwrite a +version. Decide whether this is a safe idempotent retry, a partial publication, +or a new patch/prerelease is required. + +## Build, attest, and publish + +Build from the exact commit/tag with pinned action/tool versions and least +privilege. Prefer short-lived workload identity/trusted publishing over long-lived +tokens. Separate jobs by target so a token for one registry cannot write another. + +Minimum provenance record: + +```json +{ + "name": "@scope/package", + "version": "1.2.3", + "revision": "4e1c0f7...", + "target": "npm", + "artifact": "package-1.2.3.tgz", + "sha256": "...", + "runtime": "deno 2.8.x / node 26", + "lockDigest": "...", + "buildCommand": "deno task build:npm", + "workflowRun": "..." +} +``` + +Use ecosystem provenance/attestations/signatures where supported, but verify +what they prove. An attestation can bind an artifact to a workflow/repository; +it does not prove semver correctness, runtime behavior, dependency safety, or +that the user intended this release. + +Create or publish immutable objects once. Retrying should reuse the retained +artifact, not rebuild it from a mutable branch. If a registry requires a build +inside its publish job, compare the rebuilt digest to the prepared artifact and +fail on mismatch. + +The attached Undent workflows use a useful separation: + +- the release workflow creates a GitHub release from main; +- the publish workflow reacts to a published release or an explicit existing + tag and can retry JSR, npm, or both; +- concurrency does not cancel an in-progress publication; +- JSR and npm have separate jobs/permissions; +- npm can bootstrap with a token for first publication then use trusted + publishing; JSR uses OIDC; +- release version is injected into generated artifacts from the validated tag. + +The example still needs project-specific review. `--allow-dirty` for a manifest +projection must be constrained and explained, and published bytes/digests should +be retained if reproducibility is claimed. + +## Multiple packages and registries + +For a workspace, build an explicit release graph: + +```text +package A@1.2.3 + -> package B@2.0.0 (runtime dependency) + -> adapter C@0.7.0 (peer relationship) +``` + +Decide fixed/independent versioning, internal range rewriting, publish order, +cycle policy, unchanged-package behavior, canary naming, and failure recovery. +Verify packed manifests after workspace protocols/catalog references are +resolved. A topological publish order is not enough if a registry has propagation +delay; clean install after each dependency becomes available or use bounded +retry with exact versions. + +For npm plus JSR or other multi-registry releases: + +- generate both from the same release identity; +- record different expected transforms rather than demanding byte equality; +- verify public entrypoints and behavior separately; +- track status separately so one can be retried; +- do not advance `latest`/stable documentation until supported required targets + are verified, unless policy explicitly allows partial availability. + +Channels/tags (`latest`, `next`, canary) are mutable pointers. Record prior +values before changing them and define rollback. Do not attach a prerelease to +the stable channel by default. + +## Post-publication verification + +Publication success is only registry acceptance. From a new empty consumer: + +- resolve exact version from the public registry; +- inspect metadata, provenance, deprecation/channel, tarball digest, and contents; +- install with supported package managers/runtimes; +- import every public entry, typecheck, and execute representative behavior; +- run CLI `--help`, `--version`, stable-output, failure, and cancellation paths; +- verify source maps/assets/native binaries where promised; +- compare package version/output to release/tag/changelog; +- verify GitHub release assets/checksums and documentation install snippets. + +Test eventual consistency explicitly. A short registry delay is a blocked/pending +check, not a pass and not necessarily a defective artifact. Retry with a deadline, +then report exact target state. + +## Partial failure, rollback, and repair + +Published immutable artifacts generally cannot be rolled back by deletion and +reuse. Prepare target-specific actions before release: + +| Failure | Safe response candidates | +|---|---| +| Artifact not yet published | fix/rebuild before immutable publication | +| One registry published, another failed | retain artifact; retry failed target only; report partial state | +| Bad package published | deprecate/yank if policy permits; publish fixed version; advisory/migration | +| Wrong channel pointer | restore prior pointer after verifying exact version | +| GitHub release only | edit/withdraw release per policy without inventing registry success | +| Compromised credential/artifact | stop pipeline, revoke, preserve evidence, advisory, superseding release | +| Deployment failed after library publication | roll back deployment separately; do not rewrite package history | + +Record rollback authority and retention windows. Unpublish can break downstream +builds and is time-limited or restricted in many registries; treat it as an +exception, not the normal recovery mechanism. A tag that has been consumed +should not be force-moved. Prefer a corrected version and transparent notes. + +## Security and authorization + +- Use protected environments/approvals for production publication. +- Grant `contents`, `packages`, `id-token`, or registry scopes only to the job + that needs them. +- Pin third-party actions and release tools to reviewed versions/revisions. +- Never expose tokens in command args, generated config, npm logs, provenance, + changelogs, or package contents. +- Do not run package lifecycle scripts during verification unless required and + explicitly sandboxed/reviewed. +- Preserve audit logs, artifact digests, source revision, approvals, and failed + attempts. +- Verify repository/package identity before granting trusted publishing; a + similarly named package or fork is not the target. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Tag, manifest, and CLI version differ | multiple version authorities | release-input propagation | +| Generated notes omit a breaking change | commit grammar used as impact model | public diff and compatibility fixtures | +| Re-run produces different tarball | rebuild from mutable/nondeterministic inputs | retained artifact and build provenance | +| npm succeeds, JSR missing | target state collapsed into one job/result | per-target ledger and retry | +| Clean install gets old behavior | mutable channel/cache or wrong registry | exact version/digest and registry metadata | +| Package exists but release job wants to republish | retry not idempotent | compare existing artifact and skip/repair | +| Provenance exists but package is broken | attestation mistaken for validation | consumer behavior and semver gates | +| Workspace consumer cannot resolve internal package | protocol/range not rewritten or order delay | packed manifest and registry state | +| Release cancelled midway | cancellation/concurrency policy unsafe | workflow concurrency and target states | +| Token appears in logs/artifact | secret passed through output/config | redaction, args/env, archive scan | + +## Deliberate exclusions + +- Do not publish, tag, push, deprecate, unpublish, or change channels during a + review or dry run. +- Do not use a broad release tool command when the task only requires notes or + a version proposal. +- Do not infer semver only from commit prefixes. +- Do not release from a dirty tree, mutable branch checkout, or unlocked graph. +- Do not rebuild an already approved artifact during a partial retry. +- Do not call a release complete until required post-publish consumers pass. +- Do not delete/yank history merely to make automation green. + +## Executable verification + +1. Validate exact revision/clean tree/locked dependency restore. +2. Generate candidate notes from explicit refs and compare them with public + contract diffs and migration fixtures. +3. Run the complete source/generated/pack/consumer gate. +4. Build twice in clean isolated directories and compare artifact manifests. +5. Exercise a dry-run or staging registry when supported. +6. Simulate existing-version, one-target failure, registry delay, cancelled job, + and post-publish defect; prove the state ledger selects a safe action. +7. Verify workflow permissions and ensure mutation steps require authorization. +8. After publication, install the exact public version and verify behavior. +9. Validate provenance/signatures/checksums independently from behavior. +10. Preserve a machine-readable release report including blocked and failed + checks, not just a success boolean. + +## Sources and freshness + +- Attached `undent.zip`, observed `.github/workflows/release.yml`, + `.github/workflows/publish.yml`, `.releaserc.json`, `scripts/build_npm.ts`, and + release checklist; verified 2026-07-17. +- Changelogen 0.6.2 published README/declarations/manifest; source record + `changelogen-0-6-2`, verified 2026-07-17. +- Attached production CLI guidebook v1.1, normative release gates, packaged + executable lifecycle, completion/man artifacts, and rollback expectations; + verified 2026-07-13. + +Registry rules, trusted-publisher requirements, action versions, and tool CLI +options change. Recheck official registry documentation and the pinned tool's +generated help immediately before changing a production release workflow. diff --git a/skills/build-devtools/references/toolchains.md b/skills/build-devtools/references/toolchains.md index 0f6befc..09230d7 100644 --- a/skills/build-devtools/references/toolchains.md +++ b/skills/build-devtools/references/toolchains.md @@ -13,8 +13,11 @@ licensed, verified binary. ## Aube and unfamiliar tools -Resolve canonical repository, package identity, version, config schema, generated -files, and task behavior first. If source cannot be found, record the name as an +Aube is a verified Node.js package manager in the jdx ecosystem. Load +[mise-aube.md](mise-aube.md) for current lockfile, workspace, lifecycle-build, +security, and Mise-integration behavior. Other unfamiliar tools still require +canonical repository, package identity, version, config schema, generated files, +and task behavior before use. If source cannot be found, record the name as an unverified discovery hint and do not invent configuration keys. ## Task parity diff --git a/skills/build-libraries/SKILL.md b/skills/build-libraries/SKILL.md new file mode 100644 index 0000000..7e453ad --- /dev/null +++ b/skills/build-libraries/SKILL.md @@ -0,0 +1,147 @@ +--- +name: build-libraries +description: Design, implement, refactor, review, benchmark, package, or verify reusable software libraries and SDKs. Use for public APIs, library-first or use-case-first architecture, composability, tree-shaking, ESM exports, optional integrations, arrays and iterables, async iterators, streams, batching, data-oriented design, explicit resource management, performance budgets, resumability boundaries, or extracting a reusable core from a CLI or application. Do not use for an incidental helper or an application-only internal module with no reusable consumer contract. +--- + +# Build libraries + +A library is a programming model, not code moved into a package directory. Design +from concrete consumer call sites, domain values, workload shape, resource +ownership, and selective adoption. Keep common use cases deep and convenient +while preserving independently useful lower-level capabilities. + +## Composition contract + +`deliver-software` owns request authority, repository-wide implementation, +cleanup, and the final verdict. `explore-ecosystems` owns dependency topology and +source evidence. `build-devtools` owns build automation, generated artifacts, +release mechanics, and toolchain parity. `build-clis` owns command language, +configuration sources, terminal interaction, standard streams, and exit status. +`build-workflows` owns durable orchestration, workers, leases, timers, signals, +and persisted execution authority. `build-data` owns database, query, migration, +and projection contracts. + +This skill owns: + +- the reusable public programming model and information-hiding boundaries; +- value, data-flow, capability, policy, ecosystem, lifecycle, package, and + operational composition; +- cardinality and flow contracts such as values, arrays, iterables, async + iterables, streams, and explicit batches; +- public versus internal data representations and data-oriented hot paths; +- library resource acquisition, ownership, borrowing, transfer, cancellation, + and disposal; +- ESM entrypoints, public subpaths, side-effect boundaries, optional adapters, + and selective-adoption evidence; +- library workload budgets, benchmark stories, and resource-regression gates; +- restartable and checkpoint-resumable library contracts, while deferring + durable workflow execution to `build-workflows`; +- migration from application-shaped abstractions to reusable capabilities. + +When composed with another skill, inspect the repository once, agree on one +ownership map, and produce one integrated plan. Do not create duplicate owners +for configuration, logging, persistence, workflows, packaging, or rendering. + +## Evidence preflight + +Before changing a library surface, inspect: + +- intended consumers, current call sites, application adapters, tests, examples, + and published documentation; +- package manifests, exports, imports, side-effect metadata, build output, + declaration output, and clean-consumer behavior; +- import-time work, eager registries, globals, singletons, decorators, and + optional dependency reachability; +- request, result, failure, event, cancellation, and resource-lifetime + contracts; +- values that are materialized, streamed, buffered, copied, retained, indexed, + serialized, or persisted; +- concurrency, queue capacity, batch size, peak memory, CPU, latency, startup, + cleanup, and recovery behavior; +- checkpoint, idempotency, compatibility, and versioning claims; +- application concerns accidentally embedded in reusable code, including argv, + environment discovery, prompts, terminal rendering, process exits, and logger + configuration. + +Separate intended architecture, observed implementation, documented behavior, +published artifact behavior, and behavior proven in a clean consumer. A source +file that looks tree-shakable is not proof that the distributed package shakes. + +## Core rules + +1. Start from concrete use cases and desired consumer call sites. Do not extract + the current application's execution sequence as the public architecture. +2. Organize modules around domain knowledge and design decisions likely to + change, not generic `Runtime`, `Context`, `Stage`, or `Handler` machinery. +3. Provide a deep common-case facade and independently useful lower-level + capabilities. Do not make consumers reconstruct the library internally. +4. Compose through explicit values, protocols, focused capabilities, policies, + lifetimes, and ecosystem contracts. Do not reduce strategic dependencies to + lowest-common-denominator interfaces. +5. Choose the narrowest truthful data shape. Arrays are deliberate + materialization boundaries; iterables are lazy synchronous sequences; async + iterables are incremental asynchronous records; streams own backpressure and + transport semantics; batches amortize per-record overhead. +6. Model resource ownership explicitly. Prefer `Disposable`, `AsyncDisposable`, + `using`, `await using`, and disposal stacks where the target runtime supports + them; otherwise preserve the same ownership contract with `try/finally`. +7. Bound admission, concurrency, buffering, open resources, batch sizes, retries, + and cleanup. An async iterator with an unbounded producer is not a bounded + pipeline. +8. Apply data-oriented design to measured hot paths. Start from transforms, + access patterns, volumes, locality, allocation, and lifetime. Do not replace + readable objects with typed arrays by aesthetic preference. +9. Keep reusable modules import-safe. Importing a capability must not configure + logging, load project configuration, launch resources, install signal + handlers, mutate registries, or import unrelated adapters. +10. Preserve ESM and explicit public subpaths. Keep integrations physically + separate, declare side effects truthfully, and verify selective adoption + against built artifacts and clean consumers. +11. Name recovery guarantees precisely: restartable, checkpoint-resumable, or + durably orchestrated. Commit checkpoints only after required outputs are + durable and replay-safe. +12. Treat performance as a workload contract. Record absolute and relative + results, variability, correctness oracles, peak and retained memory, + resource counts, startup, tail latency, cleanup, and recovery where relevant. +13. Treat every public export and observable behavior as compatibility surface. + Export only what the project is prepared to version and support. +14. Add or update evals and executable acceptance checks for every material + library rule, public contract, packaging change, performance claim, or + recovery claim. + +## Reference routing + +- [architecture.md](references/architecture.md): use-case-first design, deep + modules, information hiding, public contracts, and application boundaries. +- [composition.md](references/composition.md): composition at every scale, + strategic dependencies, LogTape, c12, defu, unstorage, Hookable, Optique, and + extension ownership. +- [data-flow.md](references/data-flow.md): arrays, iterables, generators, async + iterables, streams, batching, materialization, early termination, and + backpressure. +- [data-oriented-design.md](references/data-oriented-design.md): transform-first + design, hot and cold data, representation choices, allocation, locality, and + public versus internal shapes. +- [resources-performance.md](references/resources-performance.md): explicit + resource management, cancellation, disposal, budgets, memory, CPU, latency, + concurrency, and representative benchmarks. +- [packaging.md](references/packaging.md): ESM, exports, subpaths, side effects, + optional integrations, build output, tree-shaking, declarations, and clean + consumer verification. +- [recovery-refactoring.md](references/recovery-refactoring.md): restart and + resume guarantees, committed checkpoints, idempotency, CLI-first extraction, + second consumers, and compatibility removal. +- [verification.md](references/verification.md): API, artifact, streaming, + lifecycle, performance, recovery, compatibility, and cross-skill test + matrices. + +## Completion gate + +Do not call library work complete until representative programmatic consumers +exercise the intended public entrypoints; imports do not perform surprising +work; unused integrations are absent from the built consumer graph; data flow is +incremental where claimed; early termination releases resources; concurrency +and buffering remain bounded; public results and failures retain their machine +contracts; package exports and declarations match; clean consumers pass; and +performance or recovery claims have executable evidence. Report blocked checks +separately from failures and never infer artifact behavior from source alone. diff --git a/skills/build-libraries/agents/openai.yaml b/skills/build-libraries/agents/openai.yaml new file mode 100644 index 0000000..f47f778 --- /dev/null +++ b/skills/build-libraries/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Build Libraries" + short_description: "Build composable, efficient reusable libraries." + default_prompt: "Use $build-libraries to design, implement, refactor, or verify this reusable library." diff --git a/skills/build-libraries/references/architecture.md b/skills/build-libraries/references/architecture.md new file mode 100644 index 0000000..4aa6c0b --- /dev/null +++ b/skills/build-libraries/references/architecture.md @@ -0,0 +1,290 @@ +# Library architecture and programming models + +Use this reference when creating a new library, extracting reusable code from an +application, reviewing a public API, or deciding whether an abstraction is a +library, framework, application service, or internal module. + +## Outcome + +Produce a programming model that lets a consumer perform meaningful domain work +without constructing the originating application. The public surface should be +smaller than the knowledge and implementation it hides. + +## Start from the desired consumer + +Write concrete call sites before designing generic machinery: + +```ts +import { analyzeDomains } from "@kaiju/analysis"; + +const result = await analyzeDomains( + { domains: ["example.com"] }, + { collector, detector, factRepository }, + { signal }, +); +``` + +Add at least one common-case call and one advanced composition call. Read them as +consumer code. The API should communicate domain intent, ownership, cardinality, +async behavior, failure shape, and resource lifetime without requiring knowledge +of the implementation flowchart. + +Do not begin with names such as `Runtime`, `Context`, `Stage`, `Executor`, +`Plugin`, or `Manager`. Those names are sometimes justified, but they must earn +their place by describing a product capability rather than hiding unresolved +ownership. + +## Use-case first, not chronology first + +A CLI or application often performs: + +```text +parse -> configure -> acquire -> collect -> verify -> detect -> persist +``` + +Those are execution phases. They are not automatically module boundaries. A +chronology-first extraction tends to preserve shared context, hidden sequencing, +and temporal coupling. + +Prefer boundaries around domain knowledge and change axes: + +```text +Domain analysis + request, result, failures, events, orchestration + +Observation collection + targets, browser/archive adapters, observation contract + +Detection + rule compilation, indexes, batch evaluation + +Fact resolution + evidence combination, contradictions, freshness, confidence + +Persistence + repository contract, artifacts, receipts +``` + +A module should hide a design decision or body of knowledge that other modules +do not need to understand. Processing steps can remain implementation details or +observable events. + +## Separate application and library ownership + +A reusable library normally should not own: + +- argv, environment, current working directory, project config discovery, or + interactive prompts; +- terminal formatting, colour, progress bars, stdout, stderr, or process exits; +- application logging configuration, sink selection, or telemetry consent; +- process signal installation; +- deployment-specific dependency selection; +- global singleton lifecycle. + +It may accept resolved values and focused capabilities produced by those owners. +It may emit structured domain events and library diagnostics. The application +composition root chooses concrete adapters and owns process lifecycle. + +## Design four stable contracts first + +For a named use case, define: + +1. request; +2. result; +3. structured failures; +4. observable domain events. + +```ts +export interface AnalyzeDomainsRequest { + readonly domains: readonly string[]; +} + +export interface AnalysisResult { + readonly runId: string; + readonly targets: readonly TargetResult[]; + readonly artifacts: readonly ArtifactReference[]; +} + +export type AnalysisFailure = + | InvalidTargetFailure + | CollectionFailure + | PersistenceFailure + | CancellationFailure; + +export type AnalysisEvent = + | AnalysisStarted + | TargetCompleted + | CheckpointCommitted + | AnalysisCompleted; +``` + +Do not expose internal stage objects, queue types, mutable contexts, or adapter +options merely because the first implementation uses them. + +## Deep modules and layered surfaces + +A good library commonly offers: + +```text +Convenient named use cases + analyzeDomains() + analyzeArchive() + verifyObservations() + +Independently useful capabilities + collectDomains() + evaluateObservationBatches() + resolveFactCandidates() + writeArtifacts() + +Private mechanisms + queues, indexes, buffers, lease schedulers, retries +``` + +The facade owns a supported composition for common consumers. Lower-level +capabilities remain available where independent use is real. Private mechanisms +remain private so they can change. + +Do not create dozens of shallow wrappers whose public surface equals their +implementation. Prefer modules with small interfaces and substantial hidden +knowledge. + +## Avoid temporal coupling + +This contract hides a state machine: + +```ts +await runtime.initialize(); +await runtime.collect(); +await runtime.verify(); +await runtime.commit(); +await runtime.finalize(); +``` + +Prefer a named operation: + +```ts +const result = await analyzeDomains(request, capabilities, { signal }); +``` + +When lifecycle states are genuinely public, model them as domain operations on a +run or session: + +```ts +const run = await client.startAnalysis(request); +await client.inspectAnalysis(run.id); +await client.cancelAnalysis(run.id); +``` + +Do not make all consumers manually traverse an internal lifecycle. + +## Frameworks are valid when the framework is the product + +A stage framework, plugin host, compiler pipeline, UI renderer, workflow engine, +or dependency-injection runtime can be the correct library when consumers are +supposed to define execution units and the framework's lifecycle is the public +capability. + +Require evidence: + +- multiple independent consumers need to define or reorder units; +- third-party extension is an explicit requirement; +- ordering, lifecycle, compatibility, and failure semantics are documented; +- the framework offers meaningful leverage beyond ordinary functions; +- the framework surface is versioned and tested as the product. + +Do not create a framework merely to organize code controlled by one repository. + +## Justify architecture decisions + +Do not defend a boundary with slogans such as “library first,” “composable,” +“tree-shakable,” or “best practice.” Show the path from the situation to the +decision. + +For a material architecture choice, state: + +1. the objective or failure condition being protected; +2. hard constraints separately from preferences; +3. the diagnosis and causal mechanism creating the problem; +4. credible alternatives, including inaction, delay, a partial extraction, and a + reversible pilot where applicable; +5. the exact requirement, risk threshold, or evidence that rejects each weaker + option; +6. costs, risks, and trade-offs accepted by the chosen design; +7. assumptions and defeaters that would change the decision; +8. whether the result is necessary, conditionally necessary, prudent, or merely + preferred. + +For example, “split every module into a package because libraries should be +tree-shakable” is not a justification. A defensible conclusion may instead be: + +```text +Given two independent consumers, a measured cold-start budget, and an optional +browser dependency that dominates the core bundle, expose the browser adapter +through a separate subpath. Keep the remaining internal modules private because +package-level separation would not satisfy another consumer or constraint. +``` + +Prefer the least disruptive and most reversible boundary that still satisfies +the protected objective. Record what evidence would invalidate the choice. + +## Public compatibility discipline + +Treat these as compatibility surfaces: + +- exported names and subpaths; +- type relationships and accepted structural values; +- result ordering and cardinality; +- error identity and error timing; +- import-time effects; +- resource ownership and cleanup timing; +- cancellation behavior; +- emitted events; +- serialization formats; +- performance characteristics explicitly promised by the project. + +Keep internal types internal. Add an export only when a consumer contract needs +it and the project is prepared to support it. + +## Decision questions + +Before accepting an abstraction, ask: + +1. Which concrete consumers need it? +2. What domain knowledge does it hide? +3. What can change behind it without breaking consumers? +4. Can one capability be used without constructing the whole system? +5. Is the abstraction organizing knowledge or merely restating control flow? +6. Does the common case remain simple? +7. Is advanced composition possible without importing private machinery? +8. Which observable behaviors become compatibility commitments? + +## Failure signatures + +- a `core` package still reads argv, discovers config, formats terminal output, + or exits the process; +- every function accepts one giant context containing unrelated capabilities; +- public classes represent phases rather than domain concepts; +- consumers must call methods in an undocumented order; +- the library exports internal queues, stages, and concrete adapters by default; +- a facade is absent, forcing consumers to reconstruct the implementation; +- a framework is created before a second independent extension scenario exists; +- moving files into packages is presented as successful modularization. + +## Verification + +- compile at least two realistic consumer call sites; +- exercise common and advanced use without application globals; +- test failures and cancellation through the public entrypoint; +- ensure the CLI or service is a client of the same public API; +- remove an internal mechanism or replace an adapter without changing unrelated + consumer code; +- inspect the published exports rather than source folders alone. + +## Sources and freshness + +- Library-first guidebook, reviewed 2026-07-23. +- David Parnas, "On the Criteria To Be Used in Decomposing Systems into + Modules," information-hiding basis. +- Swift API Design Guidelines, clarity at the point of use. +- Semantic Versioning, explicit public API and compatibility discipline. diff --git a/skills/build-libraries/references/composition.md b/skills/build-libraries/references/composition.md new file mode 100644 index 0000000..692c8d9 --- /dev/null +++ b/skills/build-libraries/references/composition.md @@ -0,0 +1,251 @@ +# Composition at every scale + +Use this reference when a library must combine functions, policies, adapters, +configuration systems, observability, extension points, or operational +capabilities without becoming a universal framework. + +## Composition model + +Evaluate composition at all of these scales: + +| Scale | Contract | +| --- | --- | +| Value | Functions accept and return ordinary domain values. | +| Data flow | Arrays, iterables, async iterables, streams, or batches connect operations. | +| Capability | Use cases depend on focused behavior contracts. | +| Policy | Callers select bounded policy without replacing the operation. | +| Ecosystem | The library participates in established systems without owning them. | +| Lifecycle | Resources can be acquired, borrowed, transferred, and disposed. | +| Package | Consumers import only selected capabilities and adapters. | +| Operational | Work can participate in logging, storage, checkpoints, and workers. | + +A library that composes only at the function level can still integrate poorly +with configuration, observability, storage, packaging, or recovery. + +## Compose through semantic contracts + +Prefer focused capabilities: + +```ts +export interface AnalyzeCapabilities { + readonly collect: CollectTargets; + readonly detect: DetectBatches; + readonly persist: PersistBatches; +} +``` + +Avoid one giant runtime: + +```ts +export interface RuntimeContext { + readonly config: unknown; + readonly logger: unknown; + readonly browser: unknown; + readonly queue: unknown; + readonly storage: unknown; + readonly metrics: unknown; + readonly terminal: unknown; +} +``` + +A narrow contract is not automatically good. It must preserve the semantics the +consumer needs. `log(message: string)` is smaller than LogTape, but it erases +structured properties, categories, contexts, filters, lazy evaluation, +redaction, sink routing, testing, and disposal. + +## Strategic dependencies versus adapters + +Use a dependency directly when: + +- its semantics are intentionally part of the library ecosystem; +- hiding it would require a weaker duplicate abstraction; +- consumers already benefit from the shared contract; +- the dependency is stable enough for the intended compatibility policy; +- application and library ownership remain distinct. + +Define a project-owned interface when: + +- several implementations genuinely satisfy the same required semantics; +- the project owns additional invariants; +- the dependency would leak host-specific concerns into the domain; +- testing needs a focused fake or in-memory implementation; +- replacement is a real scenario rather than speculative optionality. + +Do not wrap every dependency reflexively. Do not expose every dependency +reflexively. State the compatibility and ownership decision. + +## LogTape composition + +Library code may use hierarchical LogTape categories and structured properties: + +```ts +import { getLogger, lazy } from "@logtape/logtape"; + +const logger = getLogger(["kaiju", "analysis"]); + +logger.debug("Compiled {ruleCount} rules", { + ruleCount, + indexStats: lazy(() => inspectIndex(index)), +}); +``` + +Library code must not call `configure()` or select application sinks. The +application owns categories, filters, formatters, redaction, file or telemetry +routes, context-local storage, flushing, and disposal. + +Keep domain events separate from diagnostics: + +```ts +export type AnalysisEvent = + | { readonly type: "analysis.started"; readonly runId: string } + | { readonly type: "batch.committed"; readonly batchId: string } + | { readonly type: "analysis.completed"; readonly runId: string }; +``` + +Domain events are a stable operational contract. LogTape records are support and +observability information. The application may route domain events into LogTape, +a terminal renderer, a socket, or durable storage. + +## c12 and defu composition + +c12 may own application configuration discovery, formats, environment branches, +`extends`, factories, layer metadata, and watching. defu may assist recursive +merge mechanics. Neither should become the domain library's configuration +language by accident. + +Preferred boundary: + +```text +application composition root + c12 discovery and layer loading + -> application-owned merge algebra + -> authored/resolved/runtime schema validation + -> focused resolved options passed into library +``` + +The library receives values such as `AnalysisPolicy`, not a c12 result, current +working directory, or defu callback. Preserve source provenance in the +application when operators need `config explain`. + +Array replacement, append/prepend operations, atomic discriminated unions, +null/reset semantics, and default timing are application domain decisions. +Generic deep merge is not a substitute for a public configuration contract. + +## unstorage composition + +Unstorage is useful for runtime-neutral key-value adapters, mounts, metadata, +snapshots, watches, and many separately imported drivers. Preserve driver +capability differences. + +A common API does not imply common guarantees: + +- memory is not durable; +- filesystem and remote stores have different atomicity and visibility; +- watch support varies; +- transactions, compare-and-set, leases, TTL, and metadata may be optional or + driver-specific; +- disposal is part of the selected driver lifecycle. + +If the library requires atomic checkpoint advancement, leases, or fencing, make +those semantics explicit in a stronger project-owned contract. Do not claim +that any `Storage` implementation satisfies them. + +## Hookable and extension points + +Hookable can provide typed registration, removal, serial or parallel invocation, +error handling, and a smaller core. Use hooks when third-party extension or +runtime registration is a real product requirement. + +Before exposing a hook, define: + +- invocation order and whether it is stable; +- serial versus parallel execution; +- error aggregation and cancellation; +- input mutability and output combination; +- reentrancy; +- registration and disposal; +- compatibility and deprecation policy; +- performance cost on hot paths. + +Prefer ordinary function composition for code controlled by one repository. +Hidden callback graphs are harder to reason about than explicit calls. + +## Optique and application adapters + +Optique can express command grammar, source terms, help, completion, manuals, +and runners as composable values. It belongs at the CLI boundary. A reusable +library should receive validated domain requests rather than Optique parser +values, `DeferredValue`, or process runner state. + +The CLI may be one first-party adapter: + +```text +Optique parser + c12 resolution + LogTape configuration + -> AnalyzeDomainsRequest + -> @kaiju/analysis + -> result/failure/events + -> CLI renderer and exit mapping +``` + +## Optional integrations + +Separate optional integrations physically and semantically: + +```text +@scope/library +@scope/library/browser.js +@scope/library/unstorage.js +@scope/library/logtape.js +@scope/library/temporal.js +``` + +Do not import all integrations into a root registry. Let the application select +and compose them. Dynamic loading is appropriate when selection is genuinely +runtime-driven; static imports are preferable when maximum build-time selection +and tree-shaking matter. + +## Extension versus ecosystem checklist + +Ask: + +1. Is this an established ecosystem contract or a project-specific capability? +2. Who configures it, and who merely emits or consumes values? +3. Which semantics would a generic wrapper erase? +4. Are implementations actually substitutable under the required guarantees? +5. Is runtime extension a product requirement? +6. What is the cleanup and unregistration contract? +7. Can consumers import the integration separately? +8. Does the integration add hot-path or startup cost when unused? + +## Failure signatures + +- every dependency is hidden behind interfaces that preserve only method names; +- the library configures LogTape or installs global sinks; +- c12 and defu objects leak into domain APIs; +- any unstorage driver is described as a durable transactional checkpoint store; +- hooks organize internal code despite no third-party extension requirement; +- optional adapters are imported by the root module; +- two parsers, loggers, config loaders, or storage owners are stacked without a + coexistence contract; +- ecosystem composition is measured by number of installed sibling packages. + +## Verification + +- run the library with no LogTape application configuration and with multiple + application sink configurations; +- prove domain events remain usable independently of diagnostics; +- resolve configuration once in the application and call the library with plain + validated values; +- test required storage capabilities against each supported driver; +- register, invoke, remove, and dispose hooks when hooks are public; +- inspect bundle graphs to ensure optional integrations are absent when unused; +- test a second application adapter without changing the domain use case. + +## Sources and freshness + +- LogTape library-author guidance and configuration manual, reviewed + 2026-07-23. +- c12, defu, unstorage, and Hookable published source and documentation, + reviewed 2026-07-23. +- Optique official documentation, reviewed 2026-07-23. +- Library-first guidebook, reviewed 2026-07-23. diff --git a/skills/build-libraries/references/data-flow.md b/skills/build-libraries/references/data-flow.md new file mode 100644 index 0000000..b63acf0 --- /dev/null +++ b/skills/build-libraries/references/data-flow.md @@ -0,0 +1,296 @@ +# Data-flow contracts: arrays, iterables, async iterables, streams, and batches + +Use this reference when choosing an API's collection shape, removing accidental +materialization, designing incremental processing, or connecting domain records +to byte-oriented transport. + +## Choose the narrowest truthful shape + +| Shape | Contract | Typical use | +| --- | --- | --- | +| `T` | One immediate value | compiled rule, parsed definition | +| `Promise` | One eventual value | transaction result, remote metadata | +| `readonly T[]` | Complete bounded reusable collection | rule set, final summary | +| `Iterable` | Lazy synchronous sequence | filtering or mapping existing data | +| `AsyncIterable` | Incremental asynchronous records | observations, pages, detections | +| `ReadableStream` | queued readable source with cancellation and backpressure | bytes, transport, standard pipelines | +| `WritableStream` | incremental sink | archive, upload, encoder destination | +| `TransformStream` | backpressured transform | compression, framing, decoding | +| `AsyncIterable` | incremental batches | high-volume domain pipeline | + +Do not expose a union of every sequence type unless the operation genuinely +accepts all of them with defined semantics. A universal sequence type pushes +replayability, ownership, cancellation, cardinality, and backpressure questions +to every caller. + +## Arrays are explicit materialization boundaries + +Use an array when the operation needs a complete snapshot, repeated traversal, +random access, sorting, grouping, global aggregation, atomic validation, or a +known small bound. + +```ts +export interface DetectionRuleSet { + readonly version: string; + readonly rules: readonly DetectionRule[]; +} +``` + +Avoid returning `Promise` for a large or live source merely +because it is easy to implement. Materialization delays first output, retains all +records, prevents pipeline overlap, and increases loss when cancellation occurs. + +Provide explicit collectors when callers need an array: + +```ts +export async function collectToArray( + source: AsyncIterable, + options: { readonly limit?: number } = {}, +): Promise { + const values: T[] = []; + for await (const value of source) { + if (options.limit !== undefined && values.length >= options.limit) { + throw new RangeError("Collection limit exceeded"); + } + values.push(value); + } + return values; +} +``` + +The call site now makes buffering visible. + +## Iterables and generators + +Use `Iterable` for lazy synchronous transforms: + +```ts +export function* normalizeObservations( + source: Iterable, +): Iterable { + for (const value of source) { + const normalized = normalizeObservation(value); + if (normalized !== undefined) yield normalized; + } +} +``` + +Expose the protocol, not the implementation: + +```ts +export function normalizeObservations( + source: Iterable, +): Iterable; +``` + +Do not promise `Generator` unless callers require generator-specific +`return()`, `throw()`, or return-value semantics. + +Document whether an iterable is reusable or single-pass. A generator object is +single-pass. A container that creates a new iterator may be replayable. + +## Async iterables and async generators + +Use `AsyncIterable` for domain values produced over time: + +```ts +export async function* verifyObservations( + source: AsyncIterable, + verifier: ObservationVerifier, + options: { readonly signal?: AbortSignal } = {}, +): AsyncIterable { + for await (const observation of source) { + options.signal?.throwIfAborted(); + const result = await verifier.verify(observation, options); + if (result.accepted) yield result.observation; + } +} +``` + +Expose `AsyncIterable` rather than `AsyncGenerator` unless consumers need +the generator's implementation-specific methods or return type. + +An async iterable is pull-shaped at the consumer boundary. It does not guarantee +that the producer is bounded. Inspect internal queues, promises, worker pools, +and retained buffers. + +## Early termination + +Consumers can stop: + +```ts +for await (const detection of detections) { + if (isDesiredDetection(detection)) break; +} +``` + +Define what happens to: + +- pending network and browser work; +- queued items and worker leases; +- stream readers and locks; +- partial batches; +- output writers; +- temporary files; +- resource handles; +- checkpoint state. + +Async generators should clean up in `finally`. Streams cancel by default when +an async iterator returns early unless cancellation is prevented. Custom +iterators must implement `return()` when early close owns cleanup. + +Test early break, consumer error, source error, abort, and normal exhaustion. + +## Streams + +Use web streams when queueing strategy, backpressure, locking, piping, +cancellation, byte transport, or platform integration is part of the contract. + +```ts +export function encodeObservations( + source: AsyncIterable, +): ReadableStream; + +export function decodeObservations( + source: ReadableStream, +): AsyncIterable; +``` + +Do not call an async iterable a stream when the API does not offer stream +backpressure or stream operations. Do not wrap domain objects in a stream merely +for terminology when `AsyncIterable` is simpler and sufficient. + +## Backpressure and bounded buffers + +A pipeline is bounded only when every producer/consumer boundary has a limit: + +```text +source admission + -> queue capacity Q + -> workers C + -> output batches B + -> destination in-flight limit W +``` + +Separate: + +- admission: how much work enters the system; +- concurrency: how many operations execute; +- buffering: how many completed or pending values are retained; +- ordering: whether results can be emitted as completed; +- destination pressure: how slow sinks propagate upstream. + +An async iterator over an unbounded promise set is still unbounded. + +## Stream between subsystems, batch within subsystems + +Per-record iteration can create one suspension, object allocation, callback, +log record, database write, or network frame per item. Use explicit batches to +amortize overhead: + +```ts +export interface ObservationBatch { + readonly values: readonly Observation[]; + readonly encodedBytes: number; +} + +export function collectObservationBatches( + targets: Iterable, + options: CollectionOptions, +): AsyncIterable; +``` + +Bound batches by one or more of: + +- record count; +- encoded bytes; +- elapsed time; +- destination limits; +- memory budget; +- checkpoint frequency. + +Do not choose one global batch size for every stage. Network collection, +detection kernels, database inserts, and archive writes can have different +optimal bounds. + +## Ordering and concurrency + +State ordering explicitly: + +```ts +export type ResultOrdering = "input" | "completion"; +``` + +Input ordering may require buffering slow gaps. Completion ordering reduces +latency and memory but changes observable order. Preserve deterministic identity +when order is not guaranteed. + +Bound concurrent mapping rather than creating all promises: + +```ts +export interface ConcurrentMapOptions { + readonly concurrency: number; + readonly maxBuffered: number; + readonly ordering: ResultOrdering; + readonly signal?: AbortSignal; +} +``` + +## Materialization audit + +Trace each boundary: + +```text +source + -> decode + -> normalize + -> verify + -> detect + -> resolve + -> persist +``` + +For each arrow record: + +- input and output shape; +- maximum cardinality and bytes; +- replayability; +- ownership; +- concurrency and buffer bound; +- early-termination behavior; +- checkpoint boundary; +- reason for any full materialization. + +One hidden `await Array.fromAsync(...)`, `Promise.all(...)`, global accumulator, +or writer that buffers the full source can collapse an otherwise incremental +pipeline. + +## Failure signatures + +- large operations return `Promise` with no explicit bound; +- generator implementation types leak into the public contract unnecessarily; +- an async iterable starts all work eagerly; +- early break leaves pages, readers, workers, or temporary files open; +- `Promise.all()` scales with total input cardinality; +- queues grow while throughput metrics appear healthy; +- every record is logged or persisted independently on a hot path; +- one batch size is applied to unrelated subsystems; +- input ordering is promised without accounting for gap buffering; +- a byte stream is converted to one giant `Uint8Array` before processing. + +## Verification + +- assert first-item latency separately from completion latency; +- run a large input under a peak-memory limit; +- instrument maximum queue depth, in-flight work, and buffered bytes; +- break after a small number of items and assert upstream cleanup; +- inject a slow destination and prove source production slows; +- compare per-record and batched throughput with identical outputs; +- verify ordering under skewed task durations; +- assert a deliberate materialization limit and failure behavior. + +## Sources and freshness + +- WHATWG Streams Standard, reviewed 2026-07-23. +- TypeScript async iterator and generator documentation, reviewed 2026-07-23. +- Library-first guidebook, reviewed 2026-07-23. diff --git a/skills/build-libraries/references/data-oriented-design.md b/skills/build-libraries/references/data-oriented-design.md new file mode 100644 index 0000000..085cf8c --- /dev/null +++ b/skills/build-libraries/references/data-oriented-design.md @@ -0,0 +1,253 @@ +# Data-oriented design for TypeScript libraries + +Use this reference when performance depends on data volume, layout, access +patterns, allocations, locality, serialization, or hot transformation kernels. +Data-oriented design is not the same as data-driven design. + +## Correct definition + +Data-oriented design starts from the data and the transformations applied to it: + +- what values exist; +- how many exist; +- how they arrive and leave; +- which fields are read or written together; +- which fields are hot or cold; +- how long values live; +- how often they are copied, allocated, compared, indexed, or serialized; +- what CPU, memory, cache, GC, I/O, and runtime constraints matter. + +Data-driven design represents variable policy or definitions as data. A rule set +can be data-driven while its compiled index and evaluation batches are +data-oriented. + +## Start with the transformation graph + +Document the actual data path before selecting classes or package boundaries: + +```text +TargetDefinition[] + -> normalized target IDs + -> observation batches + -> detector index probes + -> fact candidates + -> aggregation state + -> persisted fact batches +``` + +For each transform record: + +- expected and maximum cardinality; +- average and maximum encoded size; +- fields read and written; +- allocation and copy behavior; +- ordering requirement; +- concurrency and batch policy; +- lifetime and retention; +- error and rejection path; +- materialization and persistence boundary. + +Optimize the dominant transform, not the most visually complex type. + +## Separate public and internal representations + +Public APIs should remain understandable and stable: + +```ts +export interface Observation { + readonly target: string; + readonly signalType: string; + readonly value: string; + readonly observedAt: Temporal.Instant; + readonly evidence?: EvidenceReference; +} +``` + +A hot internal batch may use encoded identifiers or columnar arrays: + +```ts +interface ObservationColumns { + readonly targetIds: Uint32Array; + readonly signalTypeIds: Uint16Array; + readonly valueIds: Uint32Array; + readonly timestamps: Float64Array; +} +``` + +Keep conversion at explicit boundaries. Do not expose an internal packed layout +unless consumers need and can support that compatibility contract. + +## Hot and cold data + +Hot data is read or written in the dominant loop. Cold metadata is retained for +explanation, provenance, errors, or final output. + +Instead of attaching large evidence objects to every hot record: + +```ts +interface HotCandidate { + readonly targetId: number; + readonly ruleId: number; + readonly confidence: number; + readonly evidenceId: number; +} + +interface EvidenceStore { + readonly entries: readonly EvidenceReference[]; +} +``` + +This may reduce repeated traversal and retention. It also adds indirection and +conversion cost. Measure both. + +## Representation options + +### Stable objects + +Use ordinary objects for small or irregular data, public boundaries, rich +metadata, and code where clarity dominates. Keep hot object shapes stable: + +- create properties in consistent order; +- avoid repeatedly adding and deleting fields; +- avoid polymorphic property meanings; +- avoid sparse arrays in hot loops; +- avoid mixing unrelated element types in the same hot array. + +### Struct-of-arrays or typed arrays + +Consider columnar or typed-array layouts when: + +- millions of homogeneous records are scanned repeatedly; +- fields have bounded numeric encodings; +- only a subset of fields is used in each kernel; +- allocation and GC dominate; +- serialization or native/Wasm interop benefits; +- benchmarks show a meaningful end-to-end gain. + +Account for: + +- encoding dictionaries and lookup cost; +- growth and resizing; +- null or optional values; +- precision and overflow; +- endianness for serialized formats; +- conversion at public boundaries; +- debug and diagnostic ergonomics; +- worker transfer and ownership. + +### Array-of-structs + +A `readonly Observation[]` can remain the best batch representation when records +are modest, commonly consumed together, and object allocation is not the +bottleneck. Do not cargo-cult a columnar layout. + +### Maps and indexes + +Choose indexes from query patterns. A map keyed by domain may help point lookup +but increase memory and build time. Precompiled detector indexes may improve +matching while increasing startup and invalidation cost. Record who builds, +owns, shares, and disposes the index. + +## Allocation and pooling + +Measure allocation rate and retained memory separately. Pooling can reduce +allocation but introduce stale state, larger retention, contention, and +lifecycle complexity. + +Use pooling only when: + +- profiles show allocation or GC as material; +- reset semantics are complete and tested; +- maximum pool size is bounded; +- borrowed values cannot escape their lifetime; +- the pool improves representative workloads. + +Do not pool public objects that consumers may retain. + +## Batching and locality + +A batch can improve locality and amortize dispatch, serialization, logging, and +I/O. A batch can also increase latency and peak memory. + +Define a batch policy: + +```ts +export interface BatchPolicy { + readonly maxRecords: number; + readonly maxBytes: number; + readonly maxDelayMs: number; +} +``` + +Measure small, typical, and large batches against: + +- throughput; +- first-result latency; +- p95/p99 latency; +- peak and retained memory; +- allocation rate; +- destination write behavior; +- cancellation waste; +- checkpoint frequency. + +## Data lifetime + +Classify values: + +```text +request lifetime +batch lifetime +run lifetime +application lifetime +persistent lifetime +``` + +Long-lived references to short-lived batches create retention. Closures, +listeners, caches, diagnostics, and retry state can keep entire object graphs +alive. Inspect heap retainers when memory does not plateau. + +## Data-oriented API review + +Ask: + +1. What is the dominant data transformation? +2. What are the real input distributions and maxima? +3. Which fields are hot together? +4. Which values need stable public meaning? +5. Where do allocation and copying occur? +6. Which values are retained across batches or runs? +7. Is an index worth its build and memory cost? +8. Does the representation help the full workflow or only a microbenchmark? +9. Can the representation remain private? +10. What semantic oracle proves optimized output is equivalent? + +## Failure signatures + +- data-oriented design is described as replacing objects with typed arrays; +- public APIs expose numeric IDs with no stable semantic layer; +- a packed representation is chosen before measuring cardinality or access; +- optimization improves a hot loop but worsens parse, conversion, startup, or + persistence time; +- object pools retain unbounded data or leak borrowed values; +- indexes are rebuilt for every request; +- cold metadata is copied through every hot transform; +- sparse or polymorphic arrays appear in measured kernels; +- benchmarks omit conversion and output costs paid by real consumers. + +## Verification + +- profile representative end-to-end workloads before changing representation; +- retain semantic output digests or property tests; +- measure allocation, GC, peak RSS, retained heap, CPU, and wall time; +- test small inputs where conversion overhead may dominate; +- test maximum cardinality and skew; +- verify no borrowed or pooled value escapes its lifetime; +- compare public API behavior before and after internal representation changes; +- reject complexity when gains do not clear the predeclared threshold. + +## Sources and freshness + +- Richard Fabian, Data-Oriented Design, transform and hardware-oriented model. +- V8 documentation on object properties, element kinds, allocation, and garbage + collection, reviewed 2026-07-23. +- Library-first guidebook, reviewed 2026-07-23. diff --git a/skills/build-libraries/references/packaging.md b/skills/build-libraries/references/packaging.md new file mode 100644 index 0000000..7eccef3 --- /dev/null +++ b/skills/build-libraries/references/packaging.md @@ -0,0 +1,268 @@ +# ESM packaging, exports, and selective adoption + +Use this reference when publishing a TypeScript library, splitting optional +integrations, designing subpath exports, diagnosing tree-shaking, or validating +the package from a clean consumer. + +## Architecture before metadata + +Tree-shaking begins with module ownership. `"sideEffects": false` cannot repair +an entrypoint that eagerly imports every adapter, constructs a registry, or +performs work at module scope. + +A selectively adoptable graph should look like: + +```text +@scope/library + core use cases and public domain values + +@scope/library/browser.js + browser adapter and browser-only dependencies + +@scope/library/storage.js + storage adapter + +@scope/library/logtape.js + optional diagnostics integration helpers +``` + +Importing core must not traverse browser, database, CLI, workflow, or telemetry +dependencies. + +## Preserve ESM + +Publish static ESM entrypoints where supported. Preserve `import` and `export` +through the library build so bundlers can reason about symbol use. Avoid making +CommonJS the only distributed representation when tree-shaking is a requirement. + +Use explicit file extensions in relative imports and public subpaths according +to the repository's runtime and package policy. + +## Public exports + +Define the public contract with `exports`: + +```json +{ + "name": "@kaiju/analysis", + "type": "module", + "exports": { + ".": { + "types": "./dist/index.d.ts", + "import": "./dist/index.js" + }, + "./browser.js": { + "types": "./dist/browser.d.ts", + "import": "./dist/browser.js" + }, + "./storage.js": { + "types": "./dist/storage.d.ts", + "import": "./dist/storage.js" + } + } +} +``` + +Explicit subpaths encapsulate internals and give consumers stable entrypoints. +Do not export source-directory wildcards merely to avoid deciding the API. +Patterns can be appropriate for large regular surfaces, but audit what they +expose. + +## Named exports and root barrels + +Named exports support selective use: + +```ts +export { analyzeDomains } from "./analyze-domains.js"; +export { evaluateBatches } from "./evaluate-batches.js"; +``` + +A root barrel is acceptable when re-exporting side-effect-free core modules. +It becomes harmful when it imports adapters, registrations, polyfills, generated +catalogues, or dependency-heavy defaults. + +Avoid eager namespace objects: + +```ts +export const library = { + analyzeDomains, + createBrowserCollector, + createDatabaseStore, + registerAllDetectors, +}; +``` + +Such objects encourage unit retention and can require importing every member. + +## Side effects + +Import-time effects include: + +- global registration; +- logging configuration; +- signal installation; +- environment or config discovery; +- resource acquisition; +- polyfill mutation; +- filesystem or network work; +- decorators or static initialization with observable effects. + +Prefer explicit functions: + +```ts +export function registerBuiltInDetectors( + registry: DetectorRegistry, +): void { + // The caller requested mutation. +} +``` + +Declare package metadata truthfully: + +```json +{ + "sideEffects": false +} +``` + +or list exact effectful built files: + +```json +{ + "sideEffects": [ + "./dist/polyfill.js", + "./dist/register.js" + ] +} +``` + +Incorrect `sideEffects` metadata can remove required behavior. Treat it as a +verified contract, not an optimization incantation. + +## Optional dependencies and peer boundaries + +Place optional adapters in separate modules or packages. Avoid top-level imports +that make optional dependencies mandatory at resolution time. + +Choose dependency ownership explicitly: + +- dependency: implementation requires it at runtime; +- peer dependency: host must supply a compatible singleton or framework; +- optional peer: integration is optional and imported separately; +- development dependency: build/test only. + +Test missing optional dependencies. Core imports should continue to work. + +## Conditional exports + +Use conditional exports only for meaningful runtime differences with equivalent +public contracts. Do not create browser, Node, Deno, worker, import, and require +branches without testing every branch. + +Keep runtime-specific code in leaf modules. Avoid conditions that silently +change semantics, precision, error types, or resource ownership. + +## Build output + +Preserve useful module boundaries. A library distributed only as one bundled +file can still tree-shake in some toolchains, but separate side-effect-free ESM +modules and explicit subpaths make ownership and optional dependencies easier to +verify. + +Verify: + +- JavaScript entrypoints; +- declaration entrypoints; +- source maps if promised; +- exports and imports targets; +- package file inclusion; +- license and metadata; +- runtime-specific conditions; +- absence of source-only or internal files from public exports. + +## Tree-shaking evidence + +Source inspection is insufficient. Create clean consumers: + +```text +consumer-core + imports one core function + +consumer-adapter + imports one optional adapter + +consumer-root + imports supported root facade +``` + +Bundle each with the supported bundler or representative bundlers. Inspect: + +- output bytes and metafile/module graph; +- presence of known optional dependency markers; +- import-time behavior; +- retained top-level initializers; +- warnings about CommonJS or side effects; +- semantic output. + +Use a negative marker in optional modules to prove they are absent, not merely a +small bundle-size threshold. + +## Import and startup paths + +Measure import-only and first-call cost for: + +- core entrypoint; +- each optional adapter; +- root convenience entrypoint; +- CLI help/version path if the package includes an executable. + +Lazy dynamic imports can reduce startup but add first-use latency and async +failure modes. Use them when runtime selection is real, not to hide a poorly +partitioned root graph. + +## Compatibility and release + +Adding `exports` to an existing package can break undocumented deep imports. +Inventory real consumers before cutover. Decide whether to: + +- preserve selected legacy subpaths temporarily; +- publish a major release; +- provide codemods or migration notes; +- intentionally remove unsupported internals. + +Do not keep compatibility surfaces indefinitely without an owner and removal +condition. + +## Failure signatures + +- root entrypoint imports every adapter and rule pack; +- module import launches work or configures globals; +- `sideEffects: false` is added without auditing top-level effects; +- source modules tree-shake but the distributed build converts them to CommonJS; +- optional dependencies are reachable from core; +- wildcard exports expose internal mechanisms; +- types and JavaScript exports disagree; +- a package test imports source paths instead of the packed artifact; +- bundle size is reported without proving which modules were retained; +- only one runtime or conditional branch is tested. + +## Verification + +- pack the exact publishable artifact; +- install it into clean Deno, Node, or bundler consumers as supported; +- import every public subpath and reject private paths; +- run type checking against declarations; +- bundle core and adapter consumers with metafile analysis; +- assert unused adapter markers and dependencies are absent; +- test import-time global and resource effects; +- verify missing optional dependencies do not break core; +- compare package contents and exports against an allowlist; +- run the published artifact's examples, not source aliases. + +## Sources and freshness + +- Node.js package `exports`, subpath, and conditional export documentation, + reviewed 2026-07-23. +- esbuild tree-shaking and side-effect documentation, reviewed 2026-07-23. +- unbuild and package tooling evidence in the repository, reviewed 2026-07-23. +- Library-first guidebook, reviewed 2026-07-23. diff --git a/skills/build-libraries/references/recovery-refactoring.md b/skills/build-libraries/references/recovery-refactoring.md new file mode 100644 index 0000000..43ca2db --- /dev/null +++ b/skills/build-libraries/references/recovery-refactoring.md @@ -0,0 +1,247 @@ +# Recovery guarantees and CLI-first refactoring + +Use this reference when a reusable library must survive interruption, resume +work, or be extracted from a CLI or application whose abstractions have become +rigid. + +## Name the recovery guarantee + +### Restartable + +The operation can run again from the beginning without corrupting results. +Requirements include deterministic identity, idempotent or reconciled effects, +and duplicate handling. + +### Checkpoint-resumable + +The operation can continue after a committed boundary. Requirements include a +durable checkpoint owner, compatibility metadata, output receipts, replay rules, +and reconciliation. + +### Durably orchestrated + +Execution state survives process and infrastructure failure through a workflow +or job authority with histories, workers, timers, signals, leases, operator +controls, and versioning. `build-workflows` owns this level. + +Do not describe retries, an iterator cursor, an in-memory stage list, or an +Effect dependency graph as durability. + +## Checkpoint committed work + +A checkpoint should represent outputs that are durable and safe to treat as +complete: + +```ts +export interface AnalysisCheckpoint { + readonly schemaVersion: number; + readonly requestFingerprint: string; + readonly engineVersion: string; + readonly ruleSetVersion: string; + readonly committedCursor: string; + readonly outputReceipts: readonly OutputReceipt[]; + readonly committedAt: Temporal.Instant; +} +``` + +Commit order: + +```text +process batch + -> write required outputs + -> verify receipts and identities + -> atomically advance checkpoint + -> release batch resources +``` + +If the process dies after output write but before checkpoint, replay occurs. +Sinks must tolerate or reconcile duplicate effects. If checkpoint advances +before output durability, work can be skipped permanently. + +## Compatibility and fingerprints + +A resume preflight validates: + +- request identity and normalized inputs; +- library, schema, rule, and adapter versions; +- source snapshot or version; +- output receipts and target state; +- configuration digest and relevant policy; +- checkpoint schema; +- replay safety at the last boundary. + +A fingerprint is an identity aid, not proof of security or semantic +compatibility. Version the canonicalization algorithm. Do not hash secrets into +logs or user-visible manifests. + +## Attempts and logical identity + +Separate logical work from attempts: + +```text +run ID + logical operation + +batch ID + deterministic unit of work + +attempt ID + one execution try + +output identity + deterministic destination key or version +``` + +Retries create new attempts, not new logical outputs. Metrics and diagnostics +should preserve both. + +## Library and workflow boundary + +A library may own: + +- checkpoint schema and compatibility; +- deterministic batch identity; +- replay-safe operation contracts; +- a `resume()` use case; +- storage capability requirements; +- structured recovery events. + +A durable workflow system owns: + +- persisted execution authority; +- workers and task delivery; +- timers and external signals; +- leases, fencing, heartbeats, and retries; +- deployment and version routing; +- operator controls and histories. + +Do not embed a workflow engine into a general library unless durable execution +is the product. Provide an adapter package where appropriate. + +## Refactoring a CLI-shaped library + +### 0. Preserve evidence + +Record current commands, inputs, outputs, errors, artifacts, cancellation, +resource cleanup, and performance. Run the current build and representative +commands before editing. + +### 1. Select one representative use case + +Choose one vertical path that exercises the important architecture. Avoid +rewriting every command simultaneously. + +### 2. Write desired programmatic call sites + +Create tests or examples for a common consumer and an advanced consumer. Do not +copy CLI types into the library API. + +### 3. Define request, result, failures, and events + +Make these the first stable contracts. Keep stages and mutable context private. + +### 4. Classify the existing context + +For every context field classify it as: + +- domain input; +- execution option; +- required capability; +- diagnostic metadata; +- application concern; +- hidden implementation detail. + +Remove unrelated fields rather than passing the whole context onward. + +### 5. Move orchestration behind a named use case + +Existing stages may remain internally during migration. Demote them from public +integration contract to private mechanism or observable event vocabulary. + +### 6. Establish resource scopes + +Replace hidden globals and manual release conventions with explicit handles and +scoped disposal. Determine which resources are application, run, batch, and item +lifetime. + +### 7. Replace accidental materialization + +Trace arrays, `Promise.all`, queues, and buffers. Introduce async iteration, +streams, and batches only where the workload requires them. + +### 8. Isolate hot kernels + +Separate pure or bounded transforms from I/O and lifecycle. Introduce internal +data-oriented representations only after profiling. + +### 9. Restore ecosystem owners + +Keep c12/defu and CLI source precedence in the application; LogTape sink +configuration in the application; storage guarantees in explicit adapters; +workflow authority in the workflow layer. + +### 10. Add a second consumer + +Use a service endpoint, worker, direct test, notebook, or another CLI command. +The second consumer exposes residual application coupling. + +### 11. Prove selective adoption + +Pack the library, import only the use case, and verify unrelated adapters and +dependencies are absent. + +### 12. Remove compatibility scaffolding + +Delete old run helpers, public stage types, duplicate configuration paths, and +legacy exports when the authorized migration does not require them. Search +code, docs, tests, examples, and package metadata. + +## Migration strategy + +Prefer an incremental vertical extraction when behavior is valuable and the +current application must remain runnable. Prefer a clean cutover when public +compatibility is explicitly rejected and the old architecture would force +permanent duplication. + +Do not maintain two orchestration paths without: + +- a named owner; +- parity tests; +- a removal condition; +- a deadline or release boundary; +- explicit consumer inventory. + +## Failure signatures + +- checkpoint stores the last attempted item rather than committed outputs; +- resume ignores version or source compatibility; +- retries create duplicate external effects with no reconciliation; +- a library claims durability because it has stages or serialized JSON; +- extraction begins by moving folders instead of writing consumer calls; +- the new core still accepts CLI config, logger configuration, and terminal + services; +- old `run...()` helpers remain the only supported entrypoints; +- no second consumer is built; +- compatibility wrappers become permanent undocumented architecture. + +## Verification + +- kill after output write but before checkpoint and verify safe replay; +- kill after checkpoint commit and verify no completed work repeats; +- change request, schema, source, and rule versions and verify resume rejection + or named migration; +- duplicate delivery and verify idempotency or reconciliation; +- compare CLI behavior before and after extraction; +- call the new library from a second consumer; +- search and remove legacy exports and docs; +- pack and consume the final artifact; +- route durable engine integration through `build-workflows` verification. + +## Sources and freshness + +- Library-first guidebook, reviewed 2026-07-23. +- Existing build-workflows durability, checkpoint, and pipeline references. +- Temporal TypeScript documentation for durable orchestration boundaries, + reviewed 2026-07-23. +- unstorage and ohash source evidence for adapter and fingerprint limitations, + reviewed 2026-07-23. diff --git a/skills/build-libraries/references/resources-performance.md b/skills/build-libraries/references/resources-performance.md new file mode 100644 index 0000000..0c763d0 --- /dev/null +++ b/skills/build-libraries/references/resources-performance.md @@ -0,0 +1,283 @@ +# Resource ownership and performance contracts + +Use this reference when a library acquires files, streams, browsers, database +sessions, workers, locks, temporary directories, timers, watchers, or other +resources, or when latency, CPU, memory, startup, throughput, cleanup, or +resource use affects the design. + +## Resources are values with lifetimes + +For every resource answer: + +1. Who acquires it? +2. Who owns it? +3. Is it borrowed or transferred? +4. What is its lifetime? +5. Is cleanup synchronous or asynchronous? +6. What happens after partial construction failure? +7. What happens on abort, early iterator return, and consumer error? +8. What happens when cleanup fails? + +A dependency is not necessarily a resource. A repository interface may be +long-lived and externally owned. A transaction, leased browser context, reader, +watcher, or temporary workspace has a specific cleanup obligation. + +## Explicit resource management + +Prefer disposable handles: + +```ts +export interface BrowserLease extends AsyncDisposable { + readonly browser: Browser; + readonly leaseId: string; +} +``` + +Where the target runtime supports explicit resource management: + +```ts +await using lease = await browserPool.acquire({ signal }); +const result = await inspectDomains(domains, lease.browser, { signal }); +``` + +For synchronous resources use `using` and `Disposable`. For several fallible +acquisitions use `DisposableStack` or `AsyncDisposableStack`, then transfer the +stack with `move()` only when ownership is transferred deliberately. + +Cross-runtime libraries may need a compatibility layer or `try/finally` until +all supported runtimes parse and implement the syntax. Preserve the same public +ownership contract in the fallback. + +## Narrow and nested lifetimes + +Do not put every resource into one application-wide runtime: + +```text +application lifetime + database pool + browser pool + shared rule index + +analysis-run lifetime + run record + temporary workspace + checkpoint session + +batch lifetime + browser context or page + stream reader + write transaction +``` + +Dispose in reverse dependency order. Do not retain a page for an entire run when +it can be scoped to one target or batch. Do not repeatedly construct an +application resource for every item. + +## Borrowing and transfer + +Make destination ownership explicit. A writer that receives a +`WritableStream` must document whether it closes, aborts, or merely +releases its writer lock. + +Prefer distinct APIs or an explicit option: + +```ts +export type Ownership = "borrow" | "transfer"; + +export function createArchiveWriter( + destination: WritableStream, + options: { readonly ownership: Ownership }, +): ArchiveWriter; +``` + +Borrowed resources remain the caller's cleanup responsibility. Transferred +resources become the callee's responsibility after successful transfer. + +## Cancellation and disposal + +`AbortSignal` asks work to stop. Disposal releases owned resources. Use both. + +A cancellation path should: + +```text +abort signal observed + -> stop admitting work + -> cancel or drain bounded in-flight work + -> close iterator/stream boundaries + -> settle or abort writes according to contract + -> dispose owned resources + -> flush bounded diagnostics + -> return stable cancellation failure +``` + +Do not equate calling `abort()` with completed cleanup. Measure cleanup latency +and assert no handles remain. + +## Workload-specific performance + +Define the workload before claiming performance: + +- command help or import-only cold path; +- one short request; +- repeated warm requests; +- high-volume batch pipeline; +- long-lived server or worker; +- cancellation and shutdown; +- restart or resume. + +Track relevant dimensions: + +| Dimension | Measure | +| --- | --- | +| Cold start | process or import to first useful result | +| Warm latency | repeated operation distribution | +| Throughput | records, targets, requests, or bytes per second | +| Tail latency | p95 and p99 under defined load | +| CPU | process CPU time and utilization | +| Allocation | bytes or objects allocated per operation | +| Peak memory | RSS, heap, external, and buffer peaks | +| Retained memory | live state after controlled lifecycle cycles | +| I/O | operations and bytes by destination | +| Resource count | files, sockets, pages, workers, timers, readers | +| Cleanup | time and residual resources after normal/error/abort | +| Recovery | restart or resume time and duplicate work | +| Package cost | import time, bundle bytes, dependency graph | + +## Model budgets before tuning + +For a browser pipeline: + +```text +peak memory = fixed process state + + browser processes + + active targets x per-target working set + + queued inputs + + completed output buffers + + persistence buffers + + diagnostics and retry state +``` + +Define explicit policy: + +```ts +export interface ExecutionBudget { + readonly concurrency: number; + readonly maxQueued: number; + readonly maxBufferedBatches: number; + readonly maxBatchBytes: number; + readonly maxOpenResources: number; + readonly cleanupTimeoutMs: number; +} +``` + +Admission, concurrency, and buffering are separate. Throughput achieved while a +queue grows without bound is not sustainable throughput. + +## Resource reuse + +Reuse can reduce startup and acquisition cost while increasing memory, stale +state, contention, and recovery complexity. + +For a pool define: + +```ts +export interface PoolPolicy { + readonly maximum: number; + readonly maximumIdle: number; + readonly idleTimeoutMs: number; + readonly maximumUses: number; + readonly acquireTimeoutMs: number; +} +``` + +Test reuse, eviction, invalidation, partial failure, cancellation while waiting, +and disposal of idle and leased resources. + +## Diagnostics on hot paths + +Use structured, lazy, aggregated diagnostics. Avoid per-record info logs. + +```ts +logger.debug("Processed observation batch", { + records: batch.values.length, + encodedBytes: batch.encodedBytes, + durationMs, +}); +``` + +Measure disabled logging, filter dispatch, structured record creation, lazy +property evaluation, redaction, formatting, sink buffering, and flush. Do not +optimize by disabling required security or diagnostics without a product policy. + +## Benchmark protocol + +Before running: + +1. state question and mechanism; +2. name target and protected workflows; +3. define semantic oracle; +4. set practical threshold and regression guardrails; +5. record baseline source, harness, fixtures, runtime, and environment; +6. choose cold/warm, process isolation, warmup, order, repetitions, and seeds; +7. define treatment of failures, timeouts, and OOM; +8. preserve raw samples. + +Use representative end-to-end stories as decision gates. Microbenchmarks may +explain mechanism but cannot override a protected workflow regression. + +For async work, hold arrival rate, concurrency, queue capacity, and destination +behavior constant. Report latency distributions under fixed load or sustainable +throughput under a latency and error objective. + +## Memory and leak tests + +Distinguish allocation, peak, and retained memory. Run repeated lifecycle cycles +and require a plateau. Exercise: + +- normal completion; +- source and destination errors; +- early iterator return; +- cancellation; +- failed partial construction; +- pool eviction; +- watcher or listener removal; +- worker termination; +- cache invalidation; +- explicit disposal. + +A before/after heap subtraction without lifecycle repetition is not a leak test. + +## Failure signatures + +- a resource-producing factory returns a handle with no cleanup contract; +- one universal runtime owns resources with unrelated lifetimes; +- cancellation returns before owned resources settle; +- cleanup exceptions hide the primary failure or disappear silently; +- a pool has no maximum, idle policy, or invalidation; +- a throughput test allows queue growth; +- only mean latency is reported; +- memory claims use one ambiguous metric; +- microbenchmark results justify an API redesign without end-to-end evidence; +- debug property construction occurs even when no sink accepts the record; +- performance improvements change semantics or error handling. + +## Verification + +- use runtime resource and operation sanitizers where available; +- instrument acquisition, active count, queue depth, and disposal count; +- force partial-construction failures at every acquisition boundary; +- cancel while waiting, active, writing, and cleaning up; +- run repeated lifecycle cycles and inspect plateau behavior; +- benchmark cold and warm paths separately; +- preserve raw samples and correctness digests; +- compare absolute, relative, variability, and guardrail results; +- test under realistic CPU, memory, network, and destination constraints. + +## Sources and freshness + +- TypeScript explicit resource management documentation, reviewed 2026-07-23. +- ECMAScript Explicit Resource Management proposal and current runtime support, + reviewed 2026-07-23. +- V8 memory and garbage collection documentation, reviewed 2026-07-23. +- Deno benchmark and resource-sanitizer guidance, reviewed 2026-07-23. +- Library-first guidebook, reviewed 2026-07-23. diff --git a/skills/build-libraries/references/verification.md b/skills/build-libraries/references/verification.md new file mode 100644 index 0000000..986ff2d --- /dev/null +++ b/skills/build-libraries/references/verification.md @@ -0,0 +1,229 @@ +# Library verification matrix + +Use this reference when planning tests, reviewing completion, validating a +published package, or evaluating claims about composition, streaming, +performance, cleanup, tree-shaking, or recovery. + +## Evidence hierarchy + +Different claims require different evidence: + +| Claim | Minimum useful evidence | +| --- | --- | +| Type relationship | type test or clean consumer type check | +| Runtime behavior | executable public-entrypoint test | +| Import safety | fresh-process import observation | +| Tree-shaking | built clean consumer plus module graph or marker absence | +| Streaming | first-item, peak-memory, and slow-sink behavior | +| Resource safety | normal/error/abort/early-return lifecycle checks | +| Performance | representative benchmark against recorded baseline | +| Resume | kill/restart at controlled commit windows | +| Compatibility | previous consumer and artifact contract suite | +| Publication | packed artifact installed in clean consumers | + +Source review can diagnose a design. It cannot prove built-artifact behavior or +recovery after process death. + +## API and composition tests + +Test: + +- common facade call; +- each independently supported lower-level capability; +- a second application adapter; +- focused capability fakes; +- structured failures and cancellation; +- domain event ordering and identity; +- absence of application globals; +- replacement of one adapter without unrelated changes; +- unsupported composition rejected clearly. + +Use compile-time assertions for inference, readonly contracts, discriminated +unions, and public subpaths. Use runtime tests for semantics. + +## Import safety tests + +Run each public entrypoint in a fresh process. Observe: + +- stdout and stderr; +- environment and global mutations; +- files and network; +- timers, workers, sockets, and open handles; +- configuration discovery; +- optional dependency resolution; +- import duration and memory where important. + +A library entrypoint should normally expose values only. Effectful registration +entrypoints must be named and declared in side-effect metadata. + +## Packaging and tree-shaking tests + +1. Build or pack the exact publishable artifact. +2. Install it in an empty consumer. +3. Import every public subpath. +4. reject private subpaths; +5. type-check declarations; +6. bundle one-function, one-adapter, and root-facade consumers; +7. inspect metafiles or module graphs; +8. assert known unused adapter markers and dependency names are absent; +9. execute each bundle and compare semantic output; +10. inspect package contents against an allowlist. + +Run supported runtime and condition branches separately. + +## Data-flow tests + +For incremental APIs test: + +- first value before complete source exhaustion; +- empty, one-item, typical, and large inputs; +- slow source and slow destination; +- bounded queue and buffered bytes; +- skewed task durations and ordering; +- early break; +- consumer throw; +- source throw; +- abort at admission, active work, and write; +- deliberate materialization limit; +- batch count and byte boundaries; +- no duplicate or lost records. + +A test that only uses ten records cannot establish bounded behavior. + +## Resource tests + +Instrument acquisition and disposal. Assert: + +- every successful acquisition is released exactly once; +- partial construction releases prior resources; +- borrowed resources remain open; +- transferred resources close according to contract; +- disposal order respects dependencies; +- cleanup occurs after iterator return and stream cancellation; +- pool maximum and idle eviction hold; +- repeated cycles plateau; +- cleanup failures retain the primary failure and remain observable. + +Use Deno resource and operation sanitizers where available. Use platform handle +inspection or fresh-process exit behavior elsewhere. + +## Performance tests + +Keep correctness as a gate. Define: + +- baseline and candidate revision; +- representative workload and distribution; +- target and protected stories; +- primary metric and practical threshold; +- tail, memory, resource, and error guardrails; +- environment, runtime, flags, warmup, order, repetitions, and seed; +- raw artifact location; +- decision rule. + +Measure cold import and first call separately from warm throughput. Measure peak +and retained memory separately. For concurrent work, bound arrival, concurrency, +and queue capacity. + +Do not claim a win from one fastest run, a microbenchmark alone, or a benchmark +that omits conversion, output, cleanup, or error behavior paid by consumers. + +## Recovery tests + +For restartable operations: + +- run twice; +- duplicate inputs; +- partially existing outputs; +- stale attempts; +- reconciliation. + +For checkpoint resume: + +- crash before output; +- crash after output and before checkpoint; +- crash after checkpoint; +- corrupt or missing receipt; +- incompatible request, schema, source, engine, and policy; +- cancellation and manual retry; +- final reconciliation. + +For durable orchestration compose with `build-workflows` and test worker loss, +lease expiry, duplicate task delivery, timer/signal behavior, deployment +versioning, and operator controls. + +## Compatibility tests + +Inventory real consumers and frozen examples. Test: + +- public import paths; +- type-level source compatibility where promised; +- runtime results, errors, ordering, and event shapes; +- serialized formats and checkpoint schemas; +- deep imports scheduled for removal; +- deprecation messages and migration path; +- old artifacts against new consumers where supported; +- new artifacts against old consumers where supported. + +Do not keep accidental behavior merely because it is observable. Decide whether +to support, deprecate, or break it with an explicit release policy. + +## Cross-skill verification + +### With build-clis + +Prove the CLI calls the public library API, resolves configuration once, owns +LogTape configuration and process lifecycle, and preserves stdout, stderr, exit, +help, completion, and cancellation contracts. + +### With explore-ecosystems + +Prove selected companions and adapters are first-party or intentionally +interoperable, version-compatible, and smaller than alternative package sets. + +### With build-devtools + +Prove build tasks, generated exports, package contents, release provenance, and +clean-consumer commands. + +### With build-workflows + +Prove the line between library resume contracts and durable execution authority. + +### With build-data + +Prove persistence guarantees, query behavior, migrations, and projection +rebuilds rather than assuming them from an interface. + +## Completion report + +Report: + +- public surfaces added, changed, and removed; +- consumer stories exercised; +- package entrypoints and optional integrations; +- data-flow and resource bounds; +- benchmark and memory evidence; +- recovery guarantee and crash windows tested; +- compatibility decisions; +- checks passed, failed, or blocked; +- claims intentionally not made. + +## Failure signatures + +- tests import source aliases instead of the package artifact; +- bundle size is the only tree-shaking oracle; +- streaming tests never slow the destination or break early; +- resource tests cover only normal completion; +- performance output lacks baseline, variability, or semantic oracle; +- resumability is tested by calling the same function twice in one process; +- type tests pass while runtime exports are missing; +- a clean consumer is never created; +- blocked checks are described as passed. + +## Sources and freshness + +- Repository evaluation model and executable fixture conventions. +- Library-first guidebook verification matrix, reviewed 2026-07-23. +- Deno testing and benchmarking guidance, reviewed 2026-07-23. +- esbuild analysis and package-consumer verification guidance, reviewed + 2026-07-23. diff --git a/skills/build-sites/SKILL.md b/skills/build-sites/SKILL.md index 2d56f8e..5cfacfb 100644 --- a/skills/build-sites/SKILL.md +++ b/skills/build-sites/SKILL.md @@ -37,6 +37,10 @@ discoverability, feeds, and site deployment. drafts, media, cache, feeds, and migration. - [site-quality.md](references/site-quality.md): SEO, accessibility, performance, assets, and verification. +- [icons.md](references/icons.md): load for Astro Icon, Unplugin Icons, + renderer-specific compilers, local SVG collections, accessibility, and bundle control. +- [fonts.md](references/fonts.md): load for Astro Fonts API, Fontsource, + local/variable fonts, privacy, preload, fallback metrics, and layout-shift verification. - [casebook.md](references/casebook.md): Kaiju and ThunderStrike patterns and counterexamples. diff --git a/skills/build-sites/references/astro.md b/skills/build-sites/references/astro.md index 8de9fac..762e7fb 100644 --- a/skills/build-sites/references/astro.md +++ b/skills/build-sites/references/astro.md @@ -1,45 +1,272 @@ -# Astro sites +# Astro site architecture and runtime manual -## Rendering decision +Use this reference for Astro routes, layouts, content, endpoints, middleware, adapters, islands, view transitions, images, fonts, and generated documents. Verify the installed Astro major and integration versions before applying configuration: several uploaded projects use Astro 6 or 7 and experimental options. -- Use static output when routes can be produced at build time. -- Use server output when request-time identity, live data, or uncached dynamic - behavior requires it. -- Use per-route prerendering in a server project where a route remains static. -- Add a deployment adapter only when the selected output and host require it. +## Contents -## Page ownership +- Repository evidence inventory +- Output and route rendering +- Page, layout, and endpoint ownership +- Client islands and server islands +- Middleware, locals, and cache +- Navigation and view transitions +- Static documentation and OpenAPI +- Assets and integrations +- Configuration review +- Failure signatures +- Verification +- Sources and freshness -Astro owns routes, layouts, metadata, content, and static structure. Prefer -native disclosure, dialog, form, image, and navigation behavior. Use an inline -script or custom element for narrow DOM behavior; use a framework island for -reactive ownership that cannot remain local. +## Repository evidence inventory -Choose `client:load`, `client:idle`, `client:visible`, media conditions, or -`client:only` from urgency and rendering constraints. An above-the-fold pointer -interaction and a below-fold visualization should not automatically share the -same directive. +Inspect: -## Navigation lifecycle +```sh +rg --files | rg '(astro\.config|src/pages|src/layouts|src/middleware|src/content\.config|src/live\.config|env\.d\.ts)' +rg -n 'output:|adapter:|prerender|client:|server:defer|defineMiddleware|Astro\.locals' . +rg -n 'integrations:|fonts:|image:|security:|session:|env:' astro.config.* +``` -View transitions and `astro:page-load` can execute initialization repeatedly. -Make handlers idempotent, delegate where possible, or remove prior listeners. -Respect hash navigation, focus, browser history, and reduced motion. Do not -globally force smooth scrolling without preference and accessibility handling. +Record: -## Renderer alignment +- Astro and integration versions; +- package manager and build/check/preview scripts; +- site origin and base path; +- static/server output default; +- per-route prerender exceptions; +- adapter and target runtime; +- framework integrations and island directives; +- middleware sequence and `locals` types; +- content source and preview/live mode; +- endpoint methods and cache/security headers; +- image, icon, font, Markdown, sitemap, and RSS integrations; +- experimental configuration and its verification source. -Verify each island's framework integration, JSX settings, icon compiler, client -directive, and test environment. A bare `client:only` may require an explicit -renderer hint because Astro skips server rendering. +Do not copy a whole `astro.config.ts` between repositories. An integration can be valid only because of a particular Astro major, adapter, renderer, Vite plugin, environment, or deployment host. -For example, a React component that must never server-render uses -`client:only="react"`, and the matching Astro React integration must be installed -and configured. Use the actual renderer name and verify it with Astro check and -the production build. +## Output and route rendering -## Build proof +Astro defaults to static generation. Choose per route: -Run Astro check and the actual production build. Verify generated routes, -prerendered pages, adapter output, assets, redirects, headers, and one deployed or -adapter-equivalent request flow. +| Route need | Default | Required evidence | +|---|---|---| +| Public content known at build | Static/prerender | Content/build source available in CI | +| Request identity/cookie | On-demand SSR | Adapter and cache policy | +| Fresh uncached request data | On-demand SSR | Runtime API and failure policy | +| Mostly static page with personalized fragment | Static/cached shell plus `server:defer` | Adapter, fallback, deferred request | +| Server-default project with static page | `export const prerender = true` | No request-only dependency | +| Static-default project with dynamic page | `export const prerender = false` | Adapter installed | + +`output: "server"` changes the default; it does not add a capability beyond route defaults. Start static unless most routes are genuinely request-time. In server mode, preserve static public routes explicitly: + +```astro +--- +export const prerender = true; +--- + + +``` + +The Kaiju marketing project uses server output but prerenders its homepage. The Kaiju docs app uses static output and generates API reference routes. The finance auth pages use `prerender = false` because server capability/session data is request-time. + +Choose the adapter from the deployment runtime and features. Verify environment access, streaming, image service, sessions, server islands, headers, filesystem assumptions, and startup behavior. An auto-adapter reduces configuration only if its detection result and production behavior are tested. + +## Page, layout, and endpoint ownership + +Astro owns document structure, routes, layouts, metadata, static content, and server response boundaries. Keep layout contracts explicit: + +```astro +--- +interface Props { + title: string; + description?: string; + canonical?: URL; +} +const { title, description, canonical } = Astro.props; +--- + + + + + + + + +
+ + +``` + +Endpoints own method, content type, cache, validation, and public errors: + +```ts +import type { APIRoute } from "astro"; + +export const GET: APIRoute = async () => { + const catalog = await buildServiceCatalog(); + return Response.json(catalog, { + headers: { "cache-control": "public, max-age=300" }, + }); +}; +``` + +Use a stable service/content factory shared with runtime code rather than reconstructing schemas inside pages. Avoid endpoint module side effects such as loading dotenv, connecting to databases, or logging environment values during import. + +## Client islands and server islands + +Framework components without `client:*` render HTML but no client JavaScript. Select hydration by user-visible urgency: + +- `client:load`: immediately interactive above-fold UI; +- `client:idle`: lower priority, with optional timeout where supported; +- `client:visible`: below-fold/expensive; root margin can start before visibility; +- `client:media`: behavior exists only for a media query; +- `client:only="react"` or `"solid-js"`: no server HTML; explicit renderer and fallback required. + +```astro + + + +

Loading editor…

+ +``` + +`client:visible` does not suspend a running island after it leaves the viewport. Continuous canvas/animation work needs its own visibility policy. + +Use native HTML or a page script instead of an island when practical. The Kaiju homepage's active FAQ is `
`; its unused Solid accordion is not a reason to hydrate. + +`server:defer` creates a server island. It needs an adapter, a layout-stable fallback, and a cache/auth design. It is useful for personalized fragments inside otherwise cacheable pages. Do not put critical SEO or primary content behind a deferred request without a deliberate product decision. + +## Middleware, locals, and cache + +Middleware should derive request-scoped facts once and place typed, minimal values in `Astro.locals`: + +```text +request + -> route intent + -> session + -> active organization/authorization + -> shell/page model + -> cache and security headers + -> route +``` + +The finance upload classifies `public`, `auth`, `auth-required`, and `app` routes. That can prevent duplicated pathname logic, but the classification is security-sensitive. Test every prefix, catch-all, trailing slash, API route, static asset, and new route. + +Never let public be the accidental fallback for an unrecognized authenticated route. Prefer route metadata or a test that inventories every route when the framework exposes it. + +Cache policy follows response authority: + +- personalized/app/auth: normally `private, no-store` or explicit private caching; +- auth forms: avoid stale tokens/session-dependent output; +- static fingerprinted assets: long immutable cache; +- public documents/JSON: public validators/max-age with invalidation policy; +- CMS pages: provider cache hints mapped through one boundary. + +Verify final deployed headers, since the adapter/host may change them. + +## Navigation and view transitions + +`` enables Astro client-side navigation and route announcement. It respects reduced motion for transition animations. Every page still needs a useful `` because the announcer prefers it. + +Initialization attached to `astro:page-load` can run repeatedly. Use delegation or replace previous listeners with an `AbortController`. The Kaiju BaseHead script adds handlers to every hash/external link on every page load without cleanup; teach agents to diagnose that duplication rather than reproduce it. + +Do not globally intercept all hash links unless you preserve: + +- normal navigation and invalid target behavior; +- keyboard focus at the destination; +- browser history/back behavior; +- reduced motion; +- URL encoding and selector safety. + +`transition:persist` changes lifetime: persisted elements/islands remain instead of being replaced. By default island state persists while new props may render; `transition:persist-props` keeps old props too. Test auth/user changes, subscriptions, media, and cleanup. CSS animations may restart and iframes may reload despite persistence. + +Forms participate in ClientRouter navigation. If a form requires full browser behavior use `data-astro-reload`. For POST encoding, choose and test the intended enctype rather than assuming traditional URL encoding. + +## Static documentation and OpenAPI + +The Kaiju docs app demonstrates a valuable architecture: + +```text +service-owned endpoint/schema factory + -> build-time service catalog + -> getStaticPaths service pages + -> per-service OpenAPI JSON + -> Scalar reference wrapper +``` + +Static output is compatible with rich API docs when service factories are deterministic build inputs. Validate that importing them does not require runtime secrets or connect to production. + +For authenticated “try it” requests, docs origin, API/service origin, Better Auth cookies, credentialed CORS, and auth method must agree. A cookie header is not a bearer token. Do not tell users to paste a raw cookie into an authorization field unless the API explicitly accepts it. + +Generate and diff OpenAPI artifacts in CI. Test duplicate operation ids, broken refs, schema drift, server origins, public/private endpoints, and example redaction. + +## Assets and integrations + +Keep each integration scoped: + +- images: correct service for build/runtime, dimensions, responsive sources, and adapter support; +- icons: Astro Icon for Astro templates, compiler-specific Unplugin Icons for islands; +- fonts: Astro Fonts API or Fontsource strategy with finite variants/preloads; +- sitemap/RSS: public route/content model and absolute site origin; +- Markdown/MDX: plugin order, raw HTML policy, code themes, and custom directive security; +- framework integrations: only installed for reachable islands; +- Vite plugins: compiler target and plugin type identity must match actual Vite runtime. + +Avoid wildcard icon collections and preloading every font. The uploaded configs include both patterns; they require bundle/waterfall review. + +Experimental options such as content intellisense, client prerender, SVG optimizers, or devtools workspace must be verified against the installed Astro version and production target. Do not use an uploaded config as an API reference. + +## Configuration review + +Review this ownership table: + +| Config area | Question | +|---|---| +| `site` | Correct canonical production origin? | +| `output`/route exports | Minimal dynamic surface? | +| `adapter` | Matches deployed runtime and required features? | +| `integrations` | Every integration has reachable ownership? | +| `image` | Build/runtime service works on host? | +| `fonts` | Only required families/variants/preloads? | +| `env` | Client/server access and validation correct? | +| `server.headers`/middleware | Final policy complete and not contradictory? | +| `vite.plugins` | Correct renderer compiler and no duplicate runtime types? | +| `security` | CSP decision explicit, not disabled for convenience? | +| `session` | Driver durability and deployment topology appropriate? | +| `experimental` | Installed-version proof and fallback? | + +## Failure signatures + +| Signature | Likely cause | Inspect next | +|---|---|---| +| `client:only` cannot render | Missing renderer hint/integration | Direct import and exact hint | +| Static build connects to DB | Import side effect/request-only factory | Build import graph | +| Auth page cached across users | Route intent/header policy wrong | Middleware and deployed headers | +| Page-load action fires repeatedly | Navigation listener duplication | ClientRouter lifecycle cleanup | +| Server island fails only in production | Adapter binding/runtime mismatch | Target deployment output | +| OpenAPI docs build needs secrets | Service factory not boundary-safe | Factory imports and env access | +| Sitemap/canonical uses localhost | `site`/environment contract wrong | Built artifacts | +| Island ships but never interactive | Missing/wrong client directive | Server HTML, chunk and console | +| CSP disabled to make plugins work | Resource/nonces not inventoried | CSP reports and integration origins | + +## Verification + +1. Run Astro check and production build through repository scripts. +2. Inspect generated route/prerender/adapter output. +3. Fetch raw HTML and headers for static, SSR, auth, API, and deferred routes. +4. Hydrate each renderer with mismatch diagnostics captured. +5. Navigate repeatedly with ClientRouter, hash links, back/forward, and focus checks. +6. Test no JavaScript for static/progressively enhanced routes. +7. Test adapter-equivalent startup and one request path, not only `astro dev`. +8. Diff generated OpenAPI/feed/sitemap artifacts. +9. Inspect island chunks, icon/font/image output, and critical-resource waterfall. +10. Test missing env/binding, CMS/API failure, expired session, and static asset failure. + +## Sources and freshness + +- Astro on-demand rendering: https://docs.astro.build/en/guides/on-demand-rendering/ (reviewed 2026-07-17). +- Astro directives: https://docs.astro.build/en/reference/directives-reference/ (reviewed 2026-07-17). +- Astro server islands: https://docs.astro.build/en/guides/server-islands/ (reviewed 2026-07-17). +- Astro view transitions: https://docs.astro.build/en/guides/view-transitions/ (reviewed 2026-07-17). +- Uploaded evidence reviewed 2026-07-17: `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `thunderstrike-blog(4).zip`. +- Astro 6/7, auto-adapters, Emdash, and experimental config are version-sensitive. Inspect installed source/docs before implementing. diff --git a/skills/build-sites/references/casebook.md b/skills/build-sites/references/casebook.md index 30abb92..525ed5d 100644 --- a/skills/build-sites/references/casebook.md +++ b/skills/build-sites/references/casebook.md @@ -1,35 +1,235 @@ -# Site casebook +# Site evidence casebook + +Use these cases as reasoning patterns, not templates. Each case separates observed active architecture, useful pattern, counterexample, unresolved claim, and verification. + +## Contents + +- How to use the casebook +- Kaiju marketing site +- Kaiju documentation site +- Finance hybrid site/application +- ThunderStrike runtime CMS site +- Solid motion experiment +- Cross-case decisions +- Sources and freshness + +## How to use the casebook + +For a similar repository: + +1. Prove the same entrypoints and versions exist. +2. Classify observed source as positive, counterexample, experimental, or unresolved. +3. Extract the ownership principle, not package/file names alone. +4. Recompute the decision for current routes and deployment. +5. Run the case's verification and failure checks. ## Kaiju marketing site -The evidence supports Astro page ownership with narrow Solid islands, native -`details` for FAQ behavior, separate Astro and Solid icon integrations, -Zaidan/Kobalte/Corvu component contracts, Tailwind layers, and visual fallback -for a WebGL hero. Unused Solid FAQ and pricing components are not active patterns; -the import graph decides. +### Observed active architecture + +- Astro page imports marketing sections and prerenders the homepage even though project output defaults to server. +- Astro owns title/description, layout, content, headings, anchors, and sections. +- FAQ uses native `<details>/<summary>` and Astro Icon; a separate Solid accordion exists but is not imported. +- Hero hydrates a narrow Solid `DepthScene` with `client:visible`. +- DepthScene owns pointer state, reduced-motion query, animation frame, particles, WebGL runtime, and static image fallback. +- Solid cleanup cancels frames, removes listeners, removes media query listener, destroys particles, and destroys WebGL. +- Astro config includes Solid, Astro Icon, Unplugin Icons, fonts, Markdown, sitemap, server adapter, and experimental options. + +### Useful decisions + +```text +Static document and CTA + -> Astro HTML + +Disclosure + -> native details/summary + CSS + +Decorative depth effect + -> one Solid island + -> static image fallback + -> decorative aria-hidden canvas +``` + +The island boundary corresponds to resource ownership, not visual region size. + +### Counterexamples/review targets + +- `BaseHead` attaches link listeners on every `astro:page-load` without cleanup or delegation. +- It prevents default behavior for all hash anchors, which requires focus/history/reduced-motion care. +- Astro Icon includes wildcard collections, which needs bundle verification. +- Multiple fonts are preloaded, which needs waterfall and critical-face review. +- CSP is disabled in the shown configuration, which is not a production security recommendation. +- The renderer compiler and package versions must be verified; do not copy config by shape. + +### Verification + +- inspect built homepage for only the intended island; +- disable JS and confirm content/FAQ/CTA/static hero remain; +- force WebGL and texture failure; +- toggle reduced motion and navigate repeatedly; +- assert listener/frame/WebGL cleanup; +- inspect icon/font output and CSP/resource needs. ## Kaiju documentation site -A static Astro application can generate service catalogs, OpenAPI JSON, and API -reference pages from service-owned factories at build time. Rich API documents -do not automatically require a runtime server. +### Observed active architecture + +- Separate Astro app uses static output. +- `getStaticPaths()` enumerates service catalog entries. +- per-service pages render a Scalar reference wrapper. +- JSON routes emit service catalogs and service OpenAPI documents. +- service schema/factory packages are build inputs. +- docs explain credentialed cookie testing for an authenticated billing API. + +### Useful decisions + +```text +service endpoint/schema factories + -> docs service catalog + -> static service pages + -> static OpenAPI JSON + -> runtime services +``` + +The static docs build proves that API reference richness does not imply runtime SSR. It also creates a drift risk: the build and services must consume the same factories and compatible versions. + +For browser “try it,” the docs origin, service origin, cookie scope, trusted origins, and credentialed CORS form one contract. A copied cookie header is not a bearer token. + +### Verification + +- build without production database/provider secrets; +- diff generated OpenAPI artifacts; +- validate refs, operation ids, examples, and service origins; +- crawl every generated service page/JSON path; +- run allowed and rejected credentialed CORS from the docs origin; +- confirm examples/logs contain no credentials. + +## Finance hybrid site/application + +### Observed active architecture + +- Astro server output and Node-targeted auto adapter. +- React integration for form and application islands. +- auth pages are on-demand and server-resolve public auth capabilities. +- TanStack Form owns local field state/validation timing; Better Auth owns auth requests/session. +- middleware derives route intent and cache/security headers. +- Astro layouts own document and shell composition; slots provide route-specific regions. +- URL helper functions preserve/removes query values for server-rendered finance views. +- richer data-grid examples use TanStack Table plus TanStack Virtual. + +### Useful decisions + +```text +Astro server/middleware + -> route intent, session, cache, document layout + -> public auth capability view model + -> React form island + -> TanStack Form local draft/errors + -> Better Auth client request +``` + +The client sees enabled provider ids, not server environment configuration. Field labels, autocomplete, touched errors, pending actions, and method-specific pending state are explicit. + +### Counterexamples/review targets + +- route-prefix classification can make security mistakes when new routes are added; +- `validateSecrets: false` and optional secrets need startup/runtime contract tests; +- headers include obsolete `X-XSS-Protection` and lack a demonstrated CSP; +- wildcard icons and many font families need bundle/waterfall review; +- disabling buttons controls UX but is not server idempotency. + +### Verification + +- enumerate every route against route intent; +- prove public/auth/auth-required/app cache and session behavior; +- import client bundle without server env; +- test native/server validation plus form pending/error/focus; +- test expired session, auth callback/origin, and provider capability mismatch; +- inspect route bundles, fonts, icons, and headers. + +## ThunderStrike runtime CMS site + +### Observed active architecture + +- Astro server output with auto-selected adapter and Emdash integration. +- local SQLite/media defaults for development, with configured runtime env. +- one live collection loader for Emdash. +- project `cms.ts` maps provider entries into article/author/topic/category/media view models. +- mapping resolves relationships, headings, plain text, dates, publication status, and cache hints. +- article/listing/taxonomy/feed routes consume the adapter. + +### Useful decisions + +The adapter prevents provider records from becoming the page contract. Provider-specific Portable Text remains at the content renderer boundary. Legacy local content can remain migration input without being a second runtime source. + +### Counterexample endpoint + +The webhook source must not be copied: + +- logs API key and complete environment; +- logs payload fields containing PII; +- no demonstrated raw-body signature verification or replay window; +- broad passthrough payload schema; +- provider extraction, group mapping, delivery, and logging are tightly coupled; +- error exposure and idempotency are not a proven production contract. + +The mapping tables may be useful domain data after validation. The endpoint lifecycle is negative evidence. + +### Verification + +- public/draft/preview direct/list/feed parity; +- missing relations/media and invalid rich text; +- provider outage/cache behavior and N+1 count; +- XSS through blocks, marks, links, embeds, and search snippets; +- webhook signature/replay/idempotency/redaction rewrite; +- adapter target, storage, sessions, and environment startup. + +## Solid motion experiment + +### Observed scope + +- Solid-owned MotionState and reactive option snapshots; +- framework-neutral `motion-dom` value/DOM primitives; +- pure server initial-style resolution; +- server-render plus hydration-diagnostic tests; +- `PresenceChild` completion aggregation; +- single-slot `AnimatePresence` with limited modes; +- explicit research on priority, layout timing, presence, and Solid 2 risk. + +### Correct interpretation + +This is strong evidence about the seams a Solid animation adapter needs. It is experimental evidence, not proof of full gesture, keyed-list presence, layout projection, or Solid 2 parity. + +The case teaches: + +- type/API surface must follow runtime capability; +- logical removal and physical owner retention are separate; +- initial styles must be pure and hydration-identical; +- cleanup must survive retained exits; +- high-level animation-engine internals should not be adopted until adapter seams exist. + +### Verification -## ThunderStrike CMS site +- target priority/cancellation tests; +- explicit/false/omitted initial SSR-hydration tests; +- nested/zero-animation/cancel/reentry presence cases; +- keyed-list claims forbidden until executable coverage exists; +- resource and owner counts after disposal. -The useful pattern is a project-owned CMS adapter that maps provider records to -stable article/author/topic/category/media shapes. Portable Text remains at the -article boundary, and legacy local content is migration input rather than a -second runtime source. +## Cross-case decisions -The webhook is negative evidence: it logs secrets and PII, skips provider -verification, exposes mutation through GET, returns provider errors, and splits -environment ownership. Never copy it as a site endpoint pattern. +| Decision | Evidence lesson | +|---|---| +| Native versus island | Choose by behavior and lifetime, not file type | +| Static versus server | Choose by request-time dependency per route | +| Content source | One runtime owner plus project mapping | +| Auth capability | Server derives; client receives minimal public model | +| Generated docs | Share service factories; build statically when possible | +| Motion | Claim only implemented/tested lanes and lifetime behavior | +| Uploaded code | Distinguish positive, counterexample, and experimental source | +| Configuration | Verify installed version and target runtime before copying | -## High-value checks +## Sources and freshness -- replace a hydrated FAQ with native disclosure and prove no island bundle; -- require an explicit renderer for a `client:only` component; -- ensure repeated Astro navigation does not multiply listeners; -- retain a static visual when WebGL or an image fails; -- map incomplete CMS records through one validated adapter; -- forbid secret/PII logs and mutating GET webhooks. +- `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `old-finance-app(1).zip`, `thunderstrike-blog(4).zip`, and `solid-motion-experiments.zip`, reviewed 2026-07-17. +- These cases preserve observed ownership and failure lessons. Package APIs, versions, and experimental configs must be reverified in the target repository. diff --git a/skills/build-sites/references/content.md b/skills/build-sites/references/content.md index c211378..c47ee47 100644 --- a/skills/build-sites/references/content.md +++ b/skills/build-sites/references/content.md @@ -1,43 +1,282 @@ -# Content and CMS boundaries +# Content collections, CMS adapters, rich text, and publishing -## Stable view models +Use this reference for local Astro content, live/runtime CMS data, migrations, previews, taxonomies, media, rich text, feeds, and SEO. The project should have one explicit runtime source of truth per route and a stable view-model boundary. -Use this boundary: +## Contents + +- Content evidence inventory +- Source modes and ownership +- Astro local collections +- Runtime/live CMS adapter +- Relationships and taxonomy +- Rich text and media +- Drafts, preview, scheduling, and cache +- SEO, feeds, and discoverability +- Migration and dual-source control +- Failure signatures +- Verification +- Sources and freshness + +## Content evidence inventory + +Inspect: + +- `src/content.config.*`, `src/live.config.*`, content directories, seed/migration inputs; +- CMS integration in Astro config and generated environment/types; +- actual page/query imports; +- content schemas and relationship/reference fields; +- mapping/adapter modules; +- provider query, cache hints, pagination, status filtering, and batch APIs; +- rich-text renderer and custom block/mark handling; +- media storage/URL/transform policy; +- preview/draft/schedule authorization; +- canonical, sitemap, robots, structured data, and feed generation; +- failure behavior and tests. + +Do not infer source ownership from which directory contains more content. Trace the runtime route. + +## Source modes and ownership + +Name every source mode: + +| Mode | Appropriate use | Primary risk | +|---|---|---| +| Build-time local collection | Versioned editorial/docs content | Rebuild required for change | +| Build-time remote import | Deterministic snapshot in CI | Network nondeterminism/provenance | +| Runtime/live collection | Editor changes without rebuild | Availability, cache, authorization | +| Preview/draft | Authorized editorial review | Draft leakage/cache pollution | +| Generated documents | OpenAPI/catalog/schema output | Drift from runtime factory | +| Legacy/migration input | Reproducible transformation | Accidental second runtime source | + +For each route choose one runtime source of truth. A migration can retain raw legacy content and mapping reports without querying both providers at request time. + +Use a project-owned view model: ```text -content collection or CMS record - -> source-specific validation and mapping - -> project-owned page model - -> route, layout, and components +local/CMS provider record + -> provider schema validation + -> project mapping and policy + -> stable Article/Page/Author/Media model + -> layout and components +``` + +Pages should not know provider field names, database ids, cache hint types, or raw error shapes. + +## Astro local collections + +The Kaiju local content config demonstrates separate collections for authors, series, topics, categories, articles, and pages, with Astro `reference()` relationships and image-aware schemas. The exact `astro:content`, loader, and Zod APIs depend on the installed Astro major; verify them. + +Design schemas around editorial contracts: + +```ts +const articles = defineCollection({ + loader: glob({ pattern: "**/*.{md,mdx}", base: "./src/content/articles" }), + schema: ({ image }) => z.object({ + title: z.string(), + description: z.string().optional(), + publishedAt: z.coerce.date(), + updatedAt: z.coerce.date().optional(), + authors: z.array(reference("authors")).default([]), + heroImage: image().optional(), + heroImageAlt: z.string().optional(), + draft: z.boolean().default(true), + canonicalURL: z.url().optional(), + }), +}); ``` -Pages should not spread raw provider records through the component tree. Map -slugs, authors, taxonomies, media, body, dates, draft state, and cache hints once. -Keep provider-specific rich-text rendering at an explicit boundary. +This is a pattern, not a copy-ready current API. Define invariants that cross fields: + +- hero image requires useful alt unless decorative; +- scheduled/published content has valid dates; +- canonical URL follows external canonical policy; +- series number requires a series; +- required author/category relations resolve; +- draft defaults are safe; +- related content does not self-reference if forbidden. + +Do not model every field optional to make a migration pass. Separate incomplete migration input from publishable content. + +## Runtime/live CMS adapter + +The ThunderStrike upload uses one Emdash live collection and queries named content types through provider functions. Its `cms.ts` maps provider entries into project-owned `CmsArticle`, `CmsAuthor`, `CmsTopic`, `CmsCategory`, `CmsImage`, heading, and page types. This boundary is the reusable architecture; the provider API is version-sensitive. + +Adapter responsibilities: + +- validate provider response and distinguish no record from provider failure; +- apply published/draft/preview policy centrally; +- normalize ids/slugs/dates and invalid-date behavior; +- batch or cache relationship lookups; +- map media to stable source/alt/dimensions; +- map taxonomy and bylines; +- retain cache hints without exposing provider types to pages; +- provide stable pagination and sort order; +- convert provider errors into project diagnostics; +- expose raw/provider data only in an authorized debug path. + +```ts +type ContentResult<T> = + | { ok: true; value: T; cache?: unknown } + | { ok: false; kind: "not-found" | "invalid" | "unavailable"; cause?: unknown }; + +async function loadPublishedArticle(slug: string): Promise<ContentResult<Article>> { + const result = await provider.getEntry("posts", slug); + if (result.error) return { ok: false, kind: "unavailable", cause: result.error }; + if (!result.entry || result.entry.status !== "published") { + return { ok: false, kind: "not-found" }; + } + return mapArticle(result.entry); +} +``` + +Do not silently use `new Date(0)` for required publish dates without a product policy. Epoch fallbacks can put malformed content into archives/feeds. Return invalid content or use an explicit draft-only fallback. + +## Relationships and taxonomy + +Define identity at every boundary: + +- provider/database id; +- stable content id; +- route slug; +- canonical URL; +- author/byline identity; +- taxonomy name and term slug. + +The Emdash guidebook inside the upload warns that taxonomy names must match the seed exactly and that relationship functions may require the database id rather than route slug. Treat that as provider-specific and test it. An empty taxonomy result without an error is a dangerous failure mode. + +Avoid N+1 queries. Prefer provider batch APIs or load relationship maps for a page of entries. If the provider lacks batching, cache within the request and measure. + +Map relationships once: + +```ts +type ArticleCard = { + id: string; + slug: string; + title: string; + authors: readonly { slug: string; name: string }[]; + topics: readonly { slug: string; name: string }[]; +}; +``` + +Define missing-relation behavior: build failure, draft exclusion, omitted optional relation, safe placeholder, or operator-visible invalid state. Do not insert invented authors or categories. + +## Rich text and media + +Keep provider-specific rich text at a named renderer boundary. Support only inspected block/mark types and define unknown behavior. + +```text +Portable Text block + -> paragraph/heading/list/quote/code/image/custom block renderer + -> mark renderer (strong/em/link/code) + -> URL and embed policy + -> safe HTML/DOM +``` + +Never concatenate untrusted children into HTML. If a fallback renderer emits HTML, escape text and use an allowlisted mapping. For links validate protocol and apply external-link policy. For embeds validate provider and sandbox/capabilities. + +Unknown blocks should produce one of: + +- visible safe unsupported-block placeholder in preview; +- diagnostic plus omitted block in production; +- hard failure for publish/build if loss is unacceptable. + +Media model: + +```ts +type Media = { + id: string; + src: string; + alt: string; + width?: number; + height?: number; + mimeType?: string; + credit?: string; +}; +``` + +Resolve provider ids/URLs at the adapter, not throughout components. Verify local/private storage URLs, transformations, image service compatibility, missing dimensions, alt policy, remote domains, and deletion behavior. + +## Drafts, preview, scheduling, and cache + +Publishing state is authorization plus cache policy: + +- public queries explicitly require published status; +- preview requires an authenticated/authorized editor or a bounded signed token; +- preview responses are `private, no-store` and never enter public page/CDN cache; +- scheduled content uses one documented timezone and current-time source; +- draft/schedule changes invalidate the right public artifacts; +- feeds, sitemap, listing, taxonomy, and direct slug use the same visibility policy. + +Do not rely on hiding draft links if the direct route can query the entry. Do not cache preview HTML under the public URL without a varied/private key. + +For runtime CMS failures define stale-if-error versus fail-closed behavior. Stale public article content may be acceptable; stale preview or access policy may not be. + +## SEO, feeds, and discoverability + +Derive all discovery surfaces from the same mapped model: + +- unique title and description; +- canonical absolute URL; +- Open Graph/Twitter image and alt if supported; +- article dates/authors and structured data; +- sitemap inclusion/exclusion; +- robots policy; +- RSS/Atom content and absolute links; +- archive/taxonomy pagination; +- redirects from migrated slugs; +- 404/410 decision for removed content. + +Do not use request host blindly as canonical origin behind proxies. Configure the trusted production site origin and test preview environments. + +Feeds must escape/serialize content safely and match public visibility. A feed that sees drafts or a different sort order is a source-ownership defect. + +## Migration and dual-source control + +Migration sequence: -## Source modes +1. Preserve raw source and hashes/provenance. +2. Define source-to-target field and relationship map. +3. Transform into a versioned intermediate/project model. +4. Report invalid, missing, defaulted, and dropped data. +5. Import idempotently with stable identity. +6. Compare counts, slugs, relationships, media, dates, and sampled rendered output. +7. Generate redirects/canonical map. +8. Switch each route to one new runtime source. +9. Retain legacy files as test/migration fixtures or remove when authorized. -Distinguish: +Never merge local and CMS arrays at runtime to “avoid losing content” without identity/conflict rules. That creates duplicate slugs, conflicting dates, and nondeterministic feeds. -- build-time content collections; -- request-time/live collections; -- local legacy content used only for migration; -- generated OpenAPI or service catalogs; -- drafts and previews; -- retained raw evidence for reproducible migration. +## Failure signatures -One repository may contain more than one mode, but each route needs one runtime -source of truth. +| Signature | Likely cause | Next inspection | +|---|---|---| +| Draft appears in feed only | Visibility policy duplicated | Shared published query/model | +| Taxonomy page empty | Provider taxonomy name/id mismatch | Seed/schema and adapter calls | +| One page issues dozens of CMS calls | N+1 relationship loading | Batch/cache strategy | +| Missing author crashes whole listing | Required relation not validated | Publish schema and mapper policy | +| Article date is 1970 | Silent epoch fallback | Invalid-date mapping | +| Rich text loses links/marks | Plain-text fallback treated as full renderer | Block/mark capability matrix | +| Search snippet executes markup | Provider HTML inserted raw | Sanitization/mapping boundary | +| Local and CMS article both render | Dual runtime source | Route query and migration switch | +| Preview leaks publicly | Auth/cache key missing | Preview route and headers | +| Image works locally, fails deployed | Storage/image service URL mismatch | Adapter, remote domain, base URL | -## Failure states +## Verification -Test missing media, incomplete author/taxonomy records, drafts, deleted content, -provider outage, stale cache, invalid rich text, redirect/canonical changes, and -feed generation. Define whether build should fail, skip, retain stale content, or -render a safe fallback. +1. Parse/validate every local entry and representative provider record. +2. Test required/optional/missing relationships and invalid dates. +3. Test public/draft/scheduled/preview visibility through direct and listing routes. +4. Test allowed and unknown rich-text blocks, marks, URLs, and XSS payloads. +5. Test media missing, private, deleted, malformed, and dimensionless cases. +6. Assert query counts to catch N+1 regressions. +7. Compare article/listing/taxonomy/feed/sitemap visibility and order. +8. Test cache hints, provider outage, stale fallback, and invalid response. +9. Run migration twice and assert idempotency plus reconciliation report. +10. Crawl built/runtime routes for broken links, canonicals, assets, and redirects. -## Discoverability +## Sources and freshness -Keep canonical URLs, titles, descriptions, social cards, structured data, -sitemap, robots policy, redirects, and RSS/Atom aligned with the route/content -model. Validate absolute URLs and environment-specific origins. +- Uploaded `kaiju-website(6).zip` local Astro collection schemas, reviewed 2026-07-17. +- Uploaded `thunderstrike-blog(4).zip` Emdash guidebooks, live configuration, CMS adapter, routes, and counterexample webhook, reviewed 2026-07-17. +- Uploaded `kaiju-site-scope(17).zip` static service documentation factories, reviewed 2026-07-17. +- Astro content APIs and Emdash APIs are version-sensitive. Verify the installed Astro/EMdash packages and generated types. Local Emdash examples are observed source, not a substitute for current public documentation. diff --git a/skills/build-sites/references/fonts.md b/skills/build-sites/references/fonts.md new file mode 100644 index 0000000..5c55179 --- /dev/null +++ b/skills/build-sites/references/fonts.md @@ -0,0 +1,260 @@ +# Astro Fonts API and Fontsource + +## Contents + +- Ownership decision +- Astro Fonts API +- Fontsource packages +- Local and variable fonts +- Privacy and external providers +- Fallback metrics and layout shift +- Preload and delivery policy +- Typography contract +- Failure signatures +- Verification +- Sources and freshness + +## Ownership decision + +Do not combine every font mechanism in one project without an owner. + +| Need | Strong default | Why | +|---|---|---| +| Astro 6 site wants unified provider/local configuration and optimized fallbacks | Astro Fonts API | Central typed config, cached/self-hosted provider assets, `Font` component and selective preload | +| Vite/Solid app outside Astro | `@fontsource/*` or `@fontsource-variable/*` | Package-owned CSS/font files work at the app entry/layout | +| Existing app already imports Fontsource | Keep Fontsource unless migration has measured value | Avoid duplicate font faces and asset downloads | +| Licensed/private font file | Astro local provider or project-owned `@font-face` | Source and licensing remain explicit | +| Very custom `@font-face` descriptors/subsetting pipeline | project-owned CSS/build pipeline | Higher control than unified provider abstraction | + +Astro's `fontProviders.fontsource()` uses Fontsource as a provider through Astro's font pipeline. It is different from importing `@fontsource-variable/inter` CSS in an application. Choose one owner for each family. + +## Astro Fonts API + +The uploaded Astro configurations use the Astro 6 `fonts` array with Google and local providers: + +```ts +import { defineConfig, fontProviders } from "astro/config"; + +export default defineConfig({ + fonts: [ + { + provider: fontProviders.google(), + name: "Inter", + cssVariable: "--font-inter", + weights: [400, 500, 600, 700], + styles: ["normal"], + subsets: ["latin"], + fallbacks: ["Arial", "sans-serif"], + }, + { + provider: fontProviders.local(), + name: "American Kestrel", + cssVariable: "--font-american-kestrel", + fallbacks: ["sans-serif"], + options: { + variants: [ + { + src: ["./src/assets/fonts/american-kestrel/americankestral.woff2"], + weight: 400, + style: "normal", + }, + ], + }, + }, + ], +}); +``` + +Include the configured family in the document head: + +```astro +--- +import { Font } from "astro:assets"; +--- + +<head> + <Font cssVariable="--font-inter" preload={[{ weight: 400, style: "normal", subset: "latin" }]} /> +</head> +``` + +Configuring a family does not prove every page emits its `@font-face` and preload assets. Inspect the active root layout and built HTML. + +Astro's default `weights` is intentionally narrow (commonly 400). Declare only weights/styles actually used. A CSS class with `font-weight: 700` does not cause the correct file to exist automatically. + +Astro can use providers such as Google, Fontsource, or local depending on the installed version. Provider names, options, and availability are versioned: inspect `astro/config` types rather than copying a newer config into an older Astro release. + +## Fontsource packages + +For a Vite/Solid app: + +```ts +// One variable file for the supported weight range. +import "@fontsource-variable/inter"; +``` + +For static fonts, import only required weights/styles: + +```ts +import "@fontsource/inter/400.css"; +import "@fontsource/inter/600.css"; +import "@fontsource/inter/700.css"; +``` + +Then own the CSS token: + +```css +:root { + --font-sans: "Inter Variable", Inter, Arial, sans-serif; +} + +body { + font-family: var(--font-sans); +} +``` + +The uploaded TanStack Solid frontend uses `@fontsource-variable/inter` 5.2.8. That establishes a package choice, not that the import is wired: locate the actual CSS/entry import and built font requests. + +Fontsource variable packages expose axis-specific CSS files. Import one appropriate axis bundle; importing multiple axis bundles can duplicate the weight axis and downloads. Verify the font's actual axes before writing `font-variation-settings`. + +## Local and variable fonts + +Prefer WOFF2 for web delivery. Keep original licensed sources outside the public bundle when terms require it. Record: + +- family and PostScript names; +- weight range and style; +- axes and valid ranges; +- Unicode subsets; +- license and attribution; +- source/version/checksum; +- subsetting command and reproducibility; +- fallback family and metric tuning. + +Astro local variable font: + +```ts +{ + provider: fontProviders.local(), + name: "Inter", + cssVariable: "--font-inter", + options: { + variants: [ + { + src: ["./src/assets/fonts/InterVariable.woff2"], + weight: "100 900", + style: "normal", + }, + ], + }, +} +``` + +Do not declare a broad range the file does not contain. Do not serve TTF by default when a WOFF2 build is available. The uploaded Kaiju configuration uses a `.ttf` local source; treat conversion/licensing/visual regression as an improvement opportunity, not a silent rewrite. + +## Privacy and external providers + +Astro's font provider pipeline can download and cache provider fonts so the deployed site serves them, avoiding a direct visitor request to the third party. Verify the built output and provider behavior at the installed Astro version. + +Direct remote `<link>` tags can disclose visitor IP/user-agent/referrer and introduce an external availability dependency. If remote delivery is required, document consent/legal basis, CSP, outage fallback, caching, integrity constraints where applicable, and regional behavior. + +Fontsource package imports and local providers are self-hosted by the application build. Pin versions/checksums for reproducibility. + +## Fallback metrics and layout shift + +Font loading can change line breaks, element height, button width, and hero balance. Choose fallbacks by metrics, not generic family alone. + +Control: + +- `font-display` policy; +- fallback family order; +- size adjustment; +- ascent/descent/line-gap overrides where the owner supports them; +- fixed/robust line heights; +- container layouts tolerant of text reflow; +- language-specific glyph coverage. + +Astro optimized fallbacks are enabled by default in Astro 6 and can generate metric-adjusted fallbacks. If disabling `optimizedFallbacks`, own equivalent metrics deliberately. + +Test the fallback and final font separately. CLS can remain low while the design visibly jumps inside a fixed-height component; use screenshots/video as well as numeric metrics. + +## Preload and delivery policy + +Preload only the first-paint font files actually used above the fold. Each preload competes with CSS, JavaScript, images, and navigation requests. + +Astro: + +```astro +<Font + cssVariable="--font-inter" + preload={[ + { subset: "latin", style: "normal", weight: 400 }, + { subset: "latin", style: "normal", weight: 600 }, + ]} +/> +``` + +Fontsource/Vite manual preload uses the built asset URL: + +```ts +import inter400 from "@fontsource/inter/files/inter-latin-400-normal.woff2?url"; +``` + +```tsx +<link rel="preload" as="font" type="font/woff2" href={inter400} crossOrigin="anonymous" /> +``` + +Do not preload all families, weights, italics, and subsets. Verify that a preload URL exactly matches a later font request; otherwise it is wasted. + +Define caching for hashed immutable assets. Avoid CDN `latest`; pin an exact package or URL version. + +## Typography contract + +Inventory actual use before editing configuration: + +```bash +rg -n 'font-(sans|serif|mono|\[)|font-family|font-weight|--font-' src +rg -n 'fontProviders|fonts:|@fontsource|astro:assets' astro.config.* src package.json +``` + +Map tokens to roles: + +| Token | Role | Needed faces | +|---|---|---| +| `--font-sans` | UI/body | regular, medium, semibold, bold if used | +| `--font-display` | headings/hero | actual used range; often no italic | +| `--font-mono` | code/data | regular plus bold only if code uses it | +| brand/decorative | bounded logo/hero accent | one face, fallback, reduced criticality | + +Avoid loading a large family solely for one word if a vector/asset or existing face meets the design and licensing constraints. Do not synthesize bold/italic unknowingly; disable synthesis where fidelity matters after verifying browser support and real files. + +## Failure signatures + +| Signature | Likely cause | Required inspection | +|---|---|---| +| 400 only despite bold CSS | weights omitted from config/imports | built CSS and font network requests | +| two downloads for same face | Astro provider plus Fontsource import | source imports and generated `@font-face` | +| font flashes every navigation | CSS/head lifecycle or cache mismatch | Astro navigation and response cache headers | +| local font builds in dev only | bad source path/case or adapter asset handling | production build output | +| preload warning “not used” | preload does not match selected face/subset/CORS | built URL and computed font | +| hero shifts after font load | fallback metrics/line height mismatch | throttled recording and CLS entries | +| some language renders tofu | missing subset/glyphs | content corpus and font cmap/subsets | +| variable weights look identical | wrong package/CSS axis bundle or invalid range | built `@font-face` and computed styles | +| privacy expectation violated | direct provider request remains | browser network and CSP report | + +## Verification + +1. Run Astro check and production build. +2. Inspect generated HTML/CSS for one owner per family and correct faces. +3. Use a cold browser profile and record font requests, initiators, transfer sizes, cache headers, and third-party origins. +4. Throttle network/CPU and verify fallback, swap, layout, and no-JS rendering. +5. Test representative routes, languages, weights, italics, code blocks, forms, and dialogs. +6. Measure CLS/LCP and visually compare desktop/mobile breakpoints. +7. Confirm preloads are consumed and remove unused ones. +8. Build offline when self-hosted reproducibility is a requirement. +9. Verify licenses and asset source records. + +## Sources and freshness + +- Primary: [Astro Fonts guide](https://docs.astro.build/en/guides/fonts/) and [Fontsource documentation](https://fontsource.org/docs/), verified 2026-07-17. +- Attachments: `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `thunderstrike-blog(4).zip`, and `new-finance-app(1).zip`, inspected 2026-07-17 for real Astro/provider/CSS usage. + +Astro Fonts is a stable top-level configuration surface in current Astro and was added in Astro 6. Older Astro 5 material used an experimental flag and is not a current configuration example. Provider metadata still changes independently, so recheck the installed Astro and Fontsource versions, font licenses, generated CSS, and network behavior. diff --git a/skills/build-sites/references/icons.md b/skills/build-sites/references/icons.md new file mode 100644 index 0000000..3b8527a --- /dev/null +++ b/skills/build-sites/references/icons.md @@ -0,0 +1,219 @@ +# Astro Icon and Unplugin Icons + +## Contents + +- Ownership decision +- Astro Icon setup +- Unplugin Icons setup +- Renderer boundaries +- Local collections +- Styling and accessibility +- Bundle and security controls +- Migration and failure signatures +- Verification +- Sources and freshness + +## Ownership decision + +Choose by the component that renders the SVG, not by repository-wide preference. + +| Surface | Default owner | Reason | +|---|---|---| +| `.astro` page/layout/component icon | `astro-icon` and `<Icon>` | Astro renders static SVG without a framework island | +| Solid island/app component | `unplugin-icons` with `compiler: "solid"` | Produces a Solid component with renderer-native props | +| React island | `unplugin-icons` with `compiler: "jsx"`/React-compatible integration | Avoids passing an Astro component into React | +| Astro component that already uses Unplugin virtual imports | `unplugin-icons` with `compiler: "astro"` | Valid, but choose one convention per surface | +| dynamic user-selected icon name | explicit allowlist/registry | Unbounded runtime lookup undermines bundling and review | +| project-owned SVG | local Astro asset/component or a sanitized local collection | Keeps provenance and styling explicit | + +Do not use a Solid/React icon import in an Astro-only component merely for visual consistency. Match the visual collection (for example Fluent Regular) while keeping renderer ownership correct. + +## Astro Icon setup + +The uploaded Astro sites use `astro-icon` 1.x: + +```ts +import { defineConfig } from "astro/config"; +import icon from "astro-icon"; + +export default defineConfig({ + integrations: [ + icon({ + include: { + fluent: [ + "arrow-right-24-regular", + "checkmark-24-regular", + "navigation-24-regular", + ], + }, + }), + ], +}); +``` + +Use the server-rendered component in `.astro` files: + +```astro +--- +import { Icon } from "astro-icon/components"; +--- + +<a href="/reports" class="inline-flex items-center gap-2"> + View reports + <Icon name="fluent:arrow-right-24-regular" class="size-4" aria-hidden="true" /> +</a> +``` + +Prefer an explicit include list. The uploaded Kaiju and ThunderStrike configs currently include wildcard Fluent collections. That is evidence of existing configuration, not a recommended bundle policy. Replace wildcards only after inventorying literal and generated names so required icons do not disappear. + +Astro Icon names combine collection and icon name. Verify them against the installed Iconify collection or local collection; do not guess a Fluent suffix (`regular`, `filled`) or size. + +## Unplugin Icons setup + +For a Solid application or island: + +```ts +import { defineConfig } from "vite"; +import solid from "vite-plugin-solid"; +import Icons from "unplugin-icons/vite"; +import { FileSystemIconLoader } from "unplugin-icons/loaders"; + +export default defineConfig({ + plugins: [ + solid({ ssr: true }), + Icons({ + compiler: "solid", + customCollections: { + kaiju: FileSystemIconLoader("./src/icons"), + }, + }), + ], +}); +``` + +Add the renderer-specific type package to the TypeScript program when the installed version requires it: + +```json +{ + "compilerOptions": { + "types": ["unplugin-icons/types/solid"] + } +} +``` + +Then import only the icons used: + +```tsx +import ArrowRight from "~icons/fluent/arrow-right-24-regular"; +import KaijuMark from "~icons/kaiju/mark"; + +export function Action() { + return ( + <button type="button" class="inline-flex items-center gap-2"> + Continue + <ArrowRight aria-hidden="true" class="size-4" /> + </button> + ); +} +``` + +Verify the virtual-module prefix generated by the installed plugin. Some ecosystems use `~icons/...`; auto-import integrations may expose other conventions. Do not copy an import from a different build setup without checking resolution. + +For Astro virtual components, configure `compiler: "astro"` and include `unplugin-icons/types/astro` if needed. Do not configure `compiler: "solid"` and then use the component directly in an Astro template. + +## Renderer boundaries + +The uploaded repositories expose a real drift failure: React type entries remained in Astro/Solid-oriented projects. Audit these as one contract: + +- Astro integrations (`@astrojs/solid-js`, `@astrojs/react`); +- Vite renderer plugin; +- Unplugin compiler; +- `types` entries; +- component file extensions; +- island client directives; +- generated shadcn/Zaidan components; +- tests and SSR transforms. + +A type declaration can hide an incorrect compiler until production SSR. Build both server and client bundles. + +## Local collections + +`FileSystemIconLoader("./src/icons")` maps file paths to icon names. Establish: + +- lowercase, stable file names; +- one viewBox convention; +- whether `fill="currentColor"` or `stroke="currentColor"` is expected; +- removal of scripts, event handlers, external references, embedded images, and unsafe styles; +- SVGO policy that preserves required IDs, masks, gradients, titles, and viewBox; +- licensing/source metadata for imported assets; +- a collision policy between local and Iconify collections. + +Do not accept arbitrary uploaded SVG as a build-time icon without sanitization. Do not run aggressive optimization across hand-authored complex logos without visual regression review. + +## Styling and accessibility + +Decorative icons: + +```astro +<Icon name="fluent:checkmark-24-regular" aria-hidden="true" focusable="false" /> +``` + +An icon-only control needs an accessible name on the control, not necessarily the SVG: + +```tsx +<button type="button" aria-label="Open navigation"> + <NavigationIcon aria-hidden="true" class="size-5" /> +</button> +``` + +For an informative standalone SVG, verify how the renderer emits `<title>` and ARIA relationships; do not assume passing `title` produces an accessible name. + +Use `currentColor` so icons inherit state color. Keep icon dimensions explicit to prevent layout shift. Avoid mixing filled and regular families accidentally. Preserve stroke width and optical size conventions within one control group. + +## Bundle and security controls + +- Prefer literal imports so bundlers include only used icons. +- Avoid `autoInstall: true` in reproducible CI unless package-manager/network behavior is deliberate. +- Pin `@iconify-json/<collection>` packages when offline/reproducible builds matter. +- Restrict Astro Icon `include` to a reviewed list. +- Do not construct arbitrary virtual import paths from user input. +- Keep local SVGs free of secrets, metadata, and executable content. +- Measure generated HTML when repeated inline SVGs dominate response size; sprite behavior and component output vary by tool/version. + +## Migration and failure signatures + +| Signature | Likely cause | Correction | +|---|---|---| +| virtual icon module not found | missing collection, plugin, or wrong prefix | inspect Vite plugin and installed collection | +| component type is React in Solid | stale `types`/compiler configuration | align compiler and TS types | +| icon renders in dev but not SSR | renderer/compiler mismatch | run production SSR build | +| huge dependency/build graph | wildcard include or auto-install | inventory and pin exact collection/icons | +| icon has no visible color | hard-coded fill/stroke or missing `currentColor` | normalize source and component class | +| local icon cropped | inconsistent/missing viewBox | inspect SVG coordinate system | +| screen reader announces meaningless icon | decorative SVG not hidden | name the control and hide SVG | +| dynamic icon name fails after build | bundler could not discover runtime path | use an explicit component registry | + +## Verification + +```bash +deno task check +deno task build +rg -n 'astro-icon|~icons/|FileSystemIconLoader|unplugin-icons' src astro.config.* vite.config.* +``` + +Also: + +1. render every icon state in Astro and each active island renderer; +2. inspect server HTML for accessible names and fixed dimensions; +3. test production SSR/hydration, not dev only; +4. check bundle/module graph for unintended collections; +5. build with the network unavailable when reproducibility is required; +6. visually compare local SVGs after optimization; +7. test keyboard/focus/high-contrast/current-color behavior. + +## Sources and freshness + +- Primary: [Astro Icon documentation](https://www.astroicon.dev/) and [Unplugin Icons repository/documentation](https://github.com/unplugin/unplugin-icons), verified 2026-07-17. +- Attachments: `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `thunderstrike-blog(4).zip`, and `new-finance-app(1).zip`, inspected 2026-07-17 for actual configs and component imports. + +Compiler names, virtual-module types, custom loaders, and Astro integration options are version-sensitive. Verify the target lockfile and generated types rather than extrapolating from one attachment. diff --git a/skills/build-sites/references/site-quality.md b/skills/build-sites/references/site-quality.md index a426fec..e394b69 100644 --- a/skills/build-sites/references/site-quality.md +++ b/skills/build-sites/references/site-quality.md @@ -1,28 +1,185 @@ -# Site quality and verification +# Site quality gates -## Accessibility +Use this reference to define acceptance criteria for public sites and documentation. Quality is a set of observable route contracts, not a Lighthouse score or attractive screenshot. -Verify landmarks, heading order, link purpose, disclosure semantics, keyboard -interaction, focus visibility, dialogs, form labels/errors, contrast, zoom, -reflow, reduced motion, and fallback content. Native HTML can eliminate entire -classes of island and accessibility defects. +## Contents -## Performance +- Quality inventory +- Document and accessibility contract +- Content and discoverability +- Performance budget +- Resilience and progressive enhancement +- Security and privacy +- Visual and interaction quality +- Deployment and operations +- Verification matrix +- Sources and freshness -Measure rendered HTML, JavaScript by island, image dimensions/formats, font -loading, icon inclusion, third-party scripts, layout shift, interaction latency, -and continuous frame work. Wildcard icon collections and hydrated static sections -need explicit justification. +## Quality inventory -## Resilience +For each route family record: -Test no JavaScript where progressive enhancement is claimed, image and WebGL -failure, slow CMS/API responses, invalid content, repeated client navigation, -offline/static asset behavior, and server error pages. +```text +Audience and user task: +Primary content/action: +Static/request-time/deferred rendering: +Canonical and index policy: +Required JavaScript/islands: +Images/fonts/icons/embeds: +CMS/API/auth dependencies: +No-JS behavior: +Failure fallback: +Performance budget: +Accessibility risks: +Privacy/security boundaries: +Deployment adapter and cache: +``` -## Verification commands +Use budgets tied to the route. A WebGL marketing hero, API reference, article, and authenticated settings page need different checks. -Use repository-owned lint/type/content checks, Astro check, static/server build, -link and feed checks, browser accessibility tests, responsive screenshots where -visual changes matter, and adapter/deployment smoke tests. Keep code formatters -scoped away from authored Markdown. +## Document and accessibility contract + +Every page needs: + +- correct `<html lang>` and unique, front-loaded title; +- one useful main landmark, restrained landmarks, and a skip link when repeated navigation exists; +- meaningful heading outline; +- link purpose that remains understandable out of context; +- native interactive elements where possible; +- visible focus and logical order; +- accessible names for icon-only controls; +- labeled forms, instructions, field errors, and submission status; +- image alternatives and decorative SVG/canvas removed from the accessibility tree; +- semantic tables and text/data alternatives for charts; +- usable zoom/reflow, contrast, forced colors, coarse pointer, and reduced motion; +- route announcement/focus behavior during client navigation. + +Automated accessibility checks cannot validate focus strategy, announcement quality, alt usefulness, or whether a custom widget implements its keyboard contract. Inspect the accessibility tree and perform keyboard/screen-reader smoke tests. + +Native HTML can eliminate entire defect classes. The Kaiju homepage's `<details>` FAQ is preferable to an unused hydrated accordion when ordinary disclosure semantics are sufficient. + +## Content and discoverability + +Validate from the actual mapped content model: + +- page title, description, canonical, social card, and structured data; +- absolute production URLs with correct base path; +- sitemap, robots, feed, pagination, taxonomy, and archives; +- draft/preview/scheduled exclusion; +- canonical and redirects for migrated slugs; +- missing author/media/taxonomy and deleted content behavior; +- heading ids/table of contents stability; +- code/reference links and OpenAPI schema links; +- useful 404 and error pages. + +Do not generate metadata independently in page, feed, and sitemap layers. A shared view model/policy should make them agree. + +For API docs, diff generated OpenAPI and route catalogs. Ensure examples contain no real tokens, cookies, secrets, private hosts, or customer data. + +## Performance budget + +Measure the route rather than repeating generic advice: + +| Resource/work | Evidence | +|---|---| +| HTML | Required content present before JS; reasonable size | +| JavaScript | Per-island chunks, duplicate renderer runtimes, unused islands | +| CSS | Critical styles, generated component layers, unused/global cost | +| Images | Intrinsic dimensions, responsive sources, format, priority, alt | +| Fonts | Critical faces only, subset/variants, preload, fallback metrics | +| Icons | Finite imports/collections; renderer/compiler match | +| Third parties | Origin, blocking cost, consent, failure isolation | +| Motion/WebGL | frames, long tasks, DPR/texture cap, background/offscreen suspension | +| DOM/rendering | large lists, `content-visibility`, layout/paint cost | + +Track Core Web Vitals in representative conditions, but keep causal artifacts such as waterfalls, traces, bundle analysis, and long-task/frame evidence. A score alone does not identify ownership. + +For `content-visibility: auto`, target large below-fold sections, pair with `contain-intrinsic-size`, and test keyboard accessibility. Do not apply it indiscriminately above the fold. + +## Resilience and progressive enhancement + +Test: + +- JavaScript disabled or island chunk blocked; +- slow and failed images, fonts, icons, third-party embeds; +- no WebGL/context/texture and reduced motion; +- CMS/API timeout, invalid record, stale cache, and outage; +- repeated Astro client navigation, back/forward, and hash links; +- offline/static-asset caching if claimed; +- form/server validation when client validation is bypassed; +- preview/auth expiration; +- deployment cold start and missing binding/env; +- 404/500 and maintenance/error pages. + +Progressive enhancement means the primary task has a defined baseline, not that every flourish works without JavaScript. State the baseline per route. + +The Kaiju depth scene keeps a static image when WebGL initialization fails. This is the correct ownership shape for decorative enhancement: content and calls to action remain Astro HTML. + +## Security and privacy + +Verify: + +- raw/rich content renderer against XSS payloads; +- public forms for CSRF/origin, rate/spam, validation, size, and duplicate policy; +- webhooks for raw-body signature, replay, idempotency, and redacted logs; +- CSP/resource origins, frame policy, referrer and permissions policy; +- external links, iframes, embeds, analytics, and consent; +- cache policy for preview/personalized pages; +- no secrets/PII in HTML, JS, metadata, URLs, generated docs, logs, or source maps. + +Do not count obsolete `X-XSS-Protection` as modern XSS defense. Do not disable CSP because an integration was not inventoried. + +## Visual and interaction quality + +Inspect real content and states at multiple sizes: + +- primary hierarchy and action remain clear; +- no clipping, overlay collision, layout shift, or horizontal page scroll; +- long titles, translated labels, large text, and missing media; +- focus, hover, pressed, selected, disabled, invalid, loading, and error states; +- sticky headers/rails and safe areas; +- readable line length and typography before/after font load; +- light/dark/forced-color consistency; +- reduced-motion replacement; +- touch target and coarse-pointer usability; +- visual fallback for canvas, image, and embed failure. + +Do not use a Mermaid chart when a short sequence/table is clearer. Diagrams should clarify relationships, not decorate documentation. + +## Deployment and operations + +Verify the target adapter and host: + +- build artifact and startup command; +- runtime version and APIs; +- environment/bindings and secret scope; +- static asset, image service, and media storage paths; +- redirects, headers, compression, cache/CDN behavior; +- server island and session support; +- canonical origin and preview domains; +- logs, correlation ids, error monitoring, and redaction; +- health/readiness behavior where applicable. + +`astro dev` or a generic preview does not prove target-host behavior. Run an adapter-equivalent smoke or document the block. + +## Verification matrix + +1. Repository lint/type/content checks and Astro check. +2. Static/server production build and route/artifact inspection. +3. Link, canonical, feed, sitemap, robots, redirect, and OpenAPI validation. +4. Raw HTML/no-JS assertions and hydration diagnostics. +5. Browser keyboard, focus, accessibility-tree, zoom, responsive, touch, reduced-motion tests. +6. Screenshots for happy and failure states at narrow/desktop/stress sizes. +7. Bundle/waterfall/CWV/long-task/resource-lifetime evidence. +8. XSS, form, cache, auth/preview, webhook, and header checks. +9. CMS/API/media/WebGL/third-party failure injection. +10. Target-adapter startup and representative deployed request. + +Report passed, failed, blocked, and not-run separately. Never translate a local build into a deployed, accessible, secure, or performant verdict without corresponding evidence. + +## Sources and freshness + +- Uploaded `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, and `thunderstrike-blog(4).zip`, reviewed 2026-07-17. +- Modern Web Guidance accessibility, forms, security, and deferred-rendering guides retrieved 2026-07-17. +- Astro view-transition accessibility: https://docs.astro.build/en/guides/view-transitions/ (reviewed 2026-07-17). +- Performance thresholds, browser support, and deployment behavior are environment-sensitive. Use the project's declared targets and measured baselines. diff --git a/skills/build-web-apps/references/auth.md b/skills/build-web-apps/references/auth.md index 3321d8f..459492a 100644 --- a/skills/build-web-apps/references/auth.md +++ b/skills/build-web-apps/references/auth.md @@ -1,34 +1,248 @@ -# Better Auth in web applications +# Better Auth in Solid and TanStack Start applications -## One mounted contract +## Contents -Align base URL, base path, issuer, wildcard handler, discovery and metadata -paths, consent route, cookie scope, trusted origins, credentials, and framework -integration. Test both root and mounted paths the application supports. +- Evidence and version gate +- Server/client architecture +- Mount, issuer, cookies, and origins +- Database and schema ownership +- Plugin capability map +- Organizations and authorization +- Passkeys, social providers, and magic links +- OAuth provider and consent +- Polar/billing boundary +- Construction and resource lifetime +- Failure signatures +- Verification +- Sources and freshness -## Plugin symmetry +## Evidence and version gate -When server plugins require browser companions, configure and verify the paired -client plugins. Organization, passkey, OAuth provider, and framework cookie -integrations can have distinct server and client packages. Inspect the actual -Solid/TanStack bindings. +The uploaded Kaiju repository uses Better Auth 1.6.x with separate packages/plugins for API keys, OAuth provider, passkeys, organization, JWT, magic link, Drizzle, TanStack Start cookies, and Polar webhooks. Exact options and import paths are versioned. Inspect the installed lockfile, official docs, and plugin types before copying configuration. -## Authorization +The attached `better-auth.zip` is a user application/integration repository, not the upstream Better Auth monorepo. Its patterns are evidence of one composition, not proof of all Better Auth behavior. -Authentication establishes identity. It does not establish access to the active -organization or requested record. Resolve organization context deliberately and -enforce server-side policy or base filters on every protected query. +## Server/client architecture -Distinguish user-only scopes from organization-bound scopes. Do not trust an -organization identifier from URL or browser state without membership and policy -checks. +Separate four layers: -## Construction and data +```text +validated host environment + -> shared auth policy/options + -> host-specific server plugin (TanStack cookies) + -> one Better Auth server instance + wildcard handler -Keep module imports environment-safe. Construct auth and database resources at -an explicit composition root and share long-lived instances. Align Better Auth's -schema, Drizzle adapter, generated migrations, and application database instance. +shared browser options + -> renderer-specific createAuthClient (Solid) + -> matching client plugins +``` -Do not require email verification until sender, confirmation, expiry, resend, -and failure workflows exist. Test cookies, logout/revocation, session expiry, -cross-tab behavior, and database/provider failures. +Server example from the uploaded architecture: + +```ts +function createFrontendAuth() { + const config = parseAuthConfig(readFrontendAuthEnv()); + const baseOptions = createAuthOptions({ config }); + + return betterAuth({ + ...baseOptions, + plugins: [...baseOptions.plugins, tanstackStartCookies()], + }); +} +``` + +Browser example: + +```ts +import { createAuthClient } from "@identity/auth/solid"; +import { createAuthClientOptions } from "@identity/auth/client"; + +export const authClient = createAuthClient(createAuthClientOptions()); +``` + +Do not import `better-auth/react` in a Solid app. Do not configure a server plugin without its required browser client plugin. + +## Mount, issuer, cookies, and origins + +Align one contract: + +- canonical external `baseURL`; +- normalized `basePath`, commonly `/api/auth`; +- wildcard route mounted at that path; +- reverse proxy forwarded host/protocol behavior; +- cookie domain/path/secure/same-site settings; +- trusted origins; +- OAuth callback URLs; +- passkey RP ID and origin; +- discovery/authorization metadata paths; +- client `basePath`. + +The uploaded TanStack handler dynamically imports auth and forwards both GET and POST: + +```ts +async function handleAuthRequest(request: Request): Promise<Response> { + const { getAuth } = await import("#/lib/auth.ts"); + return getAuth().handler(request); +} +``` + +The route must preserve the original request URL, method, body, headers, and cookies. Test root and prefixed deployments through the real proxy/CDN adapter. + +Do not concatenate metadata paths by intuition. The uploaded shared route helpers distinguish OIDC discovery beneath an issuer path from RFC 8414 authorization metadata path construction. + +Trusted origins are an allowlist, not a CORS wildcard. Parse, normalize, and validate environment lists. Include production site/app origins and intentional local development origins only. + +## Database and schema ownership + +The server uses `drizzleAdapter(db, { provider: "pg", schema })`. Keep Better Auth schema tables, relations, migration generation/application, and runtime database instance aligned. + +Decide: + +- whether Better Auth and application tables share a database/schema; +- migration owner and generated-artifact review; +- database pool lifetime; +- ID generation policy; +- transaction behavior; +- session cleanup/retention; +- organization/membership uniqueness; +- plugin table migrations; +- test database isolation. + +Adding a plugin can add or change schema requirements. Run the supported schema/migration workflow and review the diff before deployment. A plugin appearing in the options does not prove its table exists. + +Do not construct a second pool inside auth when the host already owns a database. The uploaded `createAuthOptions` accepts an optional database for this reason. + +## Plugin capability map + +| Capability | Server owner in uploaded code | Browser counterpart / boundary | +|---|---|---| +| email/password | core `emailAndPassword` | core client actions | +| sessions | core | `getSession`/session client | +| magic link | `magicLink` | `magicLinkClient` | +| organizations | `organization` | `organizationClient` | +| passkeys | `@better-auth/passkey` | `passkeyClient` | +| OAuth/OIDC provider | `@better-auth/oauth-provider` | `oauthProviderClient` | +| JWT | `better-auth/plugins/jwt` | inspect whether browser extension is required | +| API keys | `@better-auth/api-key` | server/API management surface | +| TanStack cookies | `tanstackStartCookies` | server-framework integration | +| Polar | `@polar-sh/better-auth` webhooks only in uploaded policy | billing service owns organization checkout/usage | + +Client plugin order generally matters less than symmetry, but preserve upstream guidance and test generated methods. Type presence is not runtime proof. + +## Organizations and authorization + +Authentication identifies a user. It does not authorize organization data. + +For every protected server function/request: + +1. resolve the current session from request headers/cookies; +2. resolve active organization deliberately; +3. verify membership and required role/permission; +4. apply organization scope as a server-owned query predicate; +5. avoid trusting organization ID from URL/body alone; +6. return stable forbidden/not-found semantics without leaking existence. + +Define behavior for no organization, one organization, many organizations, removed membership, disabled organization, invitation pending/expired, and active organization switching. + +The uploaded OAuth policy distinguishes user-only scopes (`openid`, `profile`, `email`, `offline_access`) from organization-bound scopes. Organization scopes require explicit organization selection and bind consent to an active organization reference. Preserve that invariant across UI, provider callbacks, token claims, and API authorization. + +## Passkeys, social providers, and magic links + +Passkeys require exact WebAuthn identity: + +- `rpID` matches the registrable host policy; +- `origin` matches browser-facing origin including scheme/port; +- `rpName` is user-facing; +- proxy/public URL configuration is correct; +- local development and production credentials do not collide accidentally; +- user verification/resident-key policy is understood; +- registration, sign-in, rename/list/delete, and lost-device recovery exist. + +Social providers require complete client ID/secret pairs. The uploaded Zod config rejects half-configured GitHub or Google credentials. Register exact callback URLs for each environment and protect state/PKCE behavior through the upstream integration. + +Magic links require a real sender, expiry, single-use behavior, redirect allowlist, resend/rate limiting, and enumeration-resistant responses. The uploaded code currently logs the email and URL with `console.info`; that is development scaffolding and leaks a bearer credential. Do not retain it in production and do not use console output where the repository requires LogTape. + +Do not set `requireEmailVerification: true` until delivery, confirmation, expiry, resend, error, and support workflows work end to end. The uploaded policy intentionally keeps it false because those pieces are absent. + +## OAuth provider and consent + +If the application itself acts as an OAuth/OIDC provider, define: + +- issuer and metadata endpoints; +- authorization/token/userinfo/revocation behavior; +- client registration policy; +- authenticated versus unauthenticated dynamic registration; +- exact scopes and audiences; +- user versus organization subject/reference; +- account and organization selection; +- consent storage/revocation; +- redirect URI validation; +- key rotation and token lifetime; +- disabled/conflicting core paths. + +The uploaded policy disables Better Auth's built-in `/token` path because the OAuth provider plugin owns it. Keep one route owner. + +Dynamic unauthenticated client registration is a security/product decision, not a convenient development default. The uploaded defaults allow dynamic registration but disallow unauthenticated registration; review before exposure. + +Test metadata documents and a complete authorization-code flow with user-only and organization-bound scopes. Test consent denial, organization switch, invalid audience, revoked membership, redirect mismatch, token refresh, and revocation. + +## Polar/billing boundary + +The uploaded architecture deliberately limits the Better Auth Polar plugin to webhooks. Its product billing is organization-scoped, while generic Better Auth Polar checkout helpers can be user-scoped. Creating checkout/customer records through both paths could create duplicate customer authority. + +Choose one billing customer identity: + +- Better Auth user ID for personal products; or +- organization ID for organization products. + +Keep checkout creation, portal, usage metering, entitlements, webhook projection, retry/idempotency, and reconciliation in the billing service when organization-scoped. Better Auth may still authenticate the caller and receive verified webhooks. + +Webhook verification must use raw request bytes as required by the provider, replay protection/idempotency, stable event identity, and durable processing. Do not grant entitlements solely from the checkout success-page query string. + +## Construction and resource lifetime + +Avoid environment reads, database connections, and global logger configuration at package import. Build config through a Zod schema, then construct at the host composition root. + +The uploaded frontend lazily memoizes `getAuth()` so public demo routes do not require auth/database configuration merely because the route tree imports account pages. This is useful when public and authenticated surfaces share a deployment, but the database/client shutdown owner must remain reachable for tests and graceful termination. + +Forward request headers into `auth.api.getSession({ headers })`. Do not fabricate a browser session from client state in server loaders. + +Cache auth instances only at a scope safe for the deployment runtime. Hot-reload, serverless isolate lifetime, per-request secrets, tenant-specific config, and test isolation can change the correct scope. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| session works locally, absent behind proxy | base URL/cookie secure/domain/forwarded headers | request/response cookie trace | +| browser method missing | client plugin not paired | server/client plugin lists and renderer import | +| passkey says RP/origin mismatch | public URL, RP ID, or port differs | browser origin and parsed config | +| OAuth discovery 404 | issuer/base path/wildcard route mismatch | route helpers and mounted handler | +| organization data crosses accounts | membership not enforced in service query | direct cross-org server test | +| every email sign-in blocked | verification required without workflow | sender/routes/config | +| magic link appears in logs | development callback retained | logging and secret-redaction audit | +| migration succeeds but plugin fails | plugin tables absent/schema drift | generated schema diff and DB inspection | +| duplicate Polar customers | user-scoped plugin plus org-scoped billing service | customer external IDs and checkout owners | +| public route requires DB env | eager auth import/construction | import graph and lazy composition | +| React auth hook in Solid app | renderer binding mismatch | client import and TS config | + +## Verification + +1. Parse empty, partial, local, preview, and production environment fixtures. +2. Prove incomplete social credentials fail validation. +3. Apply auth/plugin migrations to an empty and upgraded PostgreSQL database. +4. Test wildcard GET/POST handler at direct and proxied base paths. +5. Test sign-up/sign-in/sign-out/session expiry/revocation/cross-tab behavior. +6. Test organization create/invite/accept/select/remove and cross-org denial. +7. Test passkey registration/sign-in/recovery on real origins. +8. Test OAuth metadata, user-only scopes, organization scopes, consent, audiences, refresh, and revocation. +9. Test magic-link expiry/single use/rate limit without logging tokens. +10. Test Polar webhook signature, replay, out-of-order delivery, reconciliation, and organization identity. +11. Import public route modules without auth/database environment. +12. Verify production SSR/deployment adapter cookies and graceful database shutdown. + +## Sources and freshness + +- Primary: [Better Auth documentation](https://www.better-auth.com/docs/), verified 2026-07-17 for public server/client, adapter, session, organization, and plugin concepts. +- Attachments: `better-auth.zip` and `kaiju-site-scope(17).zip/apps/frontend`, inspected 2026-07-17 for one Better Auth 1.6.x composition and Solid/TanStack consumption. + +The attachment is a user integration, not upstream Better Auth. Plugin option names, generated schema, cookie adapters, Polar behavior, and import paths are version-sensitive; unobserved private APIs remain unverified. diff --git a/skills/build-web-apps/references/data-views.md b/skills/build-web-apps/references/data-views.md index 33521b7..90a3099 100644 --- a/skills/build-web-apps/references/data-views.md +++ b/skills/build-web-apps/references/data-views.md @@ -1,34 +1,260 @@ -# Tables, lists, and virtualized views +# Data views, URL state, tables, virtualization, and selection -## Ownership +Use this reference for search, filters, sorting, faceting, pagination, tables, cards, charts, infinite lists, virtualized views, bulk selection, and mutations. Separate domain/query semantics from presentation state. -Decide which sorting, filtering, grouping, pagination, and column state is -shareable in the URL, executed by the server, cached remotely, or local to the -view. Large data sets normally keep filtering/sorting/pagination server-owned; -the table library owns presentation state, not database semantics. +## Contents -Use stable domain row identity. Array index is not a durable selection key across -sort, pagination, refresh, insertion, or virtualization. +- Evidence and ownership inventory +- State ownership model +- Validated URL and canonical query identity +- Query loading and mutation contracts +- Table and list semantics +- Selection and bulk action policy +- Pagination, infinite loading, and virtualization +- Responsive and accessible data presentation +- Failure signatures +- Verification +- Sources and freshness -## Selection +## Evidence and ownership inventory -Define whether selection applies only to loaded rows, the current query, or an -explicit all-matching set with exclusions. Keep selection policy separate from -checkbox rendering and verify organization/tenant boundaries on bulk actions. +Inspect: -## Virtualization +- route search schema, defaults, canonicalization, and `loaderDeps`; +- query-options/key factory, loader prefetch, component observer, stale/cache policy; +- server function/repository schema and tenant scope; +- sorting/filtering/pagination execution owner; +- stable row id and revision/freshness metadata; +- table, virtualizer, chart, drag/drop, and component versions; +- selection model and bulk authorization; +- loading/pending/stale/empty/partial/error states; +- mutation invalidation/rollback; +- SSR/hydration and scroll/focus restoration; +- large-data and accessibility tests. -Virtualization changes DOM presence, measurement, focus, screen-reader, and SSR -behavior. Use it only after measuring the need. Define estimated/dynamic size, -overscan, scroll restoration, sticky headers, resize handling, and hydration. +## State ownership model -Do not make keyboard focus disappear when a row leaves the rendered window. -Provide accessible row/column relationships and a non-virtualized or paginated -fallback where the product requires it. +| State | Owner | Examples | +|---|---|---| +| Shareable/navigation | Validated URL | query, filters, sort, page, page size, view mode | +| Remote/server | Query cache/route loader | results, facets, totals, revision | +| Local view | Component/table | open panel, hovered row, draft query, column resizing | +| Selection | Explicit policy store | selected ids/exclusions/scope | +| Session preference | Persistent app preference | density, optional column visibility | +| Authority | Server | tenant, allowed fields/actions, total matching set | + +Do not put modal open, hover, every keystroke, or pending animation into the URL. Do not duplicate remote results into a global store without an offline/editing contract. + +Large datasets normally execute sort/filter/page on the server. A headless table owns column/header/row presentation state; it does not make database ordering authoritative. + +## Validated URL and canonical query identity + +The Kaiju product search route demonstrates the intended chain: + +```text +URL search + -> Zod validation/defaults + -> strip canonical defaults + -> loader dependencies + -> queryOptions(organization + validated search) + -> loader ensureQueryData + -> component observes same query +``` + +```ts +const searchSchema = z.object({ + q: z.string().catch(""), + technologies: z.array(z.string().min(1)).catch([]), + page: z.number().int().positive().catch(1), + page_size: z.number().int().min(10).max(50).catch(20), + sort: z.enum(["relevance", "recent", "company_asc"]).catch("relevance"), +}); +``` + +Use the installed router's adapter/API when required; direct Zod v4 support and middleware APIs are version-sensitive. + +Define dependent reset rules: + +- query/filter/sort/page-size change resets page to 1; +- page change preserves filters; +- clearing a filter removes/canonicalizes the URL value; +- invalid values parse to safe defaults or a deliberate error; +- default values can be stripped from canonical URLs; +- arrays have stable ordering/encoding when identity depends on them. + +Canonical query identity must include every input that changes results, including organization/tenant: + +```ts +function resultsQuery(input: { organizationId: string; search: Search }) { + return queryOptions({ + queryKey: ["lead-search", input.organizationId, input.search], + queryFn: () => searchLeads({ data: input.search }), + }); +} +``` + +The uploaded source expands fields individually in the key, which is also valid if kept complete and stable. Loader, component, prefetch, and invalidation must call the same factory. + +## Query loading and mutation contracts + +Distinguish: + +- initial pending with no data; +- background fetching with usable stale data; +- empty successful result; +- partial result/facets; +- retryable transport failure; +- invalid/forbidden request; +- stale revision and refresh; +- mutation pending/conflict/rollback. + +Do not render `query.data!` unless loader/query integration guarantees it on every path and error/pending boundaries cover failures. TanStack router-query SSR integrations differ by framework/version. Current official guidance distinguishes server-executed suspense/loader prefetch from client-only plain queries; verify Solid package behavior. + +For mutations invalidate the exact affected keys. Adding items to a saved list should not refetch unrelated search results unless server facts changed. Use stable key helpers for targeted invalidation. + +## Table and list semantics + +Choose presentation from user task: + +- semantic table for comparison across consistent columns; +- cards/list for heterogeneous summary and narrow layouts; +- description list for key/value record details; +- chart plus table/text alternative for patterns over numbers; +- tree/grid only when their richer keyboard models are implemented. + +Use stable domain row identity. Array index breaks selection, row expansion, animations, drag/drop, and virtualizer measurement when rows reorder. + +Table requirements: + +- `<caption>` or equivalent contextual name; +- `<th scope="col">`/row headers and correct header groups; +- sorting control announces state with `aria-sort` on the current header; +- numeric alignment and machine-readable values where useful; +- overflow container that does not hide focus; +- sticky/pinned columns with correct overlap/background/z-index; +- resizers with keyboard/accessibility policy if user-operable; +- empty/loading/error rows use correct `colSpan` and status semantics. + +Do not add `role="grid"` to a native table unless implementing the grid keyboard/focus model. + +## Selection and bulk action policy + +Define selection scope explicitly: + +1. Loaded rows only. +2. Current page. +3. Manually selected stable ids across pages. +4. All rows matching the current query, represented as query identity plus exclusions. + +Never infer all-matching selection from a header checkbox alone. + +```ts +type Selection = + | { kind: "ids"; ids: ReadonlySet<string> } + | { kind: "all-matching"; query: SearchIdentity; excluded: ReadonlySet<string> }; +``` + +Bulk request sends an explicit versioned policy. The server revalidates tenant, current query, record eligibility, limits, and actor authorization. Counts shown to the user must correspond to the same scope. + +The Kaiju search UI keeps selected company ids local and labels its checkbox “Select page,” which honestly describes current-page behavior. Selection persists across page changes because ids are stored; product policy should decide whether that is intended and show scope/count clearly. + +## Pagination, infinite loading, and virtualization + +Pagination contract: + +- stable deterministic sort with tie-breaker; +- one-based/zero-based policy mapped once; +- total/page-count semantics, including zero results; +- page out-of-range behavior after data changes; +- cursor versus offset ownership; +- URL/back/forward/scroll restoration; +- server-enforced page-size limits. + +Infinite loading adds concurrency and completion: + +- `hasMore`, fetching state, cursor identity, retry; +- prevent repeated fetch at threshold; +- deduplicate records across pages; +- preserve ordering when response arrival is reordered; +- announce load result without noisy per-scroll updates; +- provide a reachable non-infinite/paginated alternative where required. + +TanStack Table does not include virtualization itself; pair it with TanStack Virtual or another virtualizer. The finance upload's table: + +- separates top-pinned, center, and bottom-pinned rows; +- virtualizes center rows; +- uses leading/trailing spacer rows; +- resolves scroll container; +- supports custom stable item keys and row measurement; +- exposes overscan/estimate size; +- triggers fetch-more near the final virtual item; +- renders loading/completion rows. + +That is a capable implementation shape, not proof of every accessibility behavior. + +Virtualization contract: + +- measured need and dataset size; +- stable `getItemKey` from row id, never index fallback when order changes; +- fixed or dynamic measurement and resize invalidation; +- overscan based on interaction/focus, not arbitrary large values; +- scroll element and nested scroll containers; +- pinned rows/headers outside the virtual range; +- SSR initial range and hydration; +- focus when row leaves rendered window; +- screen-reader semantics/count/position; +- find-in-page/export/print alternative; +- memory/DOM bound and fetch threshold. + +## Responsive and accessible data presentation + +Do not hide essential columns at narrow widths without an alternate path. Options: + +- horizontal scroll with clear affordance and sticky primary column; +- responsive card/details presentation from the same row view model; +- user-configurable columns persisted as a preference; +- priority columns plus an accessible row-details disclosure. + +Keep the DOM/source order meaningful. Test zoom, long content, localization, forced colors, focus across sticky elements, and touch resizing/dragging. + +Charts require title/description, textual summary, and tabular or downloadable data when users need exact values. Do not encode categories only by color. + +Loading skeletons should preserve expected geometry and not claim data. Empty states distinguish no records, no filter matches, forbidden data, and unavailable service. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Deep link refetches different data | URL schema/query key mismatch | Validated search and key factory | +| Filter resets unrelated state | URL patch replaces instead of merges | Navigation update function | +| Page change loses filters | Search params not retained | Functional search update | +| Cross-org cache leak | Tenant omitted from query key/server scope | Key and repository policy | +| Selection moves to another row | Index identity | Row id/getItemKey | +| “All selected” only affects visible rows | Scope ambiguous | Selection union/request contract | +| Infinite loader calls repeatedly | Threshold/fetch state race | last item, `isFetchingMore`, cursor | +| Virtual row overlaps/jumps | Wrong estimate/measurement/scroll element | Virtualizer config | +| Focus disappears during scroll | Focused row unmounted | Overscan/focus/alternative policy | +| Empty screen during background fetch | Initial and stale-fetch states collapsed | Query status model | +| Sort looks active but server ignores it | Table presentation disconnected from query | URL/repository sort mapping | ## Verification -Test stable sorting and pagination, row reordering, selection across refresh and -pages, empty/loading/error states, bulk authorization, keyboard navigation, -screen-reader structure, resize, zoom, large data, scroll restoration, SSR, and -memory/DOM bounds. +1. Copy/reload/back/forward URLs for every filter/sort/page combination. +2. Invalid/default URL canonicalization and dependent page reset. +3. Assert loader/component/invalidation keys are identical and tenant-scoped. +4. Test initial, stale, empty, partial, error, retry, offline, and refresh states. +5. Stable ordering/tie-breakers across insertion/deletion and pagination. +6. Selection across sort, refresh, pages, deletion, and tenant authorization. +7. Virtualizer stable keys, measurement, resize, pinned rows, dynamic height, SSR, and scroll restore. +8. Keyboard/screen-reader table semantics, focus while virtualizing, zoom, responsive overflow, and touch. +9. Infinite loading reordered responses, duplicate data, threshold, completion, and retry. +10. Measure DOM/memory/frame/interaction cost with realistic large data. + +## Sources and freshness + +- Uploaded `kaiju-site-scope(17).zip` TanStack Solid Router/Query search route and UI, reviewed 2026-07-17. +- Uploaded `new-finance-app(1).zip` React table/virtualizer and finance view code, reviewed 2026-07-17. +- TanStack Router search params: https://tanstack.com/router/latest/docs/guide/search-params (reviewed 2026-07-17). +- TanStack Router Query integration: https://tanstack.com/router/latest/docs/integrations/query (reviewed 2026-07-17). +- TanStack Table virtualization: https://tanstack.com/table/latest/docs/guide/virtualization (reviewed 2026-07-17). +- Router, Query, Table, and Virtual APIs vary by framework/version. Inspect installed package docs/types and do not translate React examples mechanically into Solid. diff --git a/skills/build-web-apps/references/forms.md b/skills/build-web-apps/references/forms.md index 1cd933c..b7ac0df 100644 --- a/skills/build-web-apps/references/forms.md +++ b/skills/build-web-apps/references/forms.md @@ -1,36 +1,321 @@ -# Forms and mutations +# Forms, validation, and mutation state machines -## State ownership +Use this reference for native, Astro, React, Solid, TanStack Form, auth, file, payment, settings, and multi-step forms. A form library can own field ergonomics; the server owns trust, authorization, and committed domain state. -Separate input draft, validated client value, committed server request, pending -mutation, authoritative response, and displayed server error. A form library can -own field ergonomics; the server schema and domain service remain authoritative. +## Contents -Use framework-local state for draft interaction. Put state in the URL only when -it is intentionally shareable/navigation state. Do not copy every keystroke into -the remote cache. +- Evidence and ownership inventory +- Form state machine +- Native form foundation +- Validation timing and schemas +- TanStack Form pattern from the finance app +- Async validation and races +- Submission, idempotency, and optimistic state +- Authentication and sensitive forms +- Multi-step, file, and long-lived forms +- Failure signatures +- Verification +- Sources and freshness -## Validation +## Evidence and ownership inventory -Client validation improves feedback but never replaces server validation. Reuse -schema semantics without importing server-only secrets or resources into the -browser. Define async validation cancellation and stale-result behavior. +Inspect: -## Submission races +- form element/action/method/enctype and progressive-enhancement target; +- renderer and hydration directive; +- field/form library plus exact version; +- default values and source; +- client and server schemas, transforms, cross-field rules; +- mutation endpoint/service and authorization; +- pending, retry, duplicate, idempotency, and navigation behavior; +- error mapping, focus, live announcements, and PII logging; +- auth/payment/provider redirects and callback allowlist; +- file size/type/storage pipeline; +- tests for invalid, slow, duplicate, offline, server-rejected, and expired-session cases. -Prevent double submission, out-of-order responses, stale validation, route -navigation during submit, and lost server errors. Decide whether later submits -cancel, supersede, queue, or conflict with earlier ones. +Write an ownership table: -## Mutations +| Concern | Owner | +|---|---| +| Draft characters/touched/dirty | Field/form state | +| Shareable step/filter | Validated URL if intentionally navigable | +| Client feedback | Native constraints plus client schema | +| Trust and domain invariants | Server schema/service | +| Session and tenant authority | Server request boundary | +| Pending request | Mutation/form controller | +| Committed record | Server/database/provider | +| Remote cache | Query cache and invalidation policy | +| Public error | Endpoint contract | +| Diagnostic cause | Redacted server logging | -Define query invalidation from the same identity factory used for reads. Use -optimistic updates only when rollback and concurrent server truth are clear. -Capture prior cache state, reconcile the authoritative response, and surface -partial or conflicting failures. +Do not copy every keystroke into a query cache or global store. Do not use the URL for passwords, tokens, private drafts, or transient validation. + +## Form state machine + +Model the phases explicitly: + +```text +pristine + -> editing + -> locally invalid / locally valid + -> submitting(request id) + -> server validation rejected + -> authorization/conflict rejected + -> committed(authoritative response) + -> retryable transport failure + -> cancelled/navigated +``` + +Track draft, validation, pending request, and authoritative result separately. A server validation error applies to the values submitted, not necessarily the user's newer draft. Attach request identity or submitted snapshot before displaying an async result. + +Define whether a new submission: + +- is ignored while one is pending; +- cancels/supersedes the prior request; +- queues behind it; +- creates a new idempotent attempt; +- conflicts and asks the user to reconcile. + +## Native form foundation + +Start with native semantics: + +```html +<form method="post" action="/account/profile"> + <div> + <label for="display-name">Display name</label> + <p id="display-name-hint">Shown to members of your organization.</p> + <input + id="display-name" + name="displayName" + autocomplete="name" + aria-describedby="display-name-hint display-name-error" + required + maxlength="80" + > + <p id="display-name-error"></p> + </div> + <button type="submit">Save profile</button> +</form> +``` + +Use: + +- explicit label association; +- correct input type, `autocomplete`, `inputmode`, `enterkeyhint`, min/max/length/pattern where semantics match; +- `<fieldset><legend>` for grouped choices; +- persistent instructions separate from labels/placeholders; +- native submit by Enter; +- buttons with explicit type; +- `FormData` names that match server schema; +- server fallback when progressive enhancement is claimed. + +Client validation improves UX but is not a security boundary. Native constraints and client schemas may be bypassed. + +Do not validate aggressively on each keystroke. Clear stale errors during input; validate once the user leaves a field or submits according to product policy. Do not disable an initially invalid submit button so thoroughly that users cannot trigger discoverable validation. + +## Validation timing and schemas + +Separate: + +- parse: string/`FormData` to typed candidate; +- field constraints: email/length/range; +- cross-field constraints: confirmation/date ordering; +- async facts: username availability/coupon/provider; +- domain invariants: authorization, balance, uniqueness, state transition. + +| Event | Recommended default | Reason | +|---|---|---| +| input/change | Clear stale field error; cheap validation only if helpful | Avoid premature noise | +| blur | Validate touched field | User indicated completion | +| submit | Validate entire form and focus summary/first invalid field | Final client gate | +| server | Reparse and enforce every trusted invariant | Client is untrusted | + +Keep the schema in a browser-safe shared module only if it imports no secret/server resource. “Shared TypeScript” does not guarantee client safety. Server and client can use related schemas with deliberate transforms: + +```ts +const clientProfile = z.object({ + displayName: z.string().trim().min(1).max(80), +}); + +const serverProfile = clientProfile.extend({ + // Derived from session, never accepted from the browser. + actor: authorizedActorSchema, +}); +``` + +Do not add `organizationId` to a public form schema if the server can derive the active organization. + +## TanStack Form pattern from the finance app + +The uploaded finance app uses `@tanstack/react-form` 1.33 with Zod via form-level `onBlur` and `onSubmit` validators. Better Auth remains the request owner. + +```tsx +const signInSchema = z.object({ + email: z.email({ error: "Enter a valid email address." }).trim(), + password: z.string().min(1, { error: "Enter your password." }), +}); + +const form = useForm({ + defaultValues: { email: "", password: "" }, + validators: { onBlur: signInSchema, onSubmit: signInSchema }, + onSubmit: async ({ value }) => { + const result = await signIn.email({ ...value, callbackURL: "/" }); + if (result.error) setFormError(toPublicMessage(result.error)); + }, +}); +``` + +Fields preserve native labels/autocomplete and show errors after touch: + +```tsx +<form.Field name="email"> + {(field) => { + const invalid = field.state.meta.isTouched && !field.state.meta.isValid; + return ( + <Field data-invalid={invalid}> + <FieldLabel htmlFor={field.name}>Email</FieldLabel> + <Input + id={field.name} + name={field.name} + type="email" + autoComplete="email" + value={field.state.value} + onBlur={field.handleBlur} + onChange={(event) => field.handleChange(event.target.value)} + aria-invalid={invalid} + required + /> + {invalid && <FieldError errors={field.state.meta.errors} />} + </Field> + ); + }} +</form.Field> +``` + +This is observed version-specific source, not a guarantee for a later TanStack Form API. Verify installed docs/types. Improve it by ensuring form-level errors are announced/focused appropriately and provider error messages are mapped to stable safe copy rather than blindly shown. + +Alternative auth actions use separate pending method state so social/passkey interactions cannot overlap email submit. Browser passkey capability checks improve UX; the server still verifies credentials and policy. + +## Async validation and races + +Debounce is not cancellation. Use a sequence or `AbortController`: + +```ts +let validationSequence = 0; +let validationController: AbortController | undefined; + +async function validateHandle(handle: string) { + const sequence = ++validationSequence; + validationController?.abort(); + validationController = new AbortController(); + + const result = await checkHandle(handle, validationController.signal); + if (sequence !== validationSequence) return; + applyValidationResult(handle, result); +} +``` + +Define aborted request behavior and ignore results that do not match the current field value. Avoid async validation for facts that can be checked only atomically at commit time; uniqueness checks can improve feedback but the final write must handle a race/conflict. + +For dependent fields, clear or revalidate only the errors affected by the dependency. Do not run whole-form remote validation on every keystroke. + +## Submission, idempotency, and optimistic state + +Once valid submission begins: + +- prevent duplicate client actions while preserving progress/status; +- snapshot submitted values; +- attach idempotency key when retried side effects require it; +- keep navigation/close policy explicit; +- handle 401/403/409/422/429/5xx distinctly; +- map field errors by stable field paths; +- focus error summary or first invalid field; +- reconcile authoritative response and invalidate exact query keys. + +Optimistic updates are appropriate only if rollback and concurrency are clear: + +```text +onMutate + -> cancel relevant query + -> snapshot prior cache + -> apply optimistic value + +onError + -> restore snapshot + -> show safe actionable error + +onSuccess/onSettled + -> replace/invalidate with authoritative server state +``` + +Do not optimistically claim payment, permission, identity, or irreversible workflow completion. + +Client disabled state is not idempotency. The server/database/provider needs a duplicate contract. + +## Authentication and sensitive forms + +Use appropriate autocomplete: + +- sign in email/username: `username` or email according to account model; +- sign in password: `current-password`; +- registration/reset password: `new-password`; +- one-time code: `one-time-code` where supported; +- name/address/payment fields: standard tokens. + +Allow password-manager paste. Provide show/hide behavior with clear accessible state and privacy warning when appropriate. Do not log passwords, codes, tokens, recovery keys, complete auth errors, or raw form payloads. + +Callback/redirect URLs require allowlisting. Social/passkey/provider availability should be derived by the server and passed as client-safe capability identifiers. An imported client auth module must not require server environment values. + +Sensitive settings need re-authentication/fresh-session policy, CSRF/origin defense, server authorization, session invalidation, and audit. + +## Multi-step, file, and long-lived forms + +For multi-step forms define: + +- canonical full schema versus per-step schema; +- URL step only if safe/shareable; +- back/forward and saved-draft policy; +- server draft identity and ownership; +- partial validation versus final commit; +- resume expiration/version migration; +- final review and irreversible action. + +For files define size/count/type/content validation, progress/cancel, retry/resume, temporary object cleanup, storage authorization, virus/content processing where required, and accessible status. + +For long-lived drafts handle server version conflicts. Do not overwrite a changed record silently; use version/etag/revision and an explicit merge/reload decision. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Error appears for old field value | Async result not sequenced | Request/value identity | +| Enter does not submit | Non-native wrapper/key handler | Form/button semantics | +| Double click creates records | No server idempotency | Mutation/service/database | +| Server field errors disappear immediately | Draft and submitted snapshot collapsed | Error ownership | +| Error visible but screen reader silent | Missing association/focus/status | Field error ids/summary | +| Social and email submit overlap | Separate pending paths not coordinated | Form/method state machine | +| Client bundle requests server secrets | Shared schema/auth imports server module | Import graph | +| Optimistic success stays after rejection | Rollback/invalidation missing | Mutation callbacks/query key | +| Back loses multi-step progress | Ownership not durable/shareable | URL/server draft policy | +| File upload succeeds but private file is public | Storage authorization mismatch | Upload/serve boundary | ## Verification -Test keyboard submit, invalid and corrected fields, slow async validation, -double click, reordered responses, network loss, server validation, optimistic -rollback, navigation, focus on errors, and accessible status announcements. +1. Submit with keyboard, pointer, touch, and Enter. +2. Test native constraints, touched/blur validation, submit validation, and corrected fields. +3. Bypass client validation and test server parsing/authorization. +4. Test slow/reordered/aborted async validation. +5. Double click/submit, retry after timeout, and idempotency. +6. Navigate/close during pending request and return with back/forward. +7. Test 401, 403, 409, validation, rate limit, network loss, and 500. +8. Verify error association, focus, live announcement, autocomplete, and password manager behavior. +9. Verify exact query invalidation/rollback and authoritative response. +10. Search logs/HTML/bundles/URLs for sensitive values. + +## Sources and freshness + +- Uploaded `new-finance-app(1).zip` and `old-finance-app(1).zip` auth forms, Astro shells, TanStack Form 1.33 usage, and auth capability boundary, reviewed 2026-07-17. +- TanStack Form validation docs: https://tanstack.com/form/latest/docs/framework/react/guides/validation (reviewed 2026-07-17). +- Modern Web Guidance forms/accessibility/security guides retrieved 2026-07-17. +- TanStack Form, Astro navigation, Better Auth, and passkey APIs are version-sensitive. Verify installed package APIs and provider policy. diff --git a/skills/build-web-apps/references/solid.md b/skills/build-web-apps/references/solid.md index 3e87f99..f643be7 100644 --- a/skills/build-web-apps/references/solid.md +++ b/skills/build-web-apps/references/solid.md @@ -1,31 +1,189 @@ -# Solid application ownership +# Solid and Solid Primitives -## Reactive values +## Contents -- Signals own mutable local state. -- Memos own derived values. -- Effects synchronize with external systems; they are not a default derivation - mechanism. -- Props are live access paths and should not be destructured casually. -- Control-flow components preserve Solid's ownership semantics better than - React-shaped list and conditional translations. +- Reactive ownership +- Components and control flow +- Effects and external systems +- SSR and hydration +- Cleanup and roots +- Solid Primitives ecosystem map +- Selection procedure +- Scheduling and global event coordination +- Motion and presence boundary +- Failure signatures +- Verification +- Sources and freshness -## Solid Primitives +## Reactive ownership -Before writing browser, scheduling, event, media, observer, root, props, or -transition utilities, inspect the relevant Solid Primitives package. Check its -maturity stage, server behavior, sibling dependencies, `make*` versus `create*` -API, owner cleanup, and tests. +Solid executes a component function once to create a reactive graph. It does not rerun the component on every state change. Preserve accessors and ownership: -Examples of distinct concerns include event listeners, scheduled debounce and -throttle, media queries, rootless subroots/singletons, ref composition, prop -merging, and transition groups. Do not install the whole ecosystem by default. +| Need | Owner | Avoid | +|---|---|---| +| mutable scalar/local value | `createSignal` | mutable untracked local variable | +| derived value | `createMemo` or inline accessor | effect that writes another signal without need | +| list keyed by identity | `<For>` / keyed primitive when necessary | React-shaped `.map()` assumptions | +| index-keyed list | `<Index>` when values change but positions are stable | using it for reordered identity lists | +| conditional subtree | `<Show>`, `<Switch>/<Match>` | eager branch evaluation | +| async resource | resource/query owner selected from data semantics | duplicate signal plus fetch effect | +| external synchronization | `createEffect`/`onMount` with cleanup | effects as general computation | -## SSR and cleanup +Props are live access paths. Do not destructure reactive props casually: -Gate DOM and layout behavior behind mount. Keep initial server/client output -deterministic. Dispose listeners, observers, frames, timers, subscriptions, -workers, animation values, and retained roots on owner cleanup. +```tsx +function Counter(props: { count: number }) { + const doubled = createMemo(() => props.count * 2); + return <output>{doubled()}</output>; +} +``` -Test mount/unmount counts and repeated route navigation. Type signatures alone -do not prove cancellation or cleanup is wired to the active owner. +Use `splitProps`, `mergeProps`, or relevant `@solid-primitives/props` helpers when adapting component APIs. Verify which properties must remain getters. + +## Components and control flow + +Components own composition, not rerender cycles. Pass children with Solid's `children()` helper when access may need normalization or repeated evaluation. Avoid reading a conditional child before its owner branch exists. + +For lists, make identity explicit. A data grid may need row IDs independent of page position. A keyed wrapper is useful only when the default control flow does not preserve the required reconciliation behavior. + +Events use native event semantics. Prefer delegated JSX handlers for common bubbling events; use explicit event listeners when target, capture/passive options, or non-delegated events require them. + +## Effects and external systems + +Use effects for synchronization with APIs that exist outside Solid: + +- DOM methods not expressible declaratively; +- storage/history synchronization; +- canvas/WebGL/animation runtime updates; +- subscriptions and external event sources; +- imperative third-party widgets. + +Read dependencies intentionally. Use `on(...)` or `untrack(...)` only when the dependency contract demands it. A self-triggering effect usually indicates derived state or ownership is misplaced. + +Every effect that starts a resource needs a teardown or a resource whose primitive registers teardown automatically. + +## SSR and hydration + +Server and first client render must agree. Guard browser globals, measurements, random values, current time, storage, media queries, and feature detection behind an SSR-aware primitive or mount boundary. + +Do not create shared singleton state at module scope in SSR. It can leak one request's state into another. The uploaded Solid Primitives `rootless` package marks hydratable singleton behavior experimental; inspect the installed version before depending on it. + +For viewport/media-dependent UI, render a deterministic usable default and enhance after hydration. Avoid hiding the entire page until `onMount` solely to silence a mismatch. + +Test streaming SSR, direct navigation, client navigation, and repeated route mounting. Development mode may not reproduce production hydration ordering. + +## Cleanup and roots + +Solid ownership automatically disposes registered cleanup when a component/root goes away. It does not clean resources that were created outside the owner or never registered. + +Dispose: + +- listeners and observers; +- intervals/timeouts/idle callbacks; +- animation frames and motion values; +- `AbortController`s and network streams; +- WebSocket/SSE subscriptions; +- workers and message channels; +- external stores and observable subscriptions; +- retained subroots/singletons; +- result objects holding native resources. + +`@solid-primitives/rootless` provides patterns such as `createDisposable`, `createSubRoot`, `createCallback`, and `createSingletonRoot`. Use them only when normal component ownership cannot represent the lifetime. `createDisposable` is appropriate for a resource that must stop before owner cleanup. + +## Solid Primitives ecosystem map + +The uploaded monorepo contains roughly eighty-five focused packages. Treat the repository as an ecosystem and search sibling packages before writing a new primitive. + +| Concern | Candidate packages from uploaded source | Selection question | +|---|---|---| +| events | `event-listener`, `event-bus`, `event-dispatcher`, `event-props` | DOM target, single channel, typed multi-event, or prop adapter? | +| scheduling | `scheduled`, `raf`, `timer`, `idle` | debounce/throttle, one frame loop, timer, or idle work? | +| observers | `intersection-observer`, `mutation-observer`, `resize-observer` | what is the SSR default and cleanup owner? | +| environment | `media`, `page-visibility`, `connectivity`, `platform`, `devices`, `permission` | is the first render deterministic? | +| state/data | `resource`, `promise`, `storage`, `db-store`, `fetch`, `graphql`, `sse`, `websocket` | remote cache, stream, browser store, or one request? | +| composition | `props`, `refs`, `context`, `destructure`, `controlled-props`, `keyed` | preserve accessors and ownership? | +| collections | `list`, `map`, `set`, `pagination`, `virtual` | stable identity, pagination, or DOM bound? | +| roots | `rootless`, `lifecycle` | can normal owner cleanup work instead? | +| motion | `spring`, `tween`, `presence`, `transition-group` | lifecycle, interruption, SSR, and reduced motion? | +| browser capability | `clipboard`, `fullscreen`, `geolocation`, `filesystem`, `workers`, `broadcast-channel` | permission, failure, and server fallback? | + +Package maturity badges matter. The uploaded `scheduled` and `rootless` docs show stage 2 while `event-listener` shows stage 3. Recheck the installed release; maturity can change independently per package. + +## Selection procedure + +For a candidate primitive: + +1. search the monorepo package list and consuming lockfile; +2. inspect the package README, source exports, tests, maturity stage, and version; +3. distinguish `make*` non-reactive setup from `create*` reactive ownership; +4. identify server behavior and initial value; +5. identify owner cleanup and any manual early-dispose API; +6. inspect sibling packages required by the example; +7. test repeated mount/unmount and SSR; +8. import the smallest package, not the entire ecosystem. + +Example event listener: + +```tsx +import { createEventListener } from "@solid-primitives/event-listener"; + +let button!: HTMLButtonElement; +createEventListener(() => button, "click", handleClick, { passive: true }); + +return <button ref={button}>Run</button>; +``` + +The reactive `createEventListener` rebinds when reactive target/type changes and cleans up with the owner. In the uploaded 2.x documentation it does not return an early clear function; use an empty target/type or a disposable subroot when early removal is required. Do not copy the older return contract. + +## Scheduling and global event coordination + +The `scheduled` package provides cancellable `debounce`, `throttle`, idle scheduling, leading/trailing wrappers, and `createScheduled`. Timers clear on owner disposal. + +```ts +import { debounce } from "@solid-primitives/scheduled"; + +const commitSearch = debounce((value: string) => navigate({ search: { q: value } }), 250); +onCleanup(() => commitSearch.clear()); +``` + +For many pointer/scroll/animation consumers, prefer one shared requestAnimationFrame scheduler rather than one loop per component. Define registration, visibility pause, document-hidden pause, elapsed/delta clamping, reduced-motion policy, and last-subscriber teardown. Use `@solid-primitives/raf` only after verifying its API fits that singleton ownership. + +EventBus is appropriate for hot one-to-many notifications. Do not replace routable URL state, remote query state, or durable workflow events with an in-memory bus. + +## Motion and presence boundary + +Treat the attached Solid motion package as experimental. The evidence notes incomplete gesture types and SSR/presence caveats. Use proven Solid Primitives or Web Animations/CSS where they satisfy the behavior. A type-compatible motion prototype is not production parity. + +Presence must preserve exit lifetime, cancellation/interruption, nested ownership, focus, reduced motion, and deterministic server output. Verify rapid enter/exit toggles and route navigation. + +## Failure signatures + +| Signature | Likely defect | Correction | +|---|---|---| +| prop stops updating | reactive prop destructured | retain accessor or use Solid prop helper | +| effect loops | derived state modeled as effect | replace with memo/accessor | +| state leaks between SSR users | module singleton/root | request-local owner or hydratable verified primitive | +| listener count grows after navigation | created outside owner or missing cleanup | primitive/cleanup plus mount-count test | +| debounce fires after unmount | scheduler not owner-bound | clear/register cleanup | +| first paint flashes | browser value differs from SSR default | deterministic initial state and mount enhancement | +| wrong rows retain state | incorrect keyed/indexed control flow | choose stable identity owner | +| virtual list loses keyboard focus | mounted window owns focus incorrectly | focus retention/overscan/fallback | +| animation never exits | subtree removed before presence owner finishes | explicit presence lifecycle | + +## Verification + +- unit-test pure state and schema logic; +- use owner-root tests that dispose and assert listener/timer/subscription counts; +- SSR render and hydrate representative routes; +- navigate repeatedly and compare retained roots/resources; +- test document visibility, reduced motion, offline/permission failure; +- use fake timers only where they preserve scheduler semantics; +- test actual browser layout/observer/RAF behavior; +- run production build and inspect renderer/compiler output. + +## Sources and freshness + +- Primary: [Solid Primitives documentation](https://primitives.solidjs.community/), verified 2026-07-17 for the published ecosystem and package-level guidance. +- Attachments: `solid-primitives(2).zip` and `kaiju-site-scope(17).zip/apps/frontend`, inspected 2026-07-17 for package inventory, maturity stages, ownership, cleanup, SSR, scheduling, and consumer integration. + +Individual primitive APIs and maturity stages are package-version-sensitive. Experimental or stage-listed packages are not stable merely because they appear in the monorepo; verify their package manifest, README, tests, and installed version. diff --git a/skills/build-web-apps/references/tanstack.md b/skills/build-web-apps/references/tanstack.md index a0b06e2..8282861 100644 --- a/skills/build-web-apps/references/tanstack.md +++ b/skills/build-web-apps/references/tanstack.md @@ -1,34 +1,237 @@ -# TanStack application state +# TanStack Start, Router, Query, Form, Table, and Virtual -## Route and URL contract +## Contents -Use a schema for search params. Strip defaults from canonical URLs where the -router supports it, preserve deep-linkable state, and reset pagination or other -dependent values when filters change. Pathless layouts can own authenticated -shells and route context without changing the URL. +- Ecosystem ownership +- Route tree and layouts +- URL/search contract +- Loaders and Query identity +- Server functions +- Mutations and invalidation +- Forms +- Tables and virtualization +- SSR, streaming, and deployment +- Failure signatures +- Verification +- Sources and freshness -## Query contract +## Ecosystem ownership -Centralize query keys and option factories. The route loader and rendered -component must reuse the same identity. Define stale time, garbage collection, -retry, placeholder/previous data, cancellation, and invalidation from product -semantics. +Treat TanStack as a family of complementary packages, not one framework import. -Do not store server-cache data in ad hoc signals. Do not store local row selection -or dialog state in Query. A search draft can remain local until debounced and -committed to validated URL state. +| Package | Owns | Does not own | +|---|---|---| +| Start | full-stack runtime, server functions, SSR/build integration | domain/service architecture | +| Router | route tree, params, search, loaders, navigation context | remote cache contents | +| Query | server-state cache, retries, freshness, invalidation, cancellation | shareable navigation semantics | +| Form | client field/form state and validation ergonomics | server trust/authorization | +| Table | headless column/row/filter/sort/selection model | remote fetching or rendering style | +| Virtual | bounded rendering window and measurement | table semantics/accessibility policy | + +Use the renderer-specific packages (`@tanstack/solid-*` in a Solid app). The attached Better Auth integration includes copied React Table/Form components inside an Astro/React surface; they are not Solid implementation evidence. + +## Route tree and layouts + +Use route groups/pathless layouts to separate organizational source layout from public URL shape. An authenticated shell can live in `_app` without adding `_app` to the URL. + +Define root context once: + +```ts +interface AppRouterContext { + queryClient: QueryClient; + session?: Session; +} + +export const Route = createRootRouteWithContext<AppRouterContext>()({ + component: RootDocument, +}); +``` + +`beforeLoad` can establish redirect/context prerequisites, but server authorization remains required in server functions/services. Preserve the intended destination when redirecting to sign-in and validate it before redirecting back. + +Route file-ignore patterns are architecture. The uploaded Kaiju Vite config excludes component folders, underscore helpers, server files, and tests from route generation. Verify generator output whenever the naming pattern changes. + +## URL/search contract + +Put only shareable/navigable state in search parameters: query, filters, sort, page/cursor when meaningful, view mode when users expect a copied link to retain it. + +Do not put transient hover, open dialog, local row selection, draft keystrokes, secrets, or opaque remote objects in the URL. + +Use a schema as the route boundary: + +```ts +const leadSearchSchema = z.object({ + q: z.string().trim().max(200).catch(""), + technologies: z.array(z.string()).catch([]), + sort: z.enum(["relevance", "name", "updated"]).catch("relevance"), + page: z.coerce.number().int().positive().catch(1), +}); + +export const Route = createFileRoute("/(product)/_app/(search)/search")({ + validateSearch: leadSearchSchema, + search: { + middlewares: [stripSearchParams({ q: "", technologies: [], sort: "relevance", page: 1 })], + }, + loaderDeps: ({ search }) => leadSearchSchema.parse(search), +}); +``` + +The exact `search.middlewares` API is version-specific. The attached source demonstrates `validateSearch`, `stripSearchParams`, and `loaderDeps`; inspect the installed Router release before copying syntax. + +Reset dependent state explicitly: changing a filter should reset page/cursor. Canonicalize empty/default values to avoid equivalent URLs fragmenting cache and analytics. + +## Loaders and Query identity + +Create one option factory per remote operation and reuse it in route loader and component: + +```ts +function leadSearchOptions(input: LeadSearch) { + return queryOptions({ + queryKey: ["leads", "search", input], + queryFn: ({ signal }) => searchLeads({ data: input, signal }), + staleTime: 30_000, + }); +} + +export const Route = createFileRoute("/search")({ + validateSearch: leadSearchSchema, + loaderDeps: ({ search }) => search, + loader: ({ context, deps }) => context.queryClient.ensureQueryData(leadSearchOptions(deps)), +}); +``` + +Check the installed server-function call signature for signal support. The important invariant is identity and cancellation, not this exact example syntax. + +Query keys must contain every value that changes the response, including organization/tenant context, permissions or projection version when the same browser session can switch them. Do not put secrets or unbounded objects in a key. + +Define: + +- stale time from product freshness; +- garbage collection from navigation and memory needs; +- retry by error kind/idempotency; +- cancellation behavior; +- placeholder/previous data versus loading distinction; +- refetch triggers; +- persistence/broadcast owner if added; +- invalidation from mutations. + +Do not mirror query results into signals. Derive presentation through memos/selectors while Query remains the remote-cache owner. ## Server functions -Treat client validation as ergonomics, not trust. Validate again at the server -boundary, authenticate, authorize tenant/organization scope, apply server-owned -filters, and return a stable safe result contract. +A server function is a transport boundary, not a service-module replacement. + +```ts +export const searchLeads = createServerFn({ method: "GET" }) + .inputValidator(leadSearchSchema) + .handler(async ({ data }) => { + const session = await requireSession(); + return leadService.search({ organizationId: session.organizationId, ...data }); + }); +``` + +Exact middleware/input APIs are versioned. Preserve this sequence regardless: + +1. parse input on the server; +2. resolve session; +3. authorize organization/resource; +4. apply server-owned scope; +5. call a reusable service operation; +6. map known failures to a safe stable result; +7. record redacted diagnostics/correlation. + +Never trust the `organizationId`, price, plan, role, or redirect URL supplied by the browser. The attached Kaiju app wraps billing and auth operations in server functions and forwards request headers to Better Auth; inspect that boundary for every mutation. + +## Mutations and invalidation + +Classify mutation behavior: + +- idempotent create/update with a stable request key; +- non-idempotent external side effect; +- optimistic local/cache update with rollback; +- durable background request returning operation status; +- synchronous server transaction. + +Use query-key factories for invalidation so reads and writes agree. Prefer updating a detail cache from the authoritative response and invalidating dependent lists/counts. Avoid “invalidate everything” when it creates expensive refetch storms. + +Optimistic update requires: + +1. cancel relevant in-flight reads; +2. snapshot previous cache; +3. apply a deterministic optimistic value; +4. restore on failure; +5. reconcile server response; +6. refetch when server-side ordering/counts may differ. + +Test two concurrent mutations and response reordering. A last-write UI assumption may not match database conflict semantics. + +## Forms + +TanStack Form can own field state, touched/dirty/pending/error state, and client validators. Keep a server schema and server authorization authoritative. + +Renderer mismatch is a real failure mode: the uploaded finance/Better Auth surface imports `@tanstack/react-form`. Do not copy that into the Solid TanStack Start app; use the installed Solid package and Solid examples. + +Define whether validation occurs on change, blur, submit, and asynchronously. Cancel stale async validators. Map stable server field errors back to fields while retaining a form-level unexpected failure. + +After submit, disable/dedupe or define superseding semantics. Preserve values after recoverable errors and move focus to an error summary/first invalid field accessibly. + +## Tables and virtualization + +Table is headless. The application still owns: + +- column definitions and stable IDs; +- controlled sorting/filtering/pagination; +- remote versus client processing mode; +- row identity and selection across pages; +- visibility/order/pinning persistence; +- empty/loading/error states; +- cell semantics and responsive layout; +- authorization for bulk actions. + +Do not enable both remote sorting and a client sorting model accidentally. Encode server-supported public fields/operators, not arbitrary database identifiers. + +Virtualization is justified by measured DOM/layout cost. It adds dynamic measurement, overscan, scroll restoration, sticky-header, focus, screen-reader, resize, and SSR complexity. A paginated table may be a better product/accessibility tradeoff. + +Keep focused/selected rows reachable. Test zoom, font loading, variable row height, expanded rows, responsive columns, and direct restoration. + +## SSR, streaming, and deployment + +Create one QueryClient per server request and a long-lived browser client. Never share a mutable server cache across users. Align dehydration/hydration and route-loader identity. + +Keep browser-only libraries out of server module initialization. Validate environment separation and adapter support for request headers, cookies, streaming, abort, background work, and native dependencies. + +The attached Kaiju app combines TanStack Solid Start, Nitro Vite, and a target deployment adapter. A successful Vite build does not prove deployed request/cookie/database behavior. Run adapter-equivalent requests. + +## Failure signatures + +| Signature | Likely defect | Evidence | +|---|---|---| +| loader fetches then component fetches again | option/key mismatch | compare factories and hydration state | +| copied URL loses filters | local state owns navigable values | route search schema/navigation | +| cache shows another organization | tenant missing from key or server scope | key plus authorization test | +| page remains high after filter | dependent URL state not reset | search transition test | +| server function can query arbitrary tenant | browser input trusted | cross-organization test | +| optimistic row jumps/disappears | server sort/count differs | reordered/concurrent mutation test | +| React package in Solid file | ecosystem binding copied | manifests/imports/compiler | +| SSR users share cache | process-global QueryClient | concurrent request isolation | +| virtual table loses focus | windowing owns keyboard target badly | keyboard/scroll test | +| auth redirect loops | route guard/session cookie mount mismatch | request trace and redirect destination | + +## Verification + +- generate/inspect route tree and direct-load every protected/public route; +- copy and reload URLs with valid, default, invalid, and legacy search values; +- assert loader/component use one option factory/key; +- test cancellation, retries, offline, stale/placeholder/error states; +- test cross-tenant server-function calls directly; +- test mutation double-submit, reordering, rollback, and invalidation; +- server-render/hydrate with isolated QueryClient instances; +- test table keyboard/selection/pagination/virtualization at representative size; +- run production SSR build and deployment-adapter smoke tests. -Align server-function input, route search schema, query key, and persistence -query semantics. One field should not mean different things at these layers. +## Sources and freshness -## Ecosystem selection +- Primary starting points: [TanStack Start](https://tanstack.com/start/latest), [Router](https://tanstack.com/router/latest), [Query](https://tanstack.com/query/latest), [Form](https://tanstack.com/form/latest), [Table](https://tanstack.com/table/latest), and [Virtual](https://tanstack.com/virtual/latest), checked 2026-07-17 for current product boundaries. +- Attachment: `kaiju-site-scope(17).zip/apps/frontend`, inspected 2026-07-17 for a Solid/Start composition, route structure, query ownership, and auth integration. -Map TanStack Start, Router, Query, Form, Table, and Virtual by responsibility. -Install only the packages required by the application surface and renderer. -Verify Solid-specific APIs rather than copying React examples. +TanStack package signatures and Start deployment behavior evolve quickly and vary by renderer. Exact imports, server-function APIs, generated route behavior, and adapters are version-sensitive; verify the target lockfile and primary docs. diff --git a/skills/build-web-apps/references/verification.md b/skills/build-web-apps/references/verification.md index a291bb4..ec98867 100644 --- a/skills/build-web-apps/references/verification.md +++ b/skills/build-web-apps/references/verification.md @@ -1,34 +1,209 @@ -# Web application verification +# Web application verification manual -## State oracles +Use this reference for stateful SSR applications, route loaders, server functions, forms, auth, tables, virtualization, query caches, mutations, and long-lived browser resources. -- A copied deep link reconstructs filters, sort, page, and view. -- Changing a filter resets dependent pagination predictably. -- Loader and component reuse one query key/options factory. -- Selection and dialog state do not pollute URL or remote cache without reason. -- Invalid URL and direct server-function input fail through the intended schema. -- Stale, loading, empty, partial, and error states remain distinguishable. +## Contents -## Identity and security +- Application contract inventory +- Route and URL oracles +- Query and server-function oracles +- Form and mutation oracles +- Auth and tenant oracles +- Data-view and virtualization oracles +- SSR, hydration, and lifetime +- Accessibility, security, and performance +- Failure matrix +- Evidence report +- Sources and freshness -- Importing auth modules does not require production environment values. -- Server/client plugin IDs and framework bindings align. -- Issuer, discovery, handler mount, cookies, and trusted origins agree. -- A user authenticated in one organization cannot query another organization's - data. -- Public errors contain no database/provider secrets while diagnostics preserve - a redacted cause. +## Application contract inventory -## Browser and lifetime +Record before verification: -Server-render and hydrate representative routes while capturing diagnostics. -Navigate repeatedly, mount/unmount interactive components, and assert listener, -observer, timer, frame, subscription, and retained-root cleanup. Verify keyboard, -focus, accessible names, responsive overflow, touch, reduced motion, and failure -fallbacks. +```text +Route tree and protected groups: +Validated params/search and defaults: +Loader dependencies and query factories: +Server functions/repositories and tenant scope: +Forms/mutations and idempotency: +Auth client/server/plugins/base paths: +Local/URL/query/session selection state: +Tables/virtualizers/charts: +SSR adapter and hydration: +Listeners/observers/frames/subscriptions: +Expected failures and safe targets: +``` -## Build and deployment +Use fixtures and authorized local targets. Do not require production/shared mutations without permission. -Run typecheck, unit/integration tests, production SSR build, server-function/API -tests, and an adapter-equivalent browser flow. Verify environment separation, -assets, source maps, headers, cookies, and clean deployment startup. +## Route and URL oracles + +For every route state: + +- direct URL renders the same view as in-app navigation; +- reload preserves shareable filters, sort, page, and view; +- back/forward traverses intentional state changes; +- invalid search values parse to safe defaults or deliberate error; +- canonical defaults are stripped/preserved consistently; +- changing filter/query/sort/page-size resets dependent page; +- local dialog/selection/draft state does not pollute URL; +- protected route redirects preserve only allowlisted callback state; +- route error/not-found/pending boundaries render distinct usable views. + +Test the pure URL patch/reset functions and the actual router. A pure function passing does not prove browser history behavior. + +## Query and server-function oracles + +Assert: + +- loader and component call the same query-options factory; +- key includes every result-changing input and server authority such as tenant; +- one SSR request gets a fresh request-scoped QueryClient where required; +- critical data is prefetched/streamed according to installed integration; +- stale time and invalidation match product freshness; +- initial pending differs from background refresh; +- server functions parse input again and derive authority from session; +- repository query enforces tenant/base filters; +- public errors are stable and diagnostics are redacted; +- abort/cancellation/reordered results cannot overwrite current state. + +Instrumentation can fail tests on duplicate equivalent fetches or unexpected query keys. + +## Form and mutation oracles + +Test: + +- label/autocomplete/description/error association; +- keyboard submit and button type; +- touched/blur/submit validation timing; +- client bypass and server validation; +- slow/reordered async validation; +- double submit, retry, idempotency, and unique conflict; +- navigation/close during pending work; +- expired session/forbidden/conflict/rate-limit/network/500; +- field/form error mapping and focus/announcement; +- optimistic snapshot, rollback, authoritative response, and exact invalidation; +- provider/social/passkey pending paths do not overlap. + +Verify committed domain state at the server/repository, not only a success toast. + +## Auth and tenant oracles + +- importing client auth code requires no server secrets; +- server/client plugin and framework binding match; +- issuer/base URL, handler mount, discovery, callbacks, cookies, and trusted origins agree; +- allowed origin can send credentialed request; rejected origin cannot read it; +- unauthenticated, expired, revoked, and wrong-role sessions behave correctly; +- organization switch updates server authority, query keys, cached data, and UI; +- user cannot query/mutate another organization by changing URL/body/header; +- personalized routes are private/no-store as designed; +- logout/session invalidation clears protected state. + +Test authorization at repository/service boundaries. A route redirect and hidden button are insufficient. + +## Data-view and virtualization oracles + +- stable sorting with deterministic tie-breaker; +- filter/facet/count semantics under combinations; +- pagination boundaries after insert/delete; +- selected id remains the same record across sort/refresh; +- selection scope (page/ids/all-matching) matches bulk request; +- server reauthorizes every bulk target; +- table caption/headers/sort state and keyboard focus; +- responsive overflow/card alternative and zoom; +- virtualizer stable keys, measurement, overscan, scroll container, pinned rows, resize, and dynamic height; +- focused row policy when outside rendered range; +- SSR initial virtual range/hydration and scroll restoration; +- infinite load prevents duplicates/repeated calls and handles reordered pages; +- empty/loading/stale/partial/error/completed states remain distinct. + +Run with realistic row counts and variable content lengths. A ten-row fixture does not prove virtualization. + +## SSR, hydration, and lifetime + +For representative routes: + +1. Capture raw server HTML and initial serialized state. +2. Assert required content/styles before hydration. +3. Hydrate while capturing mismatch diagnostics. +4. Interact before/after hydration where possible. +5. Navigate/mount/unmount repeatedly. +6. Assert listener, observer, timer, frame, subscription, animation, WebGL, and retained-owner counts return to baseline. + +Cover media/storage preferences, random/time/locale, auth capabilities, query dehydration, invalid HTML, client-only fallbacks, portals, and persisted islands. + +## Accessibility, security, and performance + +Accessibility: + +- landmarks/headings/route announcement; +- keyboard/focus in dialogs, sheets, menus, grids, virtual lists; +- accessible field/status/error semantics; +- table/chart alternatives; +- zoom/reflow/contrast/forced colors/reduced motion/touch. + +Security: + +- client bundle/HTML/source maps contain no secrets; +- raw/rich content XSS tests; +- CSRF/origin/cookie/CORS tests; +- redirect allowlist; +- cross-tenant tests; +- cache headers and redacted logs/errors. + +Performance: + +- route JS/CSS and duplicate framework packages; +- loader waterfalls/query duplication; +- hydration/interaction timing; +- large table DOM/virtualization/memory; +- long tasks and continuous work; +- font/icon/image output; +- navigation memory/resource trend. + +## Failure matrix + +| Boundary | Required cases | +|---|---| +| URL/router | malformed, defaults, reload, back/forward, unknown route | +| Query | slow, stale, offline, abort, reordered, invalid response | +| Server function | invalid input, unauthenticated, forbidden, conflict, rate limit, 500 | +| Form | invalid/corrected, duplicate, pending navigation, server field error | +| Auth | expired/revoked session, bad origin/callback, organization switch | +| Table | empty, huge, sort ties, delete current page, resize, long content | +| Virtualizer | wrong estimate, dynamic height, focus off-range, fetch failure | +| Renderer | server mismatch, chunk failure, portal/theme mismatch | +| Resources | listener/frame/subscription leak, background/offscreen work | +| Deployment | missing env/binding, cold start, wrong base URL/cookie | + +Every case needs an observable oracle: response/status, DOM/accessibility state, cache key, repository result, resource count, or artifact. “No exception” is too weak. + +## Evidence report + +```text +Passed +- exact check and contract proved + +Failed +- observed signature and owning boundary + +Blocked +- missing target/credential/authority/runtime and remaining risk + +Not run +- deliberate exclusions + +Artifacts +- route traces, network logs, screenshots, accessibility output, + hydration diagnostics, bundle report, resource counters +``` + +Do not claim cross-browser, deployed, accessibility, security, or performance success from unit/type/build evidence alone. + +## Sources and freshness + +- Uploaded `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `old-finance-app(1).zip`, `solid-motion-experiments.zip`, and `solid-primitives(2).zip`, reviewed 2026-07-17. +- TanStack Router Query integration: https://tanstack.com/router/latest/docs/integrations/query (reviewed 2026-07-17). +- TanStack Router search params: https://tanstack.com/router/latest/docs/guide/search-params (reviewed 2026-07-17). +- Modern Web Guidance accessibility/forms/security/deferred-rendering guides retrieved 2026-07-17. +- Framework and auth APIs are version-sensitive; use repository-owned scripts and installed documentation. diff --git a/skills/build-web/SKILL.md b/skills/build-web/SKILL.md index ce842a8..4375359 100644 --- a/skills/build-web/SKILL.md +++ b/skills/build-web/SKILL.md @@ -48,6 +48,8 @@ policy over an Astro docs app, runtime CMS, and TanStack product app together. - [surfaces.md](references/surfaces.md): classification and ownership. - [renderers.md](references/renderers.md): Astro, Solid, React, native HTML, islands, SSR, and hydration boundaries. +- [assets.md](references/assets.md): renderer-owned icon/font boundaries, + component-registry adaptation, accessibility, privacy, and verification. - [components.md](references/components.md): Zaidan, shadcn, Kobalte, Corvu, styles, tokens, icons, and fonts. - [motion.md](references/motion.md): Solid lifetimes, presence, Motion, SSR, diff --git a/skills/build-web/references/assets.md b/skills/build-web/references/assets.md new file mode 100644 index 0000000..8070c2f --- /dev/null +++ b/skills/build-web/references/assets.md @@ -0,0 +1,86 @@ +# Renderer-owned icons and fonts + +## Contents + +- Ownership rules +- Icon contract +- Font contract +- Component-library boundaries +- Verification +- Sources and freshness + +## Ownership rules + +Choose the asset integration at the rendering boundary: + +| Boundary | Icon owner | Font owner | +|---|---|---| +| Astro static component | Astro Icon or local SVG/Astro component | Astro Fonts API/local provider | +| Solid island/application | Unplugin Icons with Solid compiler | app-root Fontsource import or inherited site font CSS | +| React island/application | Unplugin Icons with compatible JSX/React compiler | app-root Fontsource/framework font owner | +| framework-neutral package | icon component contract or raw reviewed SVG data | no global font import; expose tokens/document requirements | + +One visual system may use the same Fluent collection across Astro and Solid while using different renderers. Do not make a framework island responsible for an otherwise static icon. Do not import fonts inside reusable leaf components. + +When `build-sites` is active, load its `icons.md` and `fonts.md` for the complete Astro-specific configuration, provider, preload, fallback, accessibility, and verification rules. + +## Icon contract + +Record: + +- collection and style family (for example Fluent Regular); +- server/client renderer and compiler; +- literal import or reviewed registry; +- dimensions and viewBox; +- current-color fill/stroke policy; +- decorative versus informative semantics; +- local SVG provenance, sanitization, and license; +- allowed names for dynamic selection; +- bundle inclusion mechanism. + +An icon-only button gets its accessible name from the button. Decorative SVGs remain hidden. An informative SVG needs a verified title/description relationship. Do not infer accessibility from a `title` prop. + +## Font contract + +Record: + +- family role and CSS token; +- provider/source, version, license, and privacy boundary; +- exact weights/styles/subsets/variable axes; +- self-hosted asset/caching owner; +- fallback sequence and metric adjustment; +- `font-display` and preload policy; +- first-paint routes using each face; +- language/glyph coverage; +- layout-shift and visual-regression evidence. + +Avoid duplicate owners such as an Astro provider plus a Fontsource CSS import for the same family. Preload only exact first-paint faces. A configured variable weight range must match the file's axes. + +## Component-library boundaries + +Open-code component registries such as shadcn/Zaidan can carry icons and font classes from a different renderer or design system. After generation: + +1. replace renderer-incompatible icon imports; +2. preserve accessible names and focus behavior; +3. map size/color classes to local tokens; +4. remove package-wide font assumptions; +5. verify SSR and hydration; +6. keep generator provenance so future updates can be reconciled. + +Do not mix Lucide, Fluent, local filled icons, and text glyphs within one surface accidentally. A deliberate exception should document why the primary collection lacks the needed shape. + +## Verification + +```bash +rg -n 'astro-icon|unplugin-icons|~icons/|lucide|fontProviders|@fontsource|@font-face' \ + src astro.config.* vite.config.* package.json +``` + +Run the production server/client build, inspect built HTML/CSS, trace network font requests, measure critical asset sizes, test no-JS/static output, and exercise screen-reader names, keyboard focus, high contrast, reduced data, slow font loading, and mobile layout. + +## Sources and freshness + +- Primary: [Astro Icon](https://www.astroicon.dev/), [Unplugin Icons](https://github.com/unplugin/unplugin-icons), [Astro Fonts](https://docs.astro.build/en/guides/fonts/), and [Fontsource](https://fontsource.org/docs/), verified 2026-07-17. +- Attachments: `kaiju-site-scope(17).zip/apps/frontend`, `kaiju-site-scope(17).zip/apps/docs`, and `kaiju-website(6).zip`, inspected 2026-07-17 for cross-renderer asset ownership. + +Renderer compilers, Astro APIs, and generated virtual modules are version-sensitive. This reference defines ownership; exact imports must be verified against the target framework and lockfile. diff --git a/skills/build-web/references/components.md b/skills/build-web/references/components.md index 29c8afb..e4ab53a 100644 --- a/skills/build-web/references/components.md +++ b/skills/build-web/references/components.md @@ -1,44 +1,240 @@ -# Components, styles, icons, and fonts +# Component systems, primitives, styles, and assets + +Use this reference when selecting, generating, copying, repairing, or verifying UI components. A component file is the visible tip of a renderer, primitive, style, token, icon, font, and accessibility contract. + +## Contents + +- Component evidence inventory +- Generated component contract +- Primitive selection +- Component API design +- Styling and token ownership +- Accessibility and interaction +- Icons, fonts, and media +- Integration and upgrade procedure +- Failure signatures +- Verification +- Sources and freshness + +## Component evidence inventory + +For every component family inspect: + +- `components.json` or other registry configuration; +- registry URL, style, target renderer, aliases, icon library, and generated path; +- package manifest and direct imports; +- headless behavior primitive and peer versions; +- variant builder and class merge helper; +- stylesheet entrypoint, layer order, generated classes, tokens, themes, and Tailwind scanning; +- renderer integration, JSX compiler, icon compiler, and test environment; +- wrapper components that add application policy; +- accessible semantics, keyboard map, focus ownership, portal/layer behavior, and form integration; +- which files are generated and which are application-owned. + +The Kaiju website's `components.json` identifies a Kobalte style, Solid-oriented aliases, Lucide icons, and a custom Zaidan registry. The finance app uses React and Base UI. Similar names such as `dialog.tsx` or `button.tsx` do not make their implementations interchangeable. ## Generated component contract -A copied component rarely stands alone. Inspect: +Treat generated source as a synchronized set: + +```text +registry configuration + -> renderer-specific generated source + -> behavioral primitive packages + -> variants/class utilities + -> generated or base stylesheet + -> semantic tokens and theme + -> application wrapper and tests +``` + +Before copying a component, search for every non-platform import and class prefix. A `button.tsx` that references `z-*`, CSS variables, `cva`, or a registry helper will render incorrectly if only the TSX moves. -- `components.json` or registry provenance; -- renderer-specific source; -- Kobalte, Corvu, cmdk, or other behavioral primitives; -- CVA/variant definitions; -- generated `z-*` or equivalent stylesheet classes; -- shadcn/base stylesheet imports; -- Tailwind content scanning and tokens; -- application-level wrappers and accessibility tests. +Choose an ownership policy: -If `button.tsx` refers to generated classes, copying only the TSX is incomplete. -Prefer composing installed primitives at the application layer before editing -generated internals. +- generated internals remain close to upstream and are regenerated intentionally; +- application wrappers add analytics, permissions, domain defaults, and composition; +- patches to generated code are recorded and covered by behavior tests; +- registry upgrades are reviewed as source changes, never accepted only because generation succeeded. + +Do not run a generator across an unrelated dirty worktree without reviewing its planned files. Snapshot the affected component/style inventory first. ## Primitive selection -Treat Solid Primitives as an ecosystem of granular packages, not a single hooks -library. For a candidate primitive inspect: +Prefer native HTML where its semantics and behavior fit. Use a headless primitive when the interaction requires a complete state machine such as dialog focus trapping, roving focus, menu keyboard behavior, combobox selection, or collision-aware popovers. + +For Solid Primitives, treat the repository as an ecosystem of granular packages rather than a single hooks library. Search the monorepo by concern and inspect each candidate's README, package status badge, server tests, peer dependencies, and sibling helpers. Examples in the uploaded monorepo include: + +- lifecycle and ownership: `@solid-primitives/lifecycle`, `rootless`, `refs`, `props`; +- browser signals: `media`, `page-visibility`, `connectivity`, `permission`, `platform`; +- observers: `intersection-observer`, `resize-observer`, `mutation-observer`; +- scheduling: `scheduled`, `raf`, `idle`; +- interaction: `event-listener`, `keyboard`, `pointer`, `gestures`, `active-element`; +- collections/state: `list`, `map`, `set`, `immutable`, `mutable`, `deep`; +- transitions/presence: `presence`, `transition-group`, `spring`; +- network/persistence: `fetch`, `broadcast-channel`, `cookies`, `db-store`. + +Package existence does not prove production maturity or suitability. Determine: + +- `make*` low-level/non-reactive primitive versus `create*` reactive owner; +- server result and whether it is safe during SSR; +- cleanup semantics and owning Solid scope; +- equality and update behavior; +- package stage/maturity; +- browser support and fallback; +- whether another sibling package must be installed. + +Do not install the whole ecosystem. Select the narrow capability and verify its actual exported API in the installed version. + +## Component API design + +Prefer an API that exposes state and semantics rather than implementation accidents: + +```ts +type DialogController = { + open: boolean; + onOpenChange(open: boolean): void; +}; + +type ConfirmDeleteProps = DialogController & { + resourceName: string; + pending: boolean; + onConfirm(): Promise<void>; +}; +``` + +Define: + +- controlled versus uncontrolled ownership; +- initial/default value separately from current value; +- stable item identity; +- event ordering and cancellation; +- pending, disabled, readonly, invalid, and unavailable semantics; +- slot/child composition and prop forwarding; +- ref forwarding and DOM ownership; +- portal container and layering policy; +- SSR output and hydration behavior; +- cleanup and exit behavior. + +Avoid boolean-prop explosions. Use variants or composed subcomponents when multiple independent behaviors produce incoherent combinations. Do not hide authorization inside a visual `disabled` prop; server policy remains authoritative. + +## Styling and token ownership + +Keep one semantic token source and an intentional layer order: + +```css +@import "tailwindcss"; +@import "tw-animate-css"; +@import "shadcn/tailwind.css"; +@import "./base.css"; + +@theme inline { + --color-background: var(--background); + --color-foreground: var(--foreground); + --radius-lg: var(--radius); +} +``` + +Verify the actual toolchain syntax before copying. Tailwind v4 `@theme`, `@custom-variant`, and layer behavior should not be backported to a different major version by appearance. + +Component styling must survive: + +- light/dark and forced-colors modes; +- 200% text zoom and narrow containers; +- long translated text and unknown content length; +- focus-visible, hover-capable and coarse-pointer devices; +- invalid, pending, disabled, read-only, selected, expanded, and destructive states; +- user font loading failure; +- portal content outside a scoped CSS ancestor. + +Do not encode semantic state only in color. Do not remove focus outlines without a visible replacement. Prefer logical properties where bidirectional layouts matter. + +## Accessibility and interaction + +Start from the semantic element: + +- action: `<button type="button">`; +- navigation: `<a href>`; +- disclosure: `<details><summary>` when its behavior fits; +- input: native labeled form control; +- grouped controls: `<fieldset><legend>`; +- data: semantic `<table>` with caption/headers when tabular; +- status: a restrained live region when an update needs announcement. + +When a primitive implements ARIA patterns, test the complete keyboard interaction and accessible tree. A `role` without required keyboard/state behavior is worse than a native element. + +For dialogs and overlays verify: + +- trigger has an accessible name; +- initial focus is intentional; +- focus stays inside modal content and returns to the correct trigger; +- Escape behavior and outside interaction match task risk; +- background becomes inert/inaccessible when modal; +- nested portals and z-index do not split the interaction tree; +- destructive action explains consequence and does not make cancellation hard. + +For repeated icon buttons, include the affected item in the accessible name, such as “Remove invoice 1042,” not twelve identical “Remove” controls. + +## Icons, fonts, and media + +Renderer ownership matters: + +- Astro Icon renders icons in `.astro` templates. +- Unplugin Icons produces framework/compiler-specific virtual components for Solid or React islands. +- local SVG collections need provenance, sanitization, viewBox consistency, and naming policy. +- decorative icons use `aria-hidden="true"`; meaningful icons need a visible label or verified accessible name. + +Avoid wildcard inclusion of complete icon sets unless bundle analysis proves the integration remains on-demand. The uploaded Astro configs include wildcard collections; treat that as a review target, not a recommended default. + +Fonts need an owner, exact variants, subsets, fallback stack, metric compatibility, preload budget, and license/provenance. Preload only critical faces used above the fold. A page that preloads every configured family delays more important resources. + +Images and canvas need dimensions/aspect ratio, responsive source policy, alt/fallback behavior, loading priority, and failure state. Complex canvas/WebGL visuals need a meaningful static alternative; do not expose decorative canvases to the accessibility tree. + +## Integration and upgrade procedure + +1. Classify renderer and component behavior. +2. Inspect the registry and upstream primitive source for the installed version. +3. Inventory generated files, styles, tokens, aliases, and dependencies. +4. Generate/copy into an isolated diff. +5. Reapply application wrappers and deliberate patches. +6. Test semantics before visual polish. +7. Test renderer build, SSR/hydration, and portal behavior. +8. Compare bundle, CSS, icons, fonts, and runtime resources. +9. Record unsupported variants or interactions. + +Never claim parity with a shadcn or upstream component by matching its public prop names. Verify behavior, DOM, keyboard, focus, forms, styles, and failure modes. + +## Failure signatures -- package maturity stage; -- peer and sibling dependencies; -- `make*` non-reactive foundation versus `create*` reactive owner; -- server behavior and hydration contract; -- cleanup semantics; -- tests and documented edge cases. +| Signature | Likely cause | Next inspection | +|---|---|---| +| Component renders unstyled | CSS layer/token/generated class missing | Registry config and style entrypoint | +| Dialog opens but focus escapes | Primitive/wrapper contract incomplete | Focus scope, portal, modal settings | +| Solid component stops reacting | Props destructured or copied | Live access paths and wrappers | +| React component imported into Solid island | Registry target mismatch | Source imports and Astro integration | +| Icons work in Astro but fail in island | Wrong Unplugin compiler/JSX mode | Vite plugin and virtual import | +| Thousands of icons in output | Wildcard/eager collection inclusion | Plugin config and bundle graph | +| Dark portal has light tokens | Theme scoped below portal root | Token scope and portal container | +| Button submits unexpectedly | Missing `type="button"` | Rendered HTML/form context | +| Menu works by pointer only | Incomplete ARIA primitive or wrapper | Keyboard matrix and roles | +| Layout shifts when font loads | Missing metrics/dimensions/preload policy | Font fallback and waterfall | -Select packages by concern. Do not install unrelated primitives or recreate -event listeners, scheduling, media queries, observers, roots, or prop merging -without checking the relevant package. +## Verification -## Styles and assets +- Run registry or generated-source diff in isolation. +- Run renderer typecheck and production build. +- Render the component server-side and hydrate it with diagnostics captured. +- Test accessible name, role, states, keyboard sequence, focus entry/return, and form behavior. +- Test portals inside real layouts, dialogs, scroll containers, and nested overlays. +- Test content-length, zoom, RTL, forced colors, reduced motion, touch, and narrow container cases. +- Inspect CSS output for required generated classes and token definitions. +- Inspect bundle contents for icon collections, duplicate primitives, and accidental framework runtimes. +- Count listeners/observers/frames across mount/unmount. +- Verify image/font/icon failure and fallbacks. -Keep one token source, predictable layer order, focus-visible states, contrast, -forced-colors behavior, responsive overflow, and content-length resilience. +## Sources and freshness -Astro templates and Solid/React islands may need different icon integrations. -Avoid wildcard inclusion of complete icon collections without measuring bundle -output. Define font loading, subsets, fallbacks, metric compatibility, and -render-blocking policy through the owning framework or Fontsource integration. +- Uploaded evidence reviewed 2026-07-17: `kaiju-website(6).zip`, `new-finance-app(1).zip`, `kaiju-site-scope(17).zip`, `solid-primitives(2).zip`. +- Solid Primitives official catalog: https://primitives.solidjs.community/ (reviewed 2026-07-17). +- Solid props and `splitProps`: https://docs.solidjs.com/concepts/components/props (reviewed 2026-07-17). +- Astro Icon and Unplugin Icons are detailed in `build-sites/references/icons.md`. +- Registry schemas, generated component source, Tailwind directives, and package maturity are version-sensitive. Inspect installed source rather than copying current examples blindly. diff --git a/skills/build-web/references/failures.md b/skills/build-web/references/failures.md index e5968ea..1175d4d 100644 --- a/skills/build-web/references/failures.md +++ b/skills/build-web/references/failures.md @@ -1,21 +1,131 @@ -# Web failure signatures - -| Signature | Likely cause | Next inspection | -|---|---|---| -| Interactive file exists but is never imported | Abandoned experiment | Route/component import graph | -| Static disclosure hydrates a framework | Native semantics overlooked | Required behavior and HTML primitive | -| Component renders unstyled | Generated CSS/token/registry contract missing | Registry, stylesheet layers, class definitions | -| Solid prop stops updating | Reactive prop destructured | Access paths, splitProps, memo/effect boundaries | -| Hydration mismatch or flash | Nondeterministic initial state | Server HTML and first client values | -| Removed item never disappears | Presence completion not called | Exit callback and retained owner | -| Work continues offscreen | Visibility policy only controlled hydration | Observer and frame ownership | -| Duplicate click after navigation | Repeated lifecycle listener | Idempotency and cleanup | -| Icon build/type failure | Wrong renderer compiler | Framework integration and import suffix | -| `client:only` build failure | Renderer cannot be inferred | Explicit renderer hint | -| Auth works but cross-org data leaks | Authentication mistaken for authorization | Server policy/base filter | -| Webhook logs keys or payloads | Boundary treated as page glue | Structured redaction and signature policy | -| Repository name implies an extension | Name trusted over source | Manifests, entrypoints, browser APIs | -| Motion types expose gestures but nothing responds | Declared API exceeds implementation | Event-binding source and tests | - -Do not fix these by adding a broad framework abstraction. Identify the actual -owner, preserve renderer semantics, and verify the corrected path. +# Web failure signatures and next inspections + +Use this reference during diagnosis and review. A signature narrows the next evidence to inspect; it does not authorize a broad rewrite. + +## Contents + +- Classification and render failures +- SSR, hydration, and navigation failures +- State, forms, and data failures +- Component and accessibility failures +- Motion and resource failures +- Security and connected-system failures +- Content, asset, and deployment failures +- Diagnostic procedure +- Sources and freshness + +## Classification and render failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Interactive-looking file exists but is never imported | Abandoned experiment | Route/component import graph | Declare it the current architecture | +| Entire Astro site switched to server output for one dynamic route | Route scope not classified | Per-route `prerender`, adapter need | Assume server mode adds capability | +| Static docs require runtime server | Build-time factories overlooked | Static paths and generated JSON | Add SSR without request-time need | +| Repository called extension but has no manifest/contexts | Name trusted over source | Manifest, background, content scripts | Invent extension APIs | +| Static content absent until JavaScript | Unnecessary client-only rendering | Raw server HTML and island directive | Hide with loading spinner | +| Server island never resolves | Adapter/runtime or deferred route failure | Adapter output, network request, fallback | Replace page with full client app | +| Personal content appears in shared cache | Cache key/policy omits identity | Route classification and headers | Patch client display only | + +## SSR, hydration, and navigation failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Hydration mismatch | Server/client first render differs | HTML, time/random/media/storage/ids | Suppress warning globally | +| Theme or motion flashes | Browser preference read before/after inconsistent snapshot | Inline/server theme contract and mount reconciliation | Make whole page client-only | +| `client:only` build error | Renderer cannot be inferred or integration absent | Explicit hint and installed integration | Guess `react` from `.tsx` | +| Browser API fails during build | Module/render-time DOM access | Import graph and mount gate | Wrap random lines in `try/catch` | +| Duplicate click after navigation | Repeated `astro:page-load` listeners | Delegation, AbortController, cleanup | Add a boolean on each element | +| Back button loses filters | State kept only in component | URL schema and navigation updates | Add a second global store | +| Route announcement is wrong | Missing/duplicate title or heading | Document metadata and client router | Add noisy custom live regions | +| Persisted island shows stale user | Lifetime/props persisted across auth change | `transition:persist`, prop policy, session key | Force full reload everywhere | + +## State, forms, and data failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Loader and component fetch twice | Different query keys/options | Canonical query-options factory | Disable refetch blindly | +| Filter changes but page remains out of range | Dependent pagination not reset | URL patch policy | Clamp only in render | +| Copied URL cannot reproduce view | Shareable state kept locally | URL ownership table | Serialize dialog/hover state too | +| Old async validation overwrites new value | Missing cancellation/sequence guard | Validation request identity | Increase debounce only | +| Double submit creates duplicates | UI guard without server idempotency | Pending state, idempotency/unique policy | Trust disabled button | +| Optimistic row never reconciles | Mutation invalidation/rollback incomplete | Query identity and authoritative response | Keep optimistic state forever | +| Selected rows change after sort | Array index used as identity | Domain row ids | Freeze sorting | +| “Select all” affects wrong scope | Selection policy undefined | Loaded/current-query/all-matching contract | Infer from checkbox state | +| Virtual list focus disappears | Focused row unmounted | Roving/focus/overscan/fallback policy | Inflate overscan indefinitely | +| Loading, empty, and error look identical | State model collapsed | Query/status branching | Add one generic skeleton | + +## Component and accessibility failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Generated component unstyled | CSS/token/registry dependency missing | `components.json`, layers, generated classes | Rewrite component from scratch first | +| Dialog opens but focus escapes | Primitive/wrapper/portal mismatch | Modal focus scope and container | Add `tabindex` to every element | +| Menu works by mouse only | Incomplete keyboard state machine | Primitive source and APG behavior | Add only Enter handler | +| Button submits parent form unexpectedly | Missing `type="button"` | Rendered HTML/form nesting | Prevent all form submit events | +| Icon-only actions announced identically | Accessible names omit target | Button labels per item | Put state in icon filename | +| Hidden control still receives focus | Visual/a11y state diverges | `hidden`, `inert`, `aria-hidden`, tab order | Use `aria-hidden` on focusable nodes | +| Error visible but not announced | Missing association/focus/live policy | `aria-describedby`, summary, status | Make every message assertive | +| Table read as layout | Missing semantic table/header relations | Rendered roles/caption/headers | Add ARIA grid without keyboard model | +| Component behavior changes after registry update | Generated source/primitive drift | Diff and behavior suite | Trust version number alone | + +## Motion and resource failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Removed item never disappears | Exit completion path missing | Presence registry and zero-animation case | Add arbitrary timeout | +| Exit never appears | Owner disposed before retention | Parent control-flow boundary | Start animation in cleanup | +| Duplicate item after reentry | Same-key policy undefined | Retained record identity | Generate random keys | +| Motion prop type exists but no response | Type surface exceeds runtime | Event binding/renderer tests | Document capability as complete | +| Animation jumps on hover release | Lane priority/resume wrong | Current value/velocity and resolver | Reset to initial value | +| Layout motion reads on server | Measurement not mount-gated | FLIP phase ownership | Use optional chaining as proof | +| Work continues offscreen | Initial hydration visibility mistaken for suspension | Page/intersection visibility | Change only `client:visible` margin | +| CPU/memory grows per route | Missing disposal | Frames, listeners, observers, roots, WebGL | Rely on garbage collection | +| WebGL failure leaves blank hero | Static fallback removed/covered | Image stacking and error state | Make WebGL critical content | +| Reduced motion still parallax-scrolls | Policy changes duration only | Behavior-level preference branch | Set duration to 1ms | + +## Security and connected-system failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Auth works but cross-org data leaks | Authentication mistaken for authorization | Server scope in repository query | Hide other-org controls | +| Credentialed CORS fails | Origin/cookie/trusted-origin mismatch | Preflight and Set-Cookie in browser | Use wildcard origin | +| OAuth loops or discovery URL wrong | Base URL/path/issuer mismatch | Handler mount and metadata | Patch callback URL only | +| Secrets appear in bundle | Server module reachable from client | Import graph and serialized props | Rename environment variable | +| Webhook accepts spoofed events | Signature/raw-body/replay missing | Provider verification sequence | Rely on obscure URL | +| Webhook retries duplicate side effects | Event id/idempotency missing | Persistence and retry contract | Always return 200 before work | +| Logs contain API key or payload PII | Boundary logging raw objects | Structured redaction policy | Remove all diagnostics | +| CSP breaks valid UI | Policy not derived from resources/nonces | Violation reports and asset origins | Disable CSP globally | +| Rich content executes script | Raw HTML not sanitized/mapped | Content render boundary | Escape only one field | +| Mutating GET route | Method semantics collapsed | Endpoint exports and caller | Add CSRF token to GET | +| Personalized page served stale | Public/shared caching on auth route | Middleware classification | Bust cache with random URL | + +## Content, asset, and deployment failures + +| Signature | Likely cause | Next inspection | Do not do | +|---|---|---|---| +| Draft appears publicly | Status filtering absent/inconsistent | CMS adapter/query and preview mode | Filter only in page component | +| Taxonomy route empty without error | Wrong taxonomy name/id field | Provider schema and adapter | Rename routes by guess | +| N+1 content requests | Relationships mapped per entry | Batch API/cache strategy | Cache forever without invalidation | +| Missing image crashes build | Schema assumes relation/media always exists | Mapper defaults and content policy | Insert fake image path | +| Canonical/social image uses wrong origin | Relative/absolute URL contract mixed | Site config and URL construction | String concatenate origins | +| Feed and page disagree | Separate content selection/mapping | Shared published view model | Patch feed-only exclusions | +| Icon build/type failure | Wrong compiler or virtual module | Astro versus island renderer config | Switch every icon package | +| Font waterfall/layout shift | Too many preloads/missing metrics | Network waterfall and fallback | Preload every family | +| Adapter works locally but deployment fails | Runtime binding/API mismatch | Target adapter build and startup | Claim preview equivalence | +| Static route unexpectedly hits database | Import side effect or dynamic source | Build trace and content ownership | Add broad network permission | + +## Diagnostic procedure + +1. Reproduce the smallest real failing path. +2. Record route, output mode, renderer, state owners, auth scope, and connected systems. +3. Inspect the active import/runtime graph; ignore unreachable examples. +4. Capture raw server response, browser console/network, accessibility tree, and resource counts as relevant. +5. Identify the first boundary where actual behavior diverges from the documented contract. +6. Fix that owner without adding a second owner. +7. Add a regression oracle at the lowest layer that reproduces the failure and one higher integration layer when the boundary crosses systems. +8. Re-run adjacent negative/failure cases. + +## Sources and freshness + +- Failure patterns synthesized from uploaded `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `thunderstrike-blog(4).zip`, `solid-motion-experiments.zip`, and `solid-primitives(2).zip`, reviewed 2026-07-17. +- Current framework-specific error messages can change. Match behavior and ownership before relying on a literal message. diff --git a/skills/build-web/references/motion.md b/skills/build-web/references/motion.md index 20a33eb..c9d18bc 100644 --- a/skills/build-web/references/motion.md +++ b/skills/build-web/references/motion.md @@ -1,42 +1,313 @@ -# Motion and presence +# Motion, presence, canvas, and continuous work -## Selection order +Use this reference when implementing or reviewing animation, transitions, presence, gesture states, layout motion, canvas/WebGL, or scroll/pointer effects. Motion is a lifecycle and state-ownership problem before it is a visual problem. -1. No motion when it adds no comprehension or feedback. -2. CSS transitions/animations for simple state changes. -3. Web Animations or a small primitive for imperative DOM behavior. -4. Renderer-aware component integration for coordinated state. -5. A presence system only when removed values must remain until exit completes. +## Contents -## Solid motion contract +- Evidence and capability inventory +- Selection ladder +- State and priority model +- Solid motion adapter boundaries +- Presence and exit retention +- SSR and hydration +- Layout motion +- WebGL and continuous work +- Accessibility and performance policy +- Failure signatures +- Verification +- Sources and freshness -Solid does not rerun component functions like React. Preserve reactive access to -props and split renderer state from host component ownership. Start DOM-backed -animation after mount and make the server/initial client style deterministic. +## Evidence and capability inventory -Presence requires more than retaining DOM values: +Do not infer runtime capability from prop types, package names, or a demo screenshot. Trace: -- stable semantic identity; -- retained Solid owner and context where descendants need it; -- explicit completion callback; -- defined remove, reorder, cancel, and same-key reentry behavior; -- cleanup on cancellation and disposal. +- public exports and the code path that consumes each prop; +- event bindings for hover, focus, press, tap, drag, and viewport states; +- value renderer, animation driver, cancellation, and completion callback; +- DOM/SVG style writers and transform composition; +- variant resolution and priority order; +- presence registration, retained ownership, and removal; +- layout measurement timing and scroll/fixed ancestor compensation; +- server initial-style serialization and hydration tests; +- reduced-motion policy; +- resource cleanup. -If exit completion is never called, the value may remain forever. Object identity -and logical identity are not always the same. +The uploaded Solid motion code is an experiment with explicit limitations. It provides source evidence for MotionState, initial styles, presence aggregation, and single-slot retention. Research documents explicitly defer keyed-list parity, full gesture/layout parity, and Solid 2 production parity. Do not turn experimental types into production claims. -## Operational budget +## Selection ladder -Reduced motion is a behavior policy, not only lower amplitude. Decide whether to -disable, shorten, replace, or preserve essential movement. Pause continuous frame -work while offscreen when appropriate. `client:visible` controls initial -hydration, not later suspension. +Choose the smallest mechanism that provides the needed semantics: -Provide a static fallback for WebGL, canvas, image, or animation failure. Test no -context, loading errors, background tabs, resize, unmount, and route navigation. +1. No motion when it adds no feedback, continuity, spatial explanation, or state change. +2. CSS transition for hover/focus/pressed/expanded visual states. +3. CSS keyframes for self-contained decorative sequences with a clear reduced-motion replacement. +4. Web Animations API for imperative timing, cancellation, or sequencing on known DOM nodes. +5. A renderer-specific primitive when motion targets reactive component state. +6. Presence only when logically removed content must remain physically present for exit. +7. Layout projection only after ordinary layout/transform approaches cannot express the continuity. +8. Canvas/WebGL only when the visual cannot be delivered efficiently and accessibly with document content or media. -## Capability honesty +Do not add a framework island solely to rotate a disclosure icon. The Kaiju FAQ uses `<details>` plus a CSS `group-open` transition; that preserves native disclosure without hydration. -Inspect implementation and tests before claiming gesture, viewport, layout, -presence, SSR, or Solid-version parity. Declared prop types and research notes do -not prove an event-binding or renderer exists. +## State and priority model + +Separate motion lanes rather than letting the last effect win. The uploaded design proposes this low-to-high order: + +```text +base style + < initial + < animate / variants + < whileInView + < whileFocus + < whileHover + < whileTap / whilePress + < drag ownership + < layout projection + < exit +``` + +This is evidence from the local experiment, not a universal Motion guarantee. Compare with the installed animation engine's actual priority model before integrating. + +Define conflict rules: + +- higher priority overrides only the value channels it owns; +- lower lanes remain available for unaffected channels; +- release resumes from the current sampled value rather than jumping to an old initial value; +- drag owns position channels while active; +- exit becomes terminal only after logical removal and cancels/replaces incompatible work; +- cancellation settles completion exactly once. + +Example state contract: + +```ts +type MotionLane = + | "animate" + | "inView" + | "focus" + | "hover" + | "press" + | "drag" + | "layout" + | "exit"; + +type ActiveTargets = Map<MotionLane, Record<string, unknown>>; +``` + +The map alone is not an implementation. A resolver must merge channels in documented priority order and a renderer must animate/cancel values. + +## Solid motion adapter boundaries + +Keep framework-neutral animation work separate from Solid ownership: + +```text +Solid adapter + - reactive prop access + - mount/ref timing + - context and presence registration + - owner cleanup + - SSR initial snapshot + +animation engine + - MotionValue/value lifecycle + - target animation and cancellation + - HTML/SVG writes + - easing, spring, time scheduling +``` + +Do not destructure reactive motion props. Snapshot options in a tracked scope only if the runtime owns a plain-object contract, and keep render-time initial-style resolution pure. + +The local experiment deliberately reuses framework-neutral `motion-dom` primitives such as value animation and DOM style writers while retaining Solid-owned `mount`, `unmount`, option update, presence, and cleanup seams. It does not yet prove that higher-level VisualElement APIs can be used unchanged. + +External work begins after mount: + +```tsx +onMount(() => { + const animation = element.animate(keyframes(), timing()); + const stopPreference = bindReducedMotion((reduced) => { + if (reduced) animation.finish(); + }); + + onCleanup(() => { + stopPreference(); + animation.cancel(); + }); +}); +``` + +Completion handlers must tolerate cancellation and disposal. Never leave a rejected `.finished` promise unobserved. + +## Presence and exit retention + +Presence separates: + +- logical presence: the item remains in application state; +- physical presence: the DOM and reactive owner remain long enough to complete exit. + +Solid control flow normally disposes a removed branch. Starting an exit effect after disposal is too late. The parent presence boundary must own retained records: + +```text +next keyed records + -> diff previous records + -> mark missing record logically absent + -> retain its owner/DOM + -> descendants register exit work + -> aggregate completion + -> release record and dispose exactly once +``` + +Minimum contracts: + +```ts +interface PresenceHandle { + id: symbol; + key?: string | number; + startExit(): void; + dispose(): void; +} + +interface PresenceBoundary { + isPresent: boolean; + register(handle: PresenceHandle): () => void; + onExitComplete(id: symbol): void; +} +``` + +Required cases: + +- no exit target: remove without deadlock; +- one exit: retain until completion; +- nested exits: wait for every registered descendant; +- cancellation: completion/release once; +- remove then same-key reentry: explicitly cancel, replace, or coexist; +- reorder without remove: preserve logical identity; +- owner disposal during exit: retained handle still has a safe terminal path; +- `wait` mode: entering child does not starve if exit has no animated values. + +The uploaded implementation's `AnimatePresence` is single-slot with `sync` and `wait` experiments. It is not evidence for keyed-list or Framer Motion parity. `@solid-primitives/transition-group` is useful evidence for transition sequencing, but DOM-element retention alone is not proof that nested Solid owners/context remain alive. + +## SSR and hydration + +Server output and first client render must use the same initial style. Resolve it without DOM access: + +```ts +function resolveInitialStyle(options: MotionOptions): JSX.CSSProperties { + if (options.initial === false || options.initial === undefined) { + return resolveTarget(options.animate); + } + return resolveTarget(options.initial); +} +``` + +The exact library rules may differ; test them. Cover: + +- explicit initial target; +- `initial={false}`; +- omitted initial; +- variants and custom data; +- CSS variables plus transforms; +- HTML and SVG; +- reduced motion known only after mount; +- server markup hydration diagnostics. + +Do not initialize to a browser media-query value on the client if the server serialized another value. Hydrate the shared snapshot first, then reconcile preferences. + +## Layout motion + +FLIP requires ordered reads and writes: + +```text +First: capture previous box +DOM update +Last: capture next box +Invert: apply delta transform +Play: animate to identity +``` + +Fine-grained updates make the pre-mutation capture boundary difficult. Start with explicit `layoutDependency` or invalidation rather than installing observers everywhere. Validate: + +- read/write phase separation; +- transforms and existing transform composition; +- scroll containers and fixed ancestors; +- resized content and fonts/images; +- server path performs no layout reads; +- cancellation/reorder during active projection; +- focus and hit testing during transforms. + +Shared-layout ids require uniqueness and a defined scope. They do not automatically solve cross-root, portal, or route-transition ownership. + +## WebGL and continuous work + +The Kaiju depth scene provides a concrete resource checklist: + +- Solid island begins after mount; +- pointer listener is passive and removed; +- reduced-motion media listener is removed; +- `requestAnimationFrame` ids are cancelled; +- `ResizeObserver` disconnects; +- WebGL textures, buffers, and programs are deleted; +- third-party particles container is destroyed; +- async initialization checks a `disposed` flag; +- a static `<img alt="">` remains when WebGL or texture loading fails; +- canvas is decorative and `aria-hidden`. + +Improve it by defining visibility/page-lifecycle suspension. `client:visible` delays initial hydration but does not pause frames after the island scrolls away. Use page visibility and intersection evidence when continuous work is expensive. + +Cap device pixel ratio and texture dimensions deliberately. Handle context loss/restoration if the feature is important; otherwise fall back permanently. Never let a decorative hero prevent content or interaction. + +## Accessibility and performance policy + +Reduced motion is a behavior decision: + +| Motion purpose | Reduced-motion response | +|---|---| +| Decorative drift/parallax | Disable | +| State feedback | Shorten or replace with opacity/color | +| Spatial navigation | Preserve minimal continuity without large travel | +| Progress/indeterminate activity | Preserve non-motion status text; reduce continuous movement | +| Essential simulation | Provide controls and a static/step alternative | + +Also: + +- never hide focus or change focus order during animation; +- avoid vestibular triggers such as large parallax, zoom, and continuous background movement; +- prefer transform/opacity but measure compositing and memory rather than assuming they are free; +- cancel work in background tabs and when removed; +- avoid animating blur/large shadows or layout properties on large trees without evidence; +- define interrupt, rapid toggle, and route navigation behavior; +- do not announce decorative animation through live regions. + +## Failure signatures + +| Signature | Likely cause | Next inspection | +|---|---|---| +| Removed item never disappears | Exit completion never settles | Registration, cancellation, zero-animation path | +| Item disappears before exit | Parent did not retain owner | Control-flow/presence boundary | +| Same key renders twice | Reentry policy missing | Record identity and cancel/replace rules | +| Hydration flash | Initial style differs server/client | Pure resolver and serialized markup | +| Hover/tap sticks | Lane release/cancellation missing | Priority resolver and event cleanup | +| Work continues offscreen | Hydration visibility mistaken for suspension | Page/intersection visibility owner | +| FPS degrades per navigation | Frame/listener/observer leak | Mount/unmount resource counts | +| WebGL blank removes hero | No fallback or wrong stacking | Static image and error state | +| Types advertise drag but nothing moves | Public surface exceeds implementation | Event bindings and integration tests | +| Layout jumps after font/image | Measurement before content settled | Invalidation and asset dimensions | + +## Verification + +1. Test target resolution and priority conflicts without the DOM. +2. Test animation start, interruption, cancellation, completion, and zero-duration paths with a deterministic driver. +3. Server-render and hydrate initial-style cases while failing on warnings. +4. Test single and nested presence, reentry, rapid toggles, reorder, and disposal. +5. Count frames, listeners, observers, animation objects, WebGL resources, and retained owners after repeated navigation. +6. Test reduced motion before mount and preference changes after mount. +7. Test background tab, offscreen, resize, context failure, texture failure, and route removal. +8. Measure long tasks, frame time, memory, layout shift, and shipped animation code. +9. Run keyboard/focus checks while elements enter, exit, reorder, and transform. +10. Claim only the capabilities covered by executable tests. + +## Sources and freshness + +- Uploaded `solid-motion-experiments.zip` source and research notes, reviewed 2026-07-17. Status: experimental; not proof of full Motion parity. +- Uploaded `kaiju-website(6).zip` depth/WebGL and native FAQ implementations, reviewed 2026-07-17. +- Uploaded `solid-primitives(2).zip`, including presence, transition, scheduling, visibility, observer, and lifecycle packages, reviewed 2026-07-17. +- Solid lifecycle docs: https://docs.solidjs.com/reference/lifecycle/on-mount and https://docs.solidjs.com/reference/lifecycle/on-cleanup (reviewed 2026-07-17). +- Browser animation and WebGL behavior varies by runtime. Verify target-browser policy and installed library source before claiming gesture, presence, layout, or context-recovery support. diff --git a/skills/build-web/references/renderers.md b/skills/build-web/references/renderers.md index 01e1bee..ff0a56f 100644 --- a/skills/build-web/references/renderers.md +++ b/skills/build-web/references/renderers.md @@ -1,46 +1,224 @@ # Renderers, islands, SSR, and hydration -## Native before island +Use this reference when a route mixes Astro, Solid, React, plain scripts, custom elements, server islands, or client-side navigation. The goal is one DOM owner per subtree and deterministic server/client output. -Use semantic HTML such as disclosure, dialog, form, navigation, and media -features when it meets the behavior. Hydrate only when reactive component -ownership materially helps. +## Contents -In an Astro site: +- Renderer evidence +- Escalation ladder +- Astro client and server directives +- Solid runtime contract +- React and cross-renderer boundaries +- SSR and hydration invariants +- Navigation and lifetime +- Failure signatures +- Verification +- Sources and freshness -- Astro owns route, layout, content, metadata, and static structure; -- a narrow script or custom element can own isolated DOM behavior; -- a Solid or React island owns a renderer-specific interactive subtree; -- the client directive should match urgency and visibility; -- `client:only` requires enough renderer information because Astro cannot infer - a renderer from skipped server output. +## Renderer evidence -## Renderer contract +Before selecting a renderer, inspect: -Align: +- the direct import from the `.astro` file; +- the installed `@astrojs/<renderer>` integration; +- JSX transform and type settings; +- component registry target and behavioral primitive; +- icon plugin compiler; +- server renderer and hydration test environment; +- client directive and urgency; +- whether server HTML is required; +- resources acquired after mount and cleanup on disposal. -- integration/plugin; -- JSX compiler and types; -- icon compiler; -- component registry and generated code; -- client directive; -- server adapter; -- auth client/server binding; -- test environment. +Type-compatible JSX does not prove that a component belongs to the configured renderer. React, Solid, Preact, and Astro components can expose similar file extensions while requiring different runtimes and lifecycle rules. -Do not copy a React shadcn example into Solid or assume a generic auth client is -the correct TanStack Start binding. +## Escalation ladder -## Solid semantics +Choose the lowest owner that meets the behavior: -Solid component props are live access paths. Do not destructure reactive props -into stale values. Use signals for mutable local state, memos for derived state, -and effects only for synchronization with external systems. +1. Semantic HTML for links, buttons, forms, disclosure, media, tables, progress, and headings. +2. CSS for presentation, simple transitions, responsive state, and preference queries. +3. A page-local Astro script for narrow DOM behavior that does not need component state. +4. A custom element when behavior must be reusable across documents without a framework owner. +5. A framework island when fine-grained state, renderer primitives, or complex lifecycle justify hydration. +6. A server island when personalized/dynamic server HTML can be deferred independently of a cacheable page. +7. A full application router when long-lived navigation and application state dominate the surface. -Every listener, observer, frame, timer, subscription, animation, singleton root, -and retained owner needs explicit lifetime and cleanup. Browser work starts after -mount. Server output and initial client state must be deterministic to avoid -hydration mismatch and first-paint flashes. +The Kaiju homepage demonstrates both sides: native `<details>` owns FAQ disclosure, while a Solid island owns a WebGL depth scene. Hydrating the FAQ would add a renderer without adding capability. -Solid version differences belong behind an adapter until tests prove the new -runtime behavior. A type-compatible prototype is not production parity. +## Astro client and server directives + +Framework components render static HTML by default. A `client:*` directive hydrates a directly imported framework component: + +| Directive | When it runs | Appropriate use | Main risk | +|---|---|---|---| +| `client:load` | Immediately | Above-fold interaction required at load | Competes with critical work | +| `client:idle` | Idle/load fallback; optional timeout | Noncritical interactive UI | Interaction may arrive before hydration | +| `client:visible` | Intersection observer; optional root margin | Below-fold or expensive island | Visibility is initial hydration, not later suspension | +| `client:media` | Matching media query | Truly media-specific functionality | Duplicates CSS visibility policy | +| `client:only="react"` / `"solid-js"` | Client render only | Browser-only component with no useful server output | Blank/fallback-first content and renderer hint required | +| `server:defer` | Independent request after shell | Personalized server fragment in cacheable page | Adapter/runtime and fallback required | + +`client:only` skips server rendering, so Astro cannot infer the renderer. Use the framework string documented for the installed integration and provide useful `slot="fallback"` content. Do not use `client:only` merely to silence an SSR error; isolate and fix nondeterministic or browser-only behavior. + +```astro +--- +import SearchPanel from "../components/SearchPanel.tsx"; +import AccountSummary from "../components/AccountSummary.astro"; +--- + +<SearchPanel client:load initialSearch={validatedSearch} /> +<AccountSummary server:defer> + <div slot="fallback" aria-busy="true">Loading account summary…</div> +</AccountSummary> +``` + +The fallback must preserve layout and communicate state. A server island is not a client framework island: it defers server rendering and still needs an adapter. + +## Solid runtime contract + +Solid components execute once to create a reactive graph; they do not rerun like React functions. Preserve live access paths: + +```tsx +import { createMemo, onCleanup, onMount, splitProps } from "solid-js"; + +function Chart(props: { points: readonly number[]; class?: string }) { + const [local, rest] = splitProps(props, ["points"]); + const maximum = createMemo(() => Math.max(0, ...local.points)); + + let canvas!: HTMLCanvasElement; + onMount(() => { + const observer = new ResizeObserver(() => draw(canvas, local.points)); + observer.observe(canvas); + onCleanup(() => observer.disconnect()); + }); + + return <canvas ref={canvas} data-max={maximum()} {...rest} />; +} +``` + +Do not write `const { points } = props` or `const points = props.points` when the value must remain reactive. Use direct property access, an accessor, `splitProps`, or `mergeProps`. Use: + +- signals for mutable local state; +- memos for pure derived state reused by consumers; +- effects for synchronization with external systems, not for ordinary derivation; +- `onMount` for browser-only setup after initial render; +- `onCleanup` in the owning reactive scope for listeners, observers, timers, frames, subscriptions, roots, and animations. + +Returning a function from `onMount` is not Solid cleanup. Register `onCleanup` explicitly. Cleanup follows reactive ownership, which is not always the same as physical DOM insertion/removal; retained presence systems require a deliberate owner-retention design. + +## React and cross-renderer boundaries + +React and Solid may coexist at route level, but never mount both into the same DOM subtree. Keep shared contracts serializable or framework-neutral: + +```text +server/domain data + -> validated plain view model + -> Astro document + -> React island A + -> Solid island B +``` + +Do not pass renderer-specific contexts, elements, hooks, signals, refs, or event objects across that boundary. If both islands need the same remote data, either render it into their initial models or define a server/query contract; do not synchronize through hidden DOM mutation. + +Renderer-specific libraries must align: + +- Kobalte and Corvu are Solid primitives; Base UI is React-oriented. +- A shadcn registry can generate different source for different renderers. +- Unplugin Icons needs a compiler/JSX mode matching the consuming renderer. +- `client:only="react"` is not valid evidence for a Solid file and vice versa. +- Auth clients may have generic, React, Solid, TanStack React, and TanStack Solid entrypoints with different provider requirements. + +## SSR and hydration invariants + +Server HTML and the first client render must agree on structure, ids, attributes, and initial styles. Sources of mismatch include: + +- `Date.now()`, randomness, locale/time zone, unstable iteration order; +- reading `window`, media queries, storage, viewport, or permissions during render; +- environment values that differ between server and client; +- async data fetched independently on each side; +- animation libraries applying client initial styles that were absent in server HTML; +- invalid HTML repaired differently by the browser; +- component libraries generating unstable ids; +- auth/session capability determined again in the browser. + +Use an explicit server snapshot: + +```tsx +type InitialPreferences = { + colorScheme: "light" | "dark"; + reducedMotion: boolean; +}; + +function PreferencesIsland(props: { initial: InitialPreferences }) { + const [preferences, setPreferences] = createSignal(props.initial); + + onMount(() => { + // Reconcile browser-only preferences after hydration; do not change the + // first render that must match the server snapshot. + setPreferences(readBrowserPreferences(props.initial)); + }); + + return <PreferenceControls value={preferences()} />; +} +``` + +For motion, compute the initial style with a pure render-time function and start the runtime after mount. The uploaded Solid motion experiment tests serialized initial, `initial={false}`, omitted-initial, SVG, CSS-variable, and transform cases by rendering server markup and hydrating while collecting mismatch diagnostics. + +Do not suppress hydration warnings without proving that the differing subtree is intentionally non-authoritative. + +## Navigation and lifetime + +Astro client-side navigation can execute page initialization repeatedly. Register idempotently or bind cleanup to navigation: + +```ts +let pageController: AbortController | undefined; + +document.addEventListener("astro:page-load", () => { + pageController?.abort(); + pageController = new AbortController(); + + document.addEventListener("click", handleDelegatedClick, { + signal: pageController.signal, + }); +}); +``` + +Prefer event delegation to adding listeners to every link on every page load. Do not prevent default behavior for all hash links without preserving focus, history, reduced motion, and invalid-selector handling. Astro's client router already provides route announcement and reduced-motion behavior for its transitions; custom code must not fight those contracts. + +`transition:persist` retains an element or island across navigation. It changes lifetime and prop behavior. Decide whether new props should flow; `transition:persist-props` retains existing props. Persisting every shell island can retain stale auth, subscriptions, and memory. + +## Failure signatures + +| Signature | Likely contract error | Inspect next | +|---|---|---| +| `NoMatchingRenderer` or `client:only` failure | Missing/wrong integration or hint | Direct import, integration, renderer string | +| Hydration mismatch/first-paint flash | Nondeterministic initial state | Server HTML versus first client render | +| Solid prop stops updating | Reactive prop destructured/read eagerly | Props access paths and `splitProps` | +| Duplicate click after navigation | Repeated listener registration | `astro:page-load`, delegation, cleanup | +| Browser API error during build | DOM access during render/module evaluation | Mount gate and server import graph | +| Static content absent until JS | Unnecessary client-only island | Native/Astro server output path | +| Island hydrates but remains stale | Initial snapshot copied without synchronization | State ownership and update contract | +| Memory/CPU grows across routes | Resource or persisted island not disposed | Frames, observers, listeners, roots | +| Component renders with wrong behavior | Renderer-specific primitive copied | Registry and peer dependencies | + +## Verification + +1. Run framework typecheck plus the production build, not only editor diagnostics. +2. Capture raw server HTML and assert required content, initial styles, headings, and fallback states. +3. Hydrate representative routes while collecting console mismatch/warning output. +4. Disable JavaScript and evaluate the stated progressive-enhancement contract. +5. Navigate repeatedly and assert listener/observer/frame/subscription counts return to baseline. +6. Test slow hydration, interaction before hydration, fallback rendering, and island load failure. +7. Verify every `client:only` has a renderer hint and meaningful fallback. +8. Inspect built chunks to confirm static components did not become accidental islands. +9. Test reduced motion, route announcement, focus movement, and back/forward navigation. +10. Run renderer-specific component tests in the same JSX/test environment as production. + +## Sources and freshness + +- Astro template directives: https://docs.astro.build/en/reference/directives-reference/ (reviewed 2026-07-17). +- Astro view transitions: https://docs.astro.build/en/guides/view-transitions/ (reviewed 2026-07-17). +- Solid props: https://docs.solidjs.com/concepts/components/props (reviewed 2026-07-17). +- Solid `onMount` and `onCleanup`: https://docs.solidjs.com/reference/lifecycle/on-mount and https://docs.solidjs.com/reference/lifecycle/on-cleanup (reviewed 2026-07-17). +- Uploaded evidence: `kaiju-website(6).zip`, `solid-motion-experiments.zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`. +- Client directives, transition persistence, Solid runtime details, and experimental renderer adapters are version-sensitive. Verify installed versions before using an option not shown in repository source. diff --git a/skills/build-web/references/security.md b/skills/build-web/references/security.md index e56ed8e..9e1322d 100644 --- a/skills/build-web/references/security.md +++ b/skills/build-web/references/security.md @@ -1,30 +1,219 @@ -# Web security and connected systems +# Web security and connected-system boundaries -## Treat endpoints as production boundaries +Use this reference for pages, forms, server functions, Astro endpoints, webhooks, auth routes, embeds, CMS rendering, file/media flows, and client-side navigation. A UI that renders correctly can still leak tenant data, secrets, or executable content. -For forms, webhooks, server functions, and auth routes inspect: +## Contents -- authentication and provider signature verification; -- authorization and organization/tenant scope; -- method semantics and idempotency; -- input schema and size limits; -- CSRF, origin, CORS, cookie, and credential behavior; -- secret and PII logging; -- safe public errors and redacted diagnostic causes; -- rate limiting, replay, retry, and abuse behavior. +- Threat and authority inventory +- Server/client boundary +- Output and content safety +- Forms and mutations +- Authentication and authorization +- Cookies, CORS, CSRF, and origins +- Headers and caching +- Webhooks and provider calls +- Browser and third-party resources +- Failure handling and logging +- Verification +- Sources and freshness -Never alias a mutating POST handler to GET. Never log complete environment -objects, provider keys, cookies, raw contact submissions, or message bodies. +## Threat and authority inventory -## Auth binding +For every externally reachable route record: -Base URL, base path, issuer, handler mount, discovery endpoints, cookie scope, -trusted origins, credentialed fetch, server framework plugin, and browser client -plugin form one contract. Authentication does not prove authorization. +```text +Method and path: +Caller identity: +Required organization/tenant/role: +Input schema and size limit: +Side effects and idempotency key: +Credentials/cookies/origin behavior: +External providers: +Public response/error shape: +Diagnostic fields and redaction: +Cache policy: +Rate/replay/abuse policy: +Verification source/signature: +``` -## Connected-system verification +Trace authority from the server-observed identity into the database/provider query. A client-provided `organizationId`, hidden button, route guard, or disabled control is not authorization. -A UI can render correctly while auth, CORS, cookies, OpenAPI generation, CMS -cache, or server adapters fail. Run representative browser requests through the -deployed routing boundary, including failure and cross-origin cases where -claimed. +## Server/client boundary + +Server secrets and authority must not enter client bundles or serialized props. Inspect import reachability, not only variable prefixes. + +The finance code demonstrates a useful pattern: server code resolves enabled auth capabilities, then passes a small client-safe list of provider ids to a React island. The browser does not import environment-backed server configuration to decide what auth methods exist. + +```ts +type PublicAuthCapabilities = { + socialProviders: readonly ("github" | "google")[]; + allowPasskey: boolean; +}; +``` + +Keep server-only modules in explicit server paths and add a build/test that importing client entrypoints does not require production secrets. Never serialize complete environment objects, session records, database errors, provider payloads, or internal stack traces into HTML. + +## Output and content safety + +Framework interpolation escapes text by default; raw HTML APIs change the contract. For Astro `set:html`, React `dangerouslySetInnerHTML`, CMS rich text, Markdown plugins, SVG, and search snippets: + +1. Identify whether the value is trusted source code, sanitized rich text, or untrusted user/provider input. +2. Parse or sanitize at one named boundary with a defined allowlist. +3. Preserve structured content as data instead of concatenating HTML when possible. +4. Test script elements, event attributes, `javascript:` URLs, SVG/script combinations, malformed markup, and encoded payloads. +5. Apply Content Security Policy as defense in depth, not a replacement for output encoding. + +The ThunderStrike CMS adapter's plain-text fallback escapes `&`, `<`, `>`, and quotes before creating HTML. That is evidence for a narrow fallback, not a complete Portable Text renderer. Full rich text requires mark, link, embed, and custom-block handling with explicit safe renderers. + +Do not render provider search snippets containing `<mark>` through a raw HTML sink without proving how all other tags/attributes were removed. + +## Forms and mutations + +Every mutation requires server-side validation and authorization even if the client uses Zod or a form library. Define: + +- accepted content types and maximum total/field/file size; +- duplicate/double-submit policy; +- idempotency or optimistic concurrency where retries occur; +- CSRF/origin policy for cookie-authenticated requests; +- spam/rate limits for public forms; +- safe redirect/callback URL allowlist; +- public validation versus internal diagnostic detail; +- audit fields that exclude secrets and excessive PII. + +Do not expose mutation through GET or alias the same handler to GET and POST. Do not disable the submit button before the user can discover validation errors; disable or gate after a valid submission begins to prevent duplicates. + +For file uploads verify MIME by content where needed, limit size/count, randomize storage names, prevent path traversal, isolate public/private storage, and scan or transform risky formats before serving. + +## Authentication and authorization + +Authentication contract: + +- issuer/base URL and handler mount agree; +- cookie domain, path, `Secure`, `HttpOnly`, and `SameSite` fit the topology; +- trusted origins and callback URLs are finite; +- provider/client plugins match the server configuration; +- discovery metadata points to reachable endpoints; +- credentialed browser requests use the intended origin and CORS policy; +- session rotation, expiration, revocation, and organization switching are tested. + +Authorization contract: + +```ts +const session = await requireSession(request); +const scope = await requireOrganizationMembership(session.user.id, routeOrg); +const result = await repository.search({ organizationId: scope.organizationId, filters }); +``` + +Do not accept an organization from the body and compare it only in the UI. Put tenant scope into the repository query or service policy so missing a later filter is harder. + +Sensitive account changes should require appropriate freshness/re-authentication and invalidate relevant sessions when the product policy requires it. + +## Cookies, CORS, CSRF, and origins + +CORS controls which browser origins may read responses; it is not authentication. For credentialed cross-origin requests: + +- use a specific allowed origin, never `*`; +- set `Access-Control-Allow-Credentials: true` only where needed; +- handle preflight methods/headers explicitly; +- configure cookies so they are actually sent; +- keep allowed origins synchronized with auth trusted origins; +- verify both allowed and rejected origins in a browser-equivalent flow. + +Cookie-authenticated mutations require a CSRF defense such as strict same-site topology plus origin checks or a correctly implemented token strategy. Evaluate top-level navigation, subdomains, embedded contexts, and OAuth callbacks before assuming `SameSite` alone is enough. + +Validate redirect destinations against an allowlist or same-origin policy. URL parsing must reject scheme-relative and encoded bypasses. + +## Headers and caching + +Set policy at the deployment/server boundary and verify the final response: + +- `Content-Security-Policy` appropriate to scripts, styles, images, fonts, frames, and connections; +- `X-Content-Type-Options: nosniff`; +- `Referrer-Policy`; +- `Permissions-Policy` for unavailable capabilities; +- clickjacking policy through CSP `frame-ancestors` (and `X-Frame-Options` as legacy defense when appropriate); +- HSTS only on production HTTPS domains with a deliberate subdomain/preload decision; +- cross-origin isolation headers only when required and compatible with dependencies. + +`X-XSS-Protection` is obsolete and is not a substitute for CSP/encoding. Do not present its presence as a modern XSS control. + +Personalized/auth routes should normally use `private, no-store` or another explicitly justified private policy. The finance middleware classifies app/auth routes then sets cache behavior. Verify that route intent cannot misclassify a personalized route as public. Public static assets and documents can use cache validators or immutable fingerprinted caching. + +Never cache one tenant's HTML under a key that omits identity/tenant. `Vary` must match any request header that changes a shared-cache response. + +## Webhooks and provider calls + +Webhook sequence: + +```text +raw request bytes + -> size limit + -> timestamp/replay window + -> provider signature over documented bytes + -> constant-time verification + -> parse and schema validate + -> idempotency/event-id check + -> authorized side effect + -> redacted audit record + -> provider-compatible status +``` + +Do not parse/re-serialize before signature verification when the provider signs raw bytes. Store enough event identity to make retries safe. + +The uploaded ThunderStrike webhook is counterexample evidence because it logs the API key and complete environment, logs payload-derived PII, permits unknown fields broadly, and does not demonstrate provider signature verification. Its package mappings may be useful domain data after validation; the endpoint architecture is not reusable. + +Provider error messages may contain request data or internal identifiers. Map them to a stable public error and retain a redacted cause in structured diagnostics. + +## Browser and third-party resources + +Treat analytics, embeds, iframes, scripts, OAuth popups, WebGL textures, fonts, and icon SVGs as supply-chain and privacy boundaries: + +- pin or control dependency versions and provenance; +- minimize third-party origins in CSP; +- sandbox iframes to the minimum capability and provide a title; +- use `rel="noopener noreferrer"` where external opener/referrer policy requires it; +- avoid injecting arbitrary SVG or remote script content; +- obtain consent before nonessential tracking where applicable; +- never put tokens or PII in analytics labels, page titles, URLs, or referrers; +- validate `postMessage` origin, source, and schema; +- clean up embedded resources on navigation. + +Do not write a global handler that mutates every external link after each client navigation if the links can be rendered safely in the first place. + +## Failure handling and logging + +Public errors should be useful and non-sensitive: + +```json +{ + "type": "https://example.invalid/problems/invalid-form", + "title": "The form could not be submitted", + "status": 400, + "errors": { "email": ["Enter a valid email address"] }, + "correlationId": "..." +} +``` + +Diagnostics can include route, operation, correlation id, safe actor/tenant identifiers, duration, provider status class, retry count, and redacted cause. Do not include authorization headers, cookies, passwords, tokens, complete payloads, or environment objects. + +Distinguish 401 (authentication required/invalid), 403 (authenticated but forbidden), 404 (including deliberate anti-enumeration policy), 409 (conflict), 422/400 (validation according to project contract), 429, and 5xx. + +## Verification + +1. Enumerate routes/methods and test authentication plus tenant/role authorization. +2. Send valid, malformed, oversized, duplicate, replayed, and cross-origin requests. +3. Verify cookie attributes and actual credentialed browser behavior. +4. Test CSRF and redirect allowlists with encoded and scheme-relative inputs. +5. Inject XSS payloads into every raw/rich content boundary and inspect rendered DOM. +6. Inspect final deployed CSP, cache, frame, MIME, referrer, and permissions headers. +7. Verify public/client bundles and HTML contain no server secret names or values. +8. Search logs/test capture for secrets, cookies, tokens, raw payloads, and PII. +9. Test webhook signature over exact raw bytes, replay window, idempotency, and retry. +10. Attempt cross-tenant reads and mutations through the real repository/service path. + +## Sources and freshness + +- Uploaded evidence reviewed 2026-07-17: `new-finance-app(1).zip`, `kaiju-site-scope(17).zip`, `better-auth.zip`, `thunderstrike-blog(4).zip`. +- Modern Web Guidance security and forms guides retrieved 2026-07-17. +- Better Auth details belong to `build-web-apps/references/auth.md`; verify current official documentation before using plugin-specific options. +- Browser header support, cookie rules, auth provider behavior, and framework raw-HTML APIs change. Verify target runtime and installed versions. This reference does not replace a project-specific threat model. diff --git a/skills/build-web/references/surfaces.md b/skills/build-web/references/surfaces.md index 304f1e2..23de613 100644 --- a/skills/build-web/references/surfaces.md +++ b/skills/build-web/references/surfaces.md @@ -1,32 +1,203 @@ -# Web surface classification - -| Surface | Primary owner | Typical rendering | State emphasis | -|---|---|---|---| -| Marketing/content | `build-sites` | Static or selective server rendering | Content and minimal interaction | -| Documentation/API reference | `build-sites` | Static generation where possible | Source-owned documents | -| Runtime CMS | `build-sites` | Server rendering with cache policy | CMS mapped to stable view models | -| Product application | `build-web-apps` | SSR plus client reactivity | URL, query cache, local, session | -| Hybrid route group | Both, with route-level ownership | Mixed | Explicit boundaries | -| Browser extension | Separate evidence required | Extension runtime-specific | Manifest, contexts, permissions | - -## Classification questions - -- Must the page reflect request-time identity or live data? -- Can content be built ahead of time? -- Which interactions need long-lived reactive ownership? -- Which state must be shareable in the URL? -- Which data is cached remote state rather than UI state? -- Which component renderer owns the DOM? -- Does the deployment host require an adapter? -- What happens without JavaScript, WebGL, network, auth, or a third-party CMS? - -Global server output does not mean every page must render at request time. A -project may prerender its homepage while serving other routes dynamically. -Static output can still generate rich API documentation from build-time OpenAPI -factories. - -## Import-graph rule - -Inspect reachable entrypoints. Unused components, migration inputs, abandoned -experiments, and installed dependencies are evidence of exploration, not -endorsed architecture. +# Web surface classification and ownership + +Use this reference before choosing a framework, renderer, hydration directive, state library, or deployment adapter. The repository name and dependency list are not enough. Classify each reachable route and subsystem from runtime evidence. + +## Contents + +- Required evidence inventory +- Surface and route classification +- Ownership decisions +- Worked repository cases +- Decision record +- Failure boundaries +- Verification +- Sources and freshness + +## Required evidence inventory + +Inspect the active graph before proposing architecture: + +1. Route and endpoint entrypoints, including catch-alls, middleware, layouts, generated route trees, content configuration, and server functions. +2. Build output, adapter, deployment runtime, prerender exports, cache middleware, redirects, and headers. +3. Framework integrations and which `.astro`, `.tsx`, `.jsx`, custom-element, or plain-script files are actually imported. +4. Data sources: local content, build-time factories, request-time CMS, auth/session, query cache, database, browser persistence, and URL parameters. +5. Client resources: listeners, observers, animation frames, workers, WebGL contexts, media queries, subscriptions, retained roots, and cleanup. +6. Failure contracts: no JavaScript, no network, stale CMS, expired session, blocked storage, no WebGL, asset failure, and unsupported browser feature. +7. Connected systems: auth mount and base URL, CORS and cookies, CMS cache hints, OpenAPI factories, icon/font compilers, deployment adapter, and test runtime. + +Treat unused files, installed packages, research notes, migration inputs, and commented prototypes as evidence of exploration. They become architecture only when reachable from an entrypoint or named as an intended target by the task. + +Useful inspection commands include: + +```sh +rg --files | rg '(astro\.config|package\.json|src/pages|src/routes|middleware|content\.config|live\.config|components\.json)' +rg -n 'client:(load|idle|visible|media|only)|server:defer|prerender|output:|adapter:' . +rg -n 'addEventListener|requestAnimationFrame|ResizeObserver|IntersectionObserver|createRoot|subscribe' src +rg -n 'from ["\x27](react|solid-js|@astrojs/|@tanstack/|better-auth)' src +``` + +## Surface and route classification + +Classify at route or route-group granularity. A repository can contain several surfaces. + +| Surface | Default document owner | Typical rendering | Durable state owner | Escalation signal | +|---|---|---|---|---| +| Marketing or editorial | Astro/static document layer | Static HTML; isolated hydration | Content source and URL | Request-time personalization or live CMS preview | +| Documentation/API reference | Astro/static document layer | Build-time pages and JSON artifacts | Service-owned schemas/factories | Private viewer-specific documentation | +| Runtime CMS site | Astro server routes | Request-time or cached SSR | CMS mapped through project view models | Author preview, live collections, dynamic plugins | +| Product application | Application router | SSR plus client reactivity | URL, query cache, local UI, session, server | Long-lived workflows or offline synchronization | +| Hybrid product/marketing | Route-level split | Static public routes plus SSR app routes | Different owner per route group | Shared shell must not erase cache/security differences | +| Embedded widget | Host document plus isolated component | Script/custom element or island | Explicit embed instance | Cross-origin messaging, versioned embed contract | +| Browser extension | Extension runtime | Manifest/context-specific | Extension storage/background context | Content-script, service-worker, and permission boundaries | + +Ask these questions for every route: + +- Can the response be generated without the incoming request? +- Does it depend on identity, cookie, organization, entitlement, locale, preview token, or live data? +- Must the state survive refresh, be shareable, or participate in browser history? +- Is the data remote server state, local interaction state, session state, or derived display state? +- Which renderer owns the DOM subtree after hydration? +- What remains usable before JavaScript loads or when it fails? +- What cache policy is safe for this exact response? +- Does a deployment adapter provide a required capability, or was one added without a route need? + +Do not infer that `output: "server"` makes every route dynamic. An Astro server project can and should set `export const prerender = true` on static routes. Conversely, static output cannot read request-time cookies during page generation. + +## Ownership decisions + +Write down one primary owner per concern. Split ownership by boundary, not by convenience. + +| Concern | Valid owner examples | Invalid split | +|---|---|---| +| Route and canonical URL | Astro route or application router | Component-local string concatenation in many islands | +| Request identity and authorization | Server middleware/service | Browser visibility checks presented as security | +| Shareable filters/sort/page | Validated URL schema | Duplicated URL, component state, and query state | +| Remote result lifecycle | Query cache or route loader | Copying query results into a second global store | +| Draft input | Form/component state | Writing every keystroke to remote cache | +| CMS provider record | Source adapter | Raw provider fields spread across pages | +| Layout and metadata | Document framework/layout | Each interactive island mutating document head | +| Interactive subtree | One renderer | Two frameworks mutating the same DOM nodes | +| Browser resource | Mounted owner with cleanup | Module singleton with no disposal policy | +| Stable static result | Server/build generator | Client fetching content already known at build time | + +A useful route decision record is: + +```text +Route: /search +Response: request-time, organization-scoped +Document owner: application router +URL state: query, technology filters, sort, page, page size +Remote state: canonical query-options factory keyed by organization + validated URL +Local state: query draft before debounce, selected row ids, open dialogs +Security: server derives organization; client cannot supply authority +Failure: route error boundary, retry, empty state, expired-session redirect +Verification: direct URL, reload, back/forward, cross-org request, SSR hydration +``` + +## Worked repository cases + +### Kaiju marketing site + +The active homepage imports Astro sections and uses native `<details>` for FAQ disclosure. A separate Solid `Faq.tsx` exists but is not imported by the homepage. The import graph therefore supports: + +- Astro ownership for page copy, headings, anchors, metadata, and section layout; +- native disclosure with no hydrated FAQ bundle; +- one narrow Solid island for the WebGL depth scene because it owns pointer input, media-query state, animation frames, WebGL, and cleanup; +- a static fallback image inside the island for context or texture failure. + +Do not conclude that every `.tsx` marketing component is an island. Do not hydrate the unused Solid FAQ merely because it is more abstract. + +### Kaiju documentation application + +The docs package uses static output and generates service pages plus OpenAPI JSON endpoints from service-owned factories. Rich API references do not require a runtime server when the schemas are build inputs. The product frontend remains a separate SSR application with auth and query state. + +This is a multi-surface monorepo: + +```text +service schemas/factories + -> static docs catalog and OpenAPI JSON + -> runtime API handlers + +product frontend + -> request-scoped auth and organization + -> URL/query/local state +``` + +### Finance application + +The finance app configures Astro server output and marks auth pages `prerender = false`. Server code derives enabled auth capabilities and passes only client-safe provider identifiers into React form islands. Middleware classifies public, auth, auth-required, and app routes before applying session and cache policy. + +The architecture is not “Astro versus React.” Astro owns request routing, layouts, middleware, response headers, and document output; React owns the form interaction subtree. + +### ThunderStrike CMS site + +The site has a runtime CMS integration and a project-owned `cms.ts` adapter. The adapter maps provider records into article, author, topic, category, image, and page models. This boundary is reusable. The webhook file is counterexample evidence: it logs environment/secrets and payload data, lacks a trustworthy verification boundary, and mixes extraction, provider mapping, and delivery. + +Never generalize “the repository uses this” into “this is approved.” Inspect behavior and tests. + +### Name-only extension or app classification + +A repository called “browser,” “extension,” or “desktop” is not enough. Require a manifest, background/service-worker entrypoint, content script, extension APIs, runtime permissions, or packaging config. If those are absent, report the mismatch instead of inventing an extension architecture. + +## Decision record + +Before implementation, record: + +```text +Surface and routes: +Active entrypoints: +Static/request-time/deferred boundaries: +Document owner: +Interactive owners: +URL state: +Remote/cache state: +Local/session state: +Auth and tenant authority: +Cache policy: +No-JS/failure fallback: +Deployment adapter requirement: +Connected-system contracts: +Validation commands: +Behavioral verification: +Unresolved evidence: +``` + +If evidence is unresolved, use conditional language and inspect the installed version or source. Do not fill a missing runtime contract with a familiar framework pattern. + +## Failure boundaries + +Define what the user sees and what operators can inspect for: + +- static build cannot reach content source; +- request-time CMS is unavailable or returns invalid rich text; +- session expires during navigation or submission; +- JavaScript bundle, island, WebGL texture, or font fails; +- query is stale, partially loaded, or invalidated during interaction; +- client hydration sees different time, randomness, locale, media, or storage state; +- repeated client navigation re-runs page initialization; +- deployment adapter lacks streaming, image, session, or runtime API support; +- user opens a deep link with malformed filters; +- authenticated user attempts another tenant's resource. + +Do not collapse loading, empty, unavailable, unauthorized, forbidden, partial, and stale into one blank view. + +## Verification + +Verify classification with evidence, not a prose review: + +1. Build static and server targets that the repository claims. +2. Inspect generated routes and output artifacts. +3. Fetch representative HTML before hydration and confirm required content exists. +4. Exercise direct navigation, reload, back/forward, and copied deep links. +5. Disable JavaScript for surfaces claiming progressive enhancement. +6. Capture cache and security headers for public and personalized routes. +7. Count shipped JavaScript/islands and compare with the ownership record. +8. Run one failure for each connected system and verify the intended boundary owns it. +9. Trace an authorization decision from request identity through the server query. + +## Sources and freshness + +- Uploaded evidence reviewed 2026-07-17: `kaiju-website(6).zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `old-finance-app(1).zip`, and `thunderstrike-blog(4).zip`. +- Astro on-demand rendering: https://docs.astro.build/en/guides/on-demand-rendering/ (reviewed 2026-07-17). +- Astro server islands: https://docs.astro.build/en/guides/server-islands/ (reviewed 2026-07-17). +- Treat deployment adapters, experimental Astro options, and application-local route conventions as version-sensitive. Verify the installed manifest and adapter documentation before copying configuration. diff --git a/skills/build-web/references/verification.md b/skills/build-web/references/verification.md index a1d4a5e..4494c6a 100644 --- a/skills/build-web/references/verification.md +++ b/skills/build-web/references/verification.md @@ -1,36 +1,250 @@ -# Web verification +# Web verification manual -## Build matrix +Use this reference to prove a web change across build artifacts, server output, hydration, browser behavior, accessibility, security, connected systems, performance, and resource lifetime. A passing typecheck is not a browser result. -Run the repository-owned static and server builds separately. Verify the selected -deployment adapter, renderer integrations, generated routes, source maps, assets, -and environment contract. Typecheck does not replace a production build. +## Contents -## SSR and hydration +- Verification target inventory +- Layered matrix +- Build and artifact checks +- Server and hydration checks +- Browser behavior +- Accessibility +- Security and connected systems +- Performance and lifetime +- Visual and content verification +- Failure injection +- Evidence report +- Sources and freshness -Capture server HTML and hydrate it while collecting mismatch diagnostics. Cover -initial styles, media queries, local persistence, random/time values, client-only -components, and failure fallbacks. +## Verification target inventory + +Before running commands record: + +```text +Routes changed: +Static/server/deferred output: +Renderer and island directives: +URL/query/local/session state: +Forms and mutations: +Auth/tenant boundary: +CMS/API/provider dependencies: +Assets/icons/fonts/motion: +Deployment adapter: +Browser support policy: +Expected no-JS behavior: +Authorized test targets: +``` + +Read repository-owned scripts and CI before inventing commands. Keep formatters scoped away from authored Markdown unless the user requests document formatting. + +## Layered matrix + +| Layer | What it proves | What it does not prove | +|---|---|---| +| Schema/typecheck | Static contracts and reachable type errors | Runtime DOM, CSS, network, adapter behavior | +| Unit test | Pure policy/state cases | Real framework integration unless mounted accordingly | +| Production build | Compiler, bundler, routes, adapter output | Deployed runtime and user behavior | +| Server integration | HTTP, middleware, cookies, services | Hydration, focus, layout, browser APIs | +| Hydration test | Server/client structural agreement | Full browser accessibility/performance | +| Browser test | User flows and platform behavior | All target environments unless matrixed | +| Visual inspection | Layout/aesthetic result | Semantics, auth, hidden failures | +| Deployment smoke | Host adapter and configuration | Broad regression suite | + +Do not substitute a lower layer for a higher-risk claim. + +## Build and artifact checks + +Run each output mode the repository claims. For Astro this often includes: + +```sh +pnpm astro check +pnpm build +pnpm preview +``` + +Use the actual package manager/scripts. Then inspect: + +- route manifest and generated paths; +- prerendered versus server routes; +- server adapter output and startup entrypoint; +- OpenAPI/feed/sitemap/robots/redirect/header artifacts; +- client chunks per island and duplicate framework runtimes; +- source maps and accidental secret/environment serialization; +- icon collection and font assets; +- image dimensions/formats and static fallback assets; +- cache-busting filenames and public base paths. + +A successful server build can still misclassify a route or ship an empty client-only placeholder. + +## Server and hydration checks + +Capture raw HTML before client JavaScript: + +```sh +curl -fsS -D /tmp/headers.txt http://127.0.0.1:4321/search?q=solid > /tmp/page.html +``` + +Assert required headings, content, canonical metadata, form labels, fallback states, and initial styles. For personalized pages, use safe local fixtures and authorized session setup. + +Hydration harness requirements: + +1. Render through the actual server renderer with hydration markers enabled. +2. Execute required hydration bootstrap. +3. Hydrate the same component/route and capture `console.error`/`console.warn`. +4. Fail on hydration/mismatch diagnostics. +5. Assert initial style/DOM before hydration and interactive state after. +6. Dispose and confirm external resources stop. + +Cover time/randomness, media queries, local storage, auth capability props, SVG, CSS variables, transforms, client-only fallback, and invalid HTML. ## Browser behavior -Test keyboard navigation, focus order/return, accessible names, validation and -errors, loading/empty/partial/stale states, responsive overflow, zoom, touch, -reduced motion, forced colors, and disabled JavaScript where the surface promises -progressive enhancement. +Test real sequences, not snapshots only: + +- direct deep link, reload, back, forward, and copied URL; +- invalid URL values and canonical default stripping; +- initial loading, stale refresh, empty, partial, offline, retry, and fatal error; +- interaction before and during hydration; +- double click/submit, reordered async response, cancellation, and route departure; +- repeated client navigation and page initialization; +- dialogs/sheets/menus with keyboard, pointer, touch, and focus return; +- responsive overflow and orientation change; +- browser zoom/text resize; +- reduced motion and preference changes; +- no JavaScript where progressive enhancement is claimed; +- image, font, icon, canvas/WebGL, CMS/API, and auth failure. + +For URL-owned filters assert that a copied link reconstructs query, filter, sort, and page. Changing a filter should reset dependent page state according to the documented policy. + +## Accessibility + +Automated checks are a starting point. Manually verify: + +- document language, unique title, landmarks, skip link, and heading outline; +- link/button semantics and accessible names; +- keyboard order, visible focus, modal trap and return; +- native form labels, autocomplete, field hints, errors, and error summary/focus; +- live-region restraint and announcements after async updates; +- tables with captions/header associations and chart alternatives; +- image alternatives and decorative SVG/canvas removal; +- zoom/reflow, contrast, forced colors, reduced motion; +- route announcement after client-side navigation; +- virtualized content focus/accessibility behavior. + +Inspect the browser accessibility tree for complex primitives and custom elements. DOM attributes alone may not reveal ElementInternals semantics. + +## Security and connected systems + +Exercise real boundaries: + +- allowed and rejected auth states; +- cross-tenant queries and mutations; +- cookie path/domain/SameSite/Secure behavior; +- trusted and untrusted origins with credentialed CORS; +- CSRF/origin checks; +- raw/rich HTML injection; +- cache headers for public and personalized routes; +- webhook signatures, replay, idempotency, and redaction; +- CMS cache hints and invalid/provider-error records; +- OpenAPI generation from the same factories as runtime services; +- deployment headers and redirects. + +Do not test production writes or shared targets without authorization. When a real target is blocked, report the exact unverified contract rather than calling the local check equivalent. + +## Performance and lifetime + +Record a before/after budget for: + +- transferred JavaScript and CSS by route/island; +- duplicate framework/runtime packages; +- icon modules and font files/preloads; +- LCP resource priority, CLS, INP/long tasks; +- offscreen content rendering and DOM size; +- hydration time and interaction-before-hydration; +- continuous animation/frame work; +- listener, observer, timer, subscription, worker, WebGL, and retained-root counts; +- memory across repeated navigation. + +Instrument lifecycle in tests: + +```ts +const add = vi.spyOn(window, "addEventListener"); +const remove = vi.spyOn(window, "removeEventListener"); + +for (let index = 0; index < 5; index += 1) { + const dispose = mountFeature(); + dispose(); +} + +expect(activeListenerBalance(add, remove)).toBe(0); +``` + +The exact helper is project-owned. Also spy on `requestAnimationFrame`/`cancelAnimationFrame`, observers, animations, and third-party destroy calls. + +When using `content-visibility: auto`, pair it with an appropriate `contain-intrinsic-size`, avoid above-fold critical content, and verify keyboard reachability across deferred sections. Treat support/fallback according to the project's browser policy. + +## Visual and content verification + +Use responsive screenshots when layout or styling changed. Cover at least narrow mobile, common desktop, and a stress width/content case. Inspect: + +- no clipping/overlap or horizontal page scroll; +- loading/empty/error states, not only happy data; +- fixed/sticky elements and safe areas; +- focus ring and destructive/disabled/invalid states; +- font fallback before/after load; +- image/canvas fallback; +- light/dark/forced colors; +- motion at normal and reduced preferences. + +For content sites verify canonical URL, title/description, social image, structured data, feed, sitemap, draft exclusion, pagination, taxonomy links, missing media, and broken internal links. + +## Failure injection + +At minimum inject: + +| Boundary | Failure | +|---|---| +| Server data | timeout, invalid schema, 401/403/404/409/429/500 | +| Client query | offline, stale cache, reordered responses | +| Form | invalid server response, double submit, route leave | +| CMS | missing relation/media, malformed rich text, outage | +| Auth | expired session, organization switch, bad callback | +| Assets | image/font/icon load failure | +| Motion | reduced motion, background tab, cancel during exit | +| WebGL | no context, context/texture failure, resize/unmount | +| Navigation | repeated route events, back/forward, hash target | +| Deployment | missing env/binding, cold startup, wrong base URL | + +The assertion must distinguish the intended safe fallback from a silent blank view. + +## Evidence report + +Report: + +```text +Passed: +- command/test and the contract it proves + +Failed: +- command/test, failure signature, likely owner + +Blocked: +- exact missing authority/environment/dependency +- what remains unverified -Dispatch navigation lifecycle events more than once. Assert handlers remain -idempotent and listener effects do not multiply. +Not run: +- deliberately out-of-scope checks -## Resource and performance checks +Artifacts: +- screenshots, logs, traces, bundle reports, generated output +``` -Count listener add/remove, observers, frame requests/cancellations, timers, -subscriptions, workers, WebGL contexts, and retained owners across mount/unmount -and navigation. Measure bundle composition, island hydration, icon inclusion, -font loading, long tasks, layout shift, and continuous background work. +Do not merge failed and blocked checks. Do not claim a deployed, cross-browser, accessibility, or performance result from source inspection alone. -## Connected systems +## Sources and freshness -Exercise real route loaders, server functions, session cookies, auth guards, -query identity/invalidation, CMS adapters, webhooks, CORS, and OpenAPI generation. -Use safe fixtures and never require production targets without authorization. +- Uploaded executable patterns reviewed 2026-07-17: `solid-motion-experiments.zip`, `kaiju-site-scope(17).zip`, `new-finance-app(1).zip`, `kaiju-website(6).zip`, `solid-primitives(2).zip`. +- Modern Web Guidance accessibility, security, forms, and deferred-rendering guides retrieved 2026-07-17. +- Astro view transitions: https://docs.astro.build/en/guides/view-transitions/ (reviewed 2026-07-17). +- Browser performance APIs and framework build commands are version-sensitive. Prefer repository-owned scripts and the target-browser policy. diff --git a/skills/build-workflows/SKILL.md b/skills/build-workflows/SKILL.md index d15bf4e..0d60c22 100644 --- a/skills/build-workflows/SKILL.md +++ b/skills/build-workflows/SKILL.md @@ -6,9 +6,10 @@ description: Design, implement, migrate, review, diagnose, or verify durable wor # Build durable workflows and pipelines Start from failure and recovery semantics. When active, `build-data` owns -storage-engine and artifact design and `build-apis` owns HTTP exposure. -Otherwise preserve those boundary checks locally. This skill owns durable -coordination and reachability. +storage-engine and artifact design, `build-apis` owns HTTP exposure, and +`build-libraries` owns reusable restart and checkpoint contracts. Otherwise +preserve those boundary checks locally. This skill owns persisted execution +authority, durable coordination, worker reachability, and operator recovery. ## Durability evidence ladder @@ -65,6 +66,11 @@ For every workflow, answer: - [durability.md](references/durability.md): engine selection, authority, identities, determinism, and completion. +- [effect-workflow.md](references/effect-workflow.md): Effect service/Layer + composition and the evidence-bounded `@effect/workflow` execution model. +- [temporal.md](references/temporal.md): Temporal clients, workers, workflows, + activities, messages, schedules, timeouts, retries, cancellation, and safe + deployment/versioning. - [control-plane.md](references/control-plane.md): definitions, registration, admission, projections, reachability, and API/CLI integration. - [atomicity.md](references/atomicity.md): transactions, idempotency, sequences, @@ -73,6 +79,10 @@ For every workflow, answer: schedules, cancellation, backpressure, and poison work. - [pipelines.md](references/pipelines.md): staged ingestion, provenance, bounded batches, checkpoints, manifests, and projections. +- [recovery.md](references/recovery.md): leases, checkpoints, resume, + reconciliation, replay, repair, and recovery testing. +- [streams.md](references/streams.md): workflow event streams, SSE delivery, + cursor authority, replay, backpressure, cancellation, and retention. - [failures.md](references/failures.md): evidence-grounded failure signatures. Do not claim durability until interruption and restart have been executed against diff --git a/skills/build-workflows/references/atomicity.md b/skills/build-workflows/references/atomicity.md index d19c8a3..715767d 100644 --- a/skills/build-workflows/references/atomicity.md +++ b/skills/build-workflows/references/atomicity.md @@ -1,50 +1,231 @@ -# Atomicity, idempotency, and reconciliation +# Atomicity, idempotency, outbox, and reconciliation -## Transaction boundaries +## Contents -Use one database transaction when execution creation, initial timeline append, -and queue insertion must either all commit or all disappear. Define unique -constraints for logical/idempotency identities and retry transaction conflicts. +- [Write the invariant first](#write-the-invariant-first) +- [Local transaction patterns](#local-transaction-patterns) +- [Idempotent starts and effects](#idempotent-starts-and-effects) +- [Transactional outbox](#transactional-outbox) +- [Inbox and deduplication](#inbox-and-deduplication) +- [Sagas and compensation](#sagas-and-compensation) +- [Reconciliation](#reconciliation) +- [Crash matrix](#crash-matrix) +- [Failure signatures](#failure-signatures) +- [Verification](#verification) -Allocate timeline sequence with a database-safe mechanism: sequence, locked -counter, atomic update, or constraint/retry. Never derive it from an unlocked -read count. +## Write the invariant first -## Cross-system gaps +For every state transition, state what must commit together. Examples: -Runtime start, database projection, external API effect, and queue acknowledgement -usually cannot share one transaction. For each gap define: +- execution record + first timeline event + ready queue item; +- winning wait transition + resume queue item + timeline event + execution status; +- lease claim + owner + expiry + attempt increment; +- domain mutation + outbox event; +- sink batch + checkpoint + manifest update; +- idempotency key + request fingerprint + accepted result identity. -- idempotent operation key; -- persisted intent and observed result; -- retry behavior; -- duplicate response handling; -- reconciliation query; -- operator repair path; -- safe terminal state. +If one database owns all rows, use one transaction and constraints. If systems +differ, state the gap and build idempotency plus reconciliation. -## Crash table +## Local transaction patterns -Test crashes: +Prefer store operations shaped around invariants, not individual tables: -- after execution insert but before enqueue; -- after engine start but before recording external ID; -- after effect success but before acknowledgement; -- after wait timeout state but before resume enqueue; -- after projection update but before timeline append; -- during cancellation and cleanup. +```ts +export interface WorkflowStore { + readonly acceptStart: (input: AcceptStartInput) => Promise<AcceptedStart> + readonly claimReady: (input: ClaimReadyInput) => Promise<readonly Lease[]> + readonly completeWaitAndEnqueueResume: ( + input: CompleteWaitInput, + ) => Promise<WaitCompletionResult> + readonly commitStage: (input: CommitStageInput) => Promise<StageCommit> +} +``` -The retry or repair must converge to one logical outcome. “At least once” does -not excuse duplicate non-idempotent effects. +An API exposing `updateWait`, `enqueue`, `appendTimeline`, and `updateExecution` +encourages partial commits. Keep low-level primitives internal or require an +explicit transaction handle. -## Idempotency scope +Use database-safe uniqueness and sequence allocation: -Name the scope and retention of keys. A key may be unique per tenant/workflow, -per endpoint, or globally. Store enough request identity to detect a reused key -with a conflicting payload. +```sql +create unique index workflow_idempotency_scope_key + on workflow_executions (tenant_id, workflow_name, idempotency_key); + +create unique index workflow_timeline_execution_sequence + on workflow_timeline (execution_id, sequence); +``` + +Allocate sequence with a locked counter, sequence, atomic update-returning, or +constraint/retry. Never use `existing.length + 1` or an unlocked `max + 1`. + +For queue claims use `FOR UPDATE SKIP LOCKED`, an atomic update-returning query, +or the engine's claim primitive. A read then conditional update races. + +## Idempotent starts and effects + +An idempotency record needs: + +- tenant/scope and operation name; +- client key; +- canonical input fingerprint; +- accepted logical execution/effect ID; +- status/result reference; +- created/expiry timestamps; +- policy for retry after permanent failure. + +```text +same key + same fingerprint -> return existing operation +same key + different fingerprint -> explicit conflict +expired key -> documented new-operation or conflict policy +``` + +Workflow-start idempotency and Activity idempotency are different. Give each +external effect its own operation key, preferably accepted by the provider. If a +provider times out ambiguously, query by that key before retrying. + +## Transactional outbox + +Use an outbox when a committed domain mutation must cause work/event publication: + +```text +BEGIN + update domain row + insert outbox event with unique event_id and aggregate revision +COMMIT + +publisher claims outbox row + -> publishes idempotently + -> records broker/engine acknowledgement + -> marks published +``` + +Outbox requirements: + +- stable event ID/type/schema version; +- aggregate/tenant identity and ordering key; +- serialized payload or reference; +- created/available timestamps; +- attempts, lease, last safe error; +- publication result/ack identity; +- retention and replay policy; +- atomic claim and multi-publisher safety. + +Do not delete immediately if audit/replay needs the record. Do not mark published +before the receiving system durably accepts it. If acknowledgement is lost after +acceptance, retry with stable event ID and receiver dedupe. + +Use Change Data Capture instead only when its operational/ordering/schema +contract is understood; “the database log has it” is not a consumer contract. + +## Inbox and deduplication + +Consumers can persist an inbox/deduplication record in the same transaction as +their local effect: + +```text +BEGIN + insert inbox(event_id) on conflict do nothing + if inserted: + apply local mutation + append local outbox if needed +COMMIT +ack delivery +``` + +Scope uniqueness to the consumer/handler when the same event legitimately feeds +multiple consumers. Retain dedupe records at least as long as redelivery/replay +can occur, or use aggregate revision semantics that remain durable. + +## Sagas and compensation + +Cross-system work is a saga unless one transaction truly spans it. For each step: + +| Step | Forward effect | Idempotency | Compensation | Irreversible consequence | +|---|---|---|---|---| +| Reserve credit | Provider reservation | reservation key | Release reservation | Expiry window | +| Create domain import | PostgreSQL insert | import ID | Mark abandoned/delete if safe | Audit record retained | +| Publish analytics | ClickHouse insert | event/batch key | Tombstone/rebuild projection | Eventual visibility | +| Notify customer | Email send | message key | Follow-up correction | Email cannot be unsent | + +Compensation is a business operation, not a database rollback across time. It +can fail and needs its own retry, idempotency, and operator state. Sometimes the +correct recovery is forward repair rather than compensation. ## Reconciliation -Run reconciliation continuously or on demand for stuck leases, missing queue -items, engine/database disagreement, incomplete projections, and partial sinks. -Record repairs in the audit timeline rather than mutating state invisibly. +For every cross-system edge define a query that finds disagreement: + +- domain rows with missing workflow start; +- engine runs missing public projection; +- completed runs with required sink incomplete; +- timed-out waits with no resume item; +- expired leases still marked running; +- outbox pending beyond threshold; +- external operation pending locally but committed at provider; +- artifact manifest missing blob or hash mismatch. + +A reconciler should: + +1. scan a bounded range using an indexed predicate; +2. classify expected lag versus anomaly; +3. obtain a repair lease; +4. inspect both authorities; +5. apply idempotent repair; +6. append an audit/timeline event; +7. expose counts/age and terminal manual-review cases. + +Do not mutate state invisibly. Reconciliation is part of the production path and +needs tests, rate limits, and operator controls. + +## Crash matrix + +Inject crashes: + +- after idempotency lookup before insert; +- after execution insert before enqueue; +- after engine start before external ID/projection record; +- after external effect succeeds before local acknowledgement; +- after domain commit before outbox publisher runs; +- after broker publish before outbox mark; +- after wait wins before resume enqueue; +- after resume enqueue before projection update; +- after sink write before checkpoint/manifest; +- during cancellation and compensation. + +For each, specify the retry/reconciler and prove convergence to one logical +outcome. + +## Failure signatures + +| Signature | Defect | Correction | +|---|---|---| +| Duplicate starts under concurrency | Read-before-insert idempotency | Unique constraint + transaction | +| Timeline sequence collision | Unlocked count/max | Atomic sequence allocation | +| Wait terminal but run never resumes | State update separated from enqueue | One transaction/reconciler | +| Provider charged, local state pending | Ambiguous timeout | Provider idempotency lookup | +| Domain row exists but no workflow | DB/start dual write | Outbox + dispatcher/reconciler | +| Event applied twice | No inbox/effect dedupe | Consumer-scoped unique event record | +| Compensation loops forever | No own lifecycle/budget | Durable compensation state/operator path | +| Repair changed data without audit | Reconciler treated as script | Timeline and repair identity | + +## Verification + +- Run high-concurrency identical starts and claims. +- Fault-inject every crash-matrix boundary. +- Retry ambiguous provider outcomes with fake and real sandbox APIs. +- Redeliver events before/after acknowledgment and after dedupe retention edges. +- Run outbox publishers concurrently and kill after publish. +- Race signal/timeout/cancellation for one wait. +- Corrupt projections and run the real reconciler twice. +- Assert audit timeline, metrics, and no duplicate external consequence. + +## Sources and freshness + +- PostgreSQL transaction documentation: https://www.postgresql.org/docs/current/tutorial-transactions.html + and locking clauses: https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE + (primary sources, checked 2026-07-17). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/postgres_store.ts`, + `control_plane.ts`, and `worker/loops.ts` (observed and counterexample evidence). +- Temporal Activity guidance: https://docs.temporal.io/develop/typescript/activities + (primary source; external-effect idempotency remains application-owned). diff --git a/skills/build-workflows/references/control-plane.md b/skills/build-workflows/references/control-plane.md index 4f1bcd3..c8c81f3 100644 --- a/skills/build-workflows/references/control-plane.md +++ b/skills/build-workflows/references/control-plane.md @@ -1,44 +1,201 @@ -# Control plane and reachability +# Workflow control plane and reachability -## Control-plane responsibilities +## Contents -A control plane can own definition registration, trigger normalization, -deterministic IDs, admission policies, execution records, timelines, queueing, -signals, cancellation, schedules, and operator APIs. Keep domain workflow code -separate from runtime/worker bootstrap. +- [Responsibilities](#responsibilities) +- [Definition and registration](#definition-and-registration) +- [Start and admission](#start-and-admission) +- [State and timeline](#state-and-timeline) +- [Public commands](#public-commands) +- [Reachability](#reachability) +- [Capability status](#capability-status) +- [Tests and failure signatures](#tests-and-failure-signatures) -## Reachability trace +## Responsibilities + +A custom control plane can own: + +- definition and runtime registration; +- direct, event, webhook, delayed, schedule, and workflow-invoke triggers; +- input validation and deterministic logical identity; +- idempotency and admission policy; +- durable execution projection and immutable timeline; +- ready queue, leases, retries, waits, signals, cancellation, replay; +- runtime adapter calls and reconciliation; +- operator APIs and safe product status. + +Do not make it a generic dumping ground for domain code. Business workflow +definitions remain with the owning service; the control plane provides durable +coordination primitives. + +## Definition and registration + +A definition needs: + +- unique name and owning service; +- semantic version and input/result schemas; +- runtime workflow value/type; +- supported triggers; +- idempotency and flow-control policy; +- retry/cancellation/observability policy; +- task queue/driver requirements; +- product status/cancel/event route ownership. + +At process startup compare definition, registration, and runtime catalogs. Reject +duplicate names, unsupported schema/driver versions, missing runtime values, and +workflow definitions whose required worker is unavailable when API readiness +depends on it. + +Dynamic TypeScript callbacks for keys/filters are inspectable and type-safe but +are code-version dependent. Persist the resolved key and relevant policy version +for each admitted run. Do not attempt to re-evaluate an old key under changed +code during recovery. + +## Start and admission + +One `startWorkflow` transaction should conceptually: + +1. validate workflow name and input schema; +2. compute organization/scope and resolved policy keys; +3. compute idempotency key/fingerprint and logical execution ID; +4. atomically return existing or reject conflicting key; +5. evaluate singleton/rate/throttle/debounce/batch/concurrency policy; +6. persist execution, initial timeline, and ready/delayed intent; +7. return accepted product URLs/status. + +Flow controls are different: + +| Policy | What it limits/changes | Collision behavior | +|---|---|---| +| Idempotency | Duplicate same logical request | Reuse or conflict | +| Concurrency | Active slots | Queue until slot | +| Throttle | Starts per period | Delay excess | +| Rate limit | Accepted operations per period | Reject/skip excess | +| Debounce | Burst where latest should win | Replace/postpone candidate | +| Batch | Events collected into one input | Close on size/window | +| Singleton | At most one active per key | Skip, return existing, or cancel existing | +| Priority | Ordering among runnable items | Does not bypass authorization/capacity | + +Specify scope (tenant/workflow/resource/global), time model, atomic store +primitive, and whether waiting runs count. Do not implement admission as an +unlocked count followed by insert. + +If engine start occurs after a PostgreSQL acceptance transaction, an accepted +record can exist without engine start. Queue an idempotent start command and run +a reconciler. If engine start occurs first, a started engine run can lack the +projection. Define the reverse repair. Avoid direct dual write from the HTTP +request without durable intent. + +## State and timeline + +Use explicit execution states and allowed transitions. Track cancellation request +separately from terminal cancellation. Record attempt, current step, worker/lease, +next retry, wait, result/error references, and timestamps. + +Timeline entries are append-only and ordered per execution. Include: + +- unique event/timeline ID and atomic sequence; +- kind, step/activity, attempt, actor/cause; +- workflow/runtime/build version; +- safe structured detail and artifact references; +- occurred and recorded timestamps; +- correlation/trigger/repair identity. + +Do not allocate sequence with collection length. Do not mutate history to hide a +repair; append a compensating/recovery event and update the projection. + +## Public commands + +Product services own routes and authorization. Shared handlers can translate to +the control-plane API, but must not expose arbitrary workflow names or tenant IDs. + +Typical commands: + +- start one authorized product operation; +- get status and safe result; +- list timeline/progress; +- request cancellation; +- send a validated signal/update; +- resume only where product/operator policy permits; +- list service-supported workflows for operators; +- replay/retry/repair under elevated authorization. + +Return 202 only after durable acceptance. Include stable status/cancel/events +links. Do not expose internal error payloads when a workflow's observability +policy disables them. + +## Reachability + +Trace: ```text -definition - -> registry - -> control-plane start - -> durable execution and timeline - -> queue or engine submission - -> reachable worker - -> runtime adapter - -> effect and result - -> public status/inspection path +product HTTP/CLI definition + -> auth and organization policy + -> control-plane start/signal/cancel + -> durable execution/timeline/queue intent + -> active queue/scheduler/wait/recovery loop + -> reachable runtime adapter/engine + -> compatible worker registration + -> activity/effect + -> projection/result/public stream +``` + +Exercise the active deployed path. A worker loop that returns zero, an adapter +that reports unsupported, or a route still wired to a legacy dispatcher breaks +reachability even if the new types/tests exist. + +## Capability status + +Represent runtime capability honestly: + +```ts +export const WorkflowRuntimeCapability = z.object({ + driver: z.string(), + durable: z.boolean(), + can_start: z.boolean(), + can_poll: z.boolean(), + can_signal: z.boolean(), + can_resume: z.boolean(), + registered_workflows: z.array(z.string()), + worker_reachable: z.boolean(), +}) ``` -Trace the active HTTP/CLI path. A new control plane can coexist with legacy -handlers that never call it. A worker loop named `dispatch` can still be a -`return 0` scaffold. +Do not derive `durable: true` from a driver name. Production readiness should +depend on configured driver, migrations, adapter operations, registration, and +worker/engine reachability. -## Admission policies +## Tests and failure signatures -Define atomic behavior for idempotency, concurrency limits, throttle, rate limit, -debounce, batch, singleton, and cancellation. State whether the policy applies to -logical runs, attempts, tenants, workflow types, or resources. +Test: -Do not implement admission as a read followed by an unconstrained insert. Back -it with uniqueness, locking, or an atomic engine primitive. +- duplicate definitions/registrations and unknown workflows; +- every trigger and input version; +- concurrent idempotency/admission keys; +- all flow-control policies under concurrency and time advancement; +- atomic initial state/timeline/ready intent; +- engine-start/projection failure in both directions; +- public auth/org boundaries and safe result/error policy; +- worker absent/incompatible readiness; +- legacy route detection and actual runtime effect; +- operator recovery and audit timeline. -## Timeline and projections +| Signature | Defect | Next proof | +|---|---|---| +| Definition lists workflow but start says unknown | Catalogs drift | Startup catalog equality | +| API returns 202 but queue empty | Acceptance not atomic | Transaction and restart test | +| Two singleton runs start | Check-then-insert policy | Constraint/lock/engine primitive | +| Status stays pending after completion | Projection reconciliation missing | Engine-authority reconciler | +| Driver says SQL but every method unsupported | Name mistaken for capability | Capability gate | +| New control plane has no traffic | Legacy endpoint wiring | Deployed request trace | -Append immutable, ordered events with actor, cause, timestamp, version, and -correlation. Derive mutable status projections deliberately. Sequence allocation -must remain unique under concurrency; `existing.length + 1` is not safe. +## Sources and freshness -Expose enough data for an operator to distinguish queued, leased, running, -waiting, retrying, cancelling, completed, failed, timed out, and orphaned work. +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/definition.ts`, + `control_plane.ts`, `store.ts`, `postgres_store.ts`, `endpoints.ts`, and tests + (observed source with known partial runtime reachability). +- Attachment, verified 2026-07-17: `evidence/app/new-finance/.agents/research/inngest_effect_architecture_handoff.md` + (normative comparative design, not implementation proof). +- Temporal TypeScript documentation: https://docs.temporal.io/develop/typescript + and Effect workflow source: https://github.com/Effect-TS/effect/tree/main/packages/workflow + (primary sources; adapter semantics remain engine/version-specific). diff --git a/skills/build-workflows/references/durability.md b/skills/build-workflows/references/durability.md index 8cdcc44..7e34630 100644 --- a/skills/build-workflows/references/durability.md +++ b/skills/build-workflows/references/durability.md @@ -1,47 +1,208 @@ -# Durability and engine selection +# Durability model and engine selection -## When an engine is justified +## Contents -Use a durable engine when work must survive process or host loss and needs one or -more of: durable timers, replay, signals/updates, long coordination, leases, -cross-process recovery, audited history, or operator control. +- [Durability is a set of claims](#durability-is-a-set-of-claims) +- [Authority](#authority) +- [Identity model](#identity-model) +- [Engine selection](#engine-selection) +- [Delivery and effect semantics](#delivery-and-effect-semantics) +- [Determinism and evolution](#determinism-and-evolution) +- [Completion contract](#completion-contract) +- [Deliberate exclusions](#deliberate-exclusions) +- [Verification](#verification) -A simple scheduled command, database job table, queue, or in-process pipeline may -be sufficient when its failure and recovery contract is explicit. Do not add a -workflow platform solely for abstraction aesthetics. +## Durability is a set of claims + +Classify each capability separately: + +| Claim | Required proof | +|---|---| +| Start survives API loss | Durable intent exists before acknowledgement | +| Work survives worker loss | Another worker can reclaim/replay it | +| Timer survives process loss | Deadline is persisted or engine-owned | +| External effect is safe to retry | Idempotency/deduplication at effect boundary | +| Wait survives restart | Wait identity, payload contract, and wake path persisted | +| Cancellation is durable | Request/state is persisted and observed after restart | +| Progress is inspectable | Authoritative history/timeline plus projection | +| Resume is correct | Checkpoint represents committed output and validates provenance | +| Upgrade is replay-safe | Old histories/records replay under deployment policy | +| Operators can recover | Inspect, retry, replay, cancel, repair, reconcile paths exist | + +An in-memory retry, promise chain, fiber, queue, or cron callback can be reliable +within one process and still not be durable. ## Authority -Choose the authoritative state: engine history, PostgreSQL execution record, -queue row, domain record, or an external provider. Projections and caches must be -rebuildable or reconciled from that authority. +Choose the authoritative source for each fact: + +- workflow lifecycle/history; +- domain state; +- external effect result; +- timer/wait status; +- queue readiness/lease; +- operator timeline; +- public status projection; +- output artifacts. + +One system can own several facts, but do not call two independently mutable +stores authoritative for the same fact. If Temporal history owns workflow +progress while PostgreSQL serves a product projection, define how the projection +is rebuilt and how disagreement is reconciled. If PostgreSQL owns a custom +control plane while `@effect/workflow` owns execution state, define the exact +commit gaps and recovery direction. + +An authority map should look like: + +| Fact | Authority | Projection/cache | Rebuild/reconcile | +|---|---|---|---| +| Run lifecycle | Temporal history | PostgreSQL `workflow_runs` | Scan/search attributes or event consumer | +| Import records | PostgreSQL domain tables | ClickHouse analytics | Outbox/replay by domain revision | +| Ready claim | PostgreSQL queue row | Worker memory | Lease expiry | +| Artifact bytes | Object store | Manifest row | Hash/list/reconcile | + +## Identity model + +Define stable identifiers for: + +- workflow type and semantic version; +- logical workflow ID; +- run/history ID; +- execution attempt; +- trigger/event; +- idempotency scope/key; +- activity/effect and activity attempt; +- queue item/lease; +- timer/wait/signal/update; +- schedule occurrence; +- checkpoint/stage/input partition; +- artifact and sink commit; +- replay/repair operation. + +Reuse the logical ID across retries/continuations where it represents the same +business operation. Generate distinct attempt/run IDs for observability. Store +idempotency scope and a request fingerprint so reuse of a key for different +input is a conflict rather than accidental success. + +## Engine selection + +| Option | Prefer when | You must add/prove | +|---|---|---| +| Synchronous request | Short work can complete within request and retry is client-owned | Cancellation, transaction, timeout | +| Scheduled command | Periodic stateless/reconcilable task | Overlap, missed runs, lock, audit | +| PostgreSQL job table | Moderate jobs, existing Postgres operations, simple timers | Atomic claim, leases, retry, poison, UI/API | +| Broker queue | High delivery throughput and consumer decoupling | Durable state, dedupe, visibility/lease, replay | +| Custom Postgres control plane + Effect | Product-specific admission/history with typed Effect execution | Engine persistence, atomic gaps, workers, recovery, operator surface | +| `@effect/workflow` | Effect integration is valuable and exact alpha capabilities are verified | Version pin, production adapter proof, recovery/replay tests | +| Temporal | Long-lived replay, durable timers/messages, mature worker/operator model | Temporal service/Cloud, deterministic code, deployment/version policy | +| Data orchestrator | Data assets, partitions, lineage, scheduled/backfill semantics dominate | Domain workflow/message needs may not fit | + +Selection questions: + +1. Longest runtime/wait? +2. Durable timers, signals, queries, updates, child workflows, or compensation? +3. Active steps per second and payload size? +4. Ordering/fairness/admission needs? +5. Which persistence/operations expertise already exists? +6. Can the runtime execute on the target host? +7. How are in-flight versions deployed and replayed? +8. What operator tooling is required on day one? +9. Is self-hosting a requirement or burden? +10. Can simpler queue/table semantics meet recovery needs? + +## Delivery and effect semantics + +Assume retryable work and messages are at least once unless the selected engine +and effect boundary prove stronger. “Exactly once workflow execution” does not +make an external HTTP call exactly once. + +For every external effect record: + +- stable operation key accepted by the provider or enforced locally; +- input fingerprint; +- pending intent before the call where needed; +- provider request/result identifier; +- ambiguous timeout handling; +- duplicate response handling; +- reconciliation query; +- compensation policy if reversible; +- redacted evidence for an operator. + +Do not retry a payment/email/provisioning call after an ambiguous timeout without +an idempotency or lookup contract. + +## Determinism and evolution + +Replayable orchestration cannot branch on unrecorded mutable facts. Isolate: + +- network/database/filesystem I/O; +- nondeterministic randomness/time not supplied by engine APIs; +- environment/config read after a run starts; +- unordered iteration whose order affects commands; +- runtime-specific APIs unavailable in the workflow sandbox; +- package functions with hidden nondeterminism. + +Capture policy/config values needed by a run in versioned input or query them +through recorded activities, depending on desired semantics. + +Evolution plan must cover: + +- compatible input/event decoding; +- workflow code changes for open histories; +- worker/build routing or explicit patch/version APIs; +- activity compatibility and timeout/retry changes; +- renamed workflow/activity/task queue identifiers; +- Continue-As-New or history compaction; +- schema migrations for custom control planes; +- rollback with newer persisted records. + +## Completion contract -If engine history and a database projection both appear authoritative, define -conflict resolution and recovery. “Keep them in sync” is not a contract. +Define run status independently from stage and sink status. A run is complete +only when every required outcome is committed and observable. -## Identity +```text +accepted -> queued -> leased/running -> waiting/retrying -> +completed | failed | cancelled | timed_out | terminated | requires_attention +``` -Define stable identifiers for logical workflow, run, attempt, trigger, -idempotency key, queue item, timer/wait, signal, external effect, and produced -artifact. Reuse the logical identity across retries; create a distinct attempt ID -for observability. +Do not flatten cancellation requested, cancellation observed, cleanup running, +and cancelled terminal into one boolean. Optional sink failure can yield +completed-with-warning only if the public contract names that state. -## Determinism and versioning +## Deliberate exclusions -Replayable orchestration cannot depend directly on wall clock, randomness, -unordered iteration, mutable configuration, network responses, or incompatible -code paths. Use engine-provided deterministic APIs and isolate I/O. +- Do not add a durable engine for a pure local computation. +- Do not put large payloads/artifacts in workflow history; store immutable blobs + and pass references with hashes. +- Do not call an adapter durable until the production persistence operations are + implemented and restarted. +- Do not use Redis or an in-memory queue as the sole authority merely because it + is fast. +- Do not claim exactly-once external effects without effect-boundary evidence. +- Do not build custom operator/control-plane features already required from a + selected platform without first identifying the product-specific gap. -Version workflow changes and define how existing histories continue. Test replay -of representative old histories before deployment. +## Verification -## Ecosystem maturity +- Kill API immediately after acceptance. +- Kill worker before claim, after claim, during effect, after effect, before ack, + and during checkpoint. +- Duplicate every trigger, message, queue delivery, and provider response. +- Advance timers across restart and clock skew. +- Replay representative old histories/records on new code. +- Start concurrent equivalent runs and enforce admission atomically. +- Corrupt/miss projections and execute reconciliation. +- Exercise operator inspect, cancel, retry, replay, resume, and repair. +- Validate results/artifacts, not only final status. -Temporal has a mature durable-history model with clients, workers, workflows, -activities, signals, queries/updates, timers, retries, and continue-as-new. -Inspect the exact SDK and deployment contracts. +## Sources and freshness -Effect's workflow packages may be prerelease or alpha at the installed revision. -Treat authored types and adapters separately from proven runtime, persistence, -worker reachability, and recovery. Record the version boundary rather than -generalizing current behavior. +- Temporal TypeScript documentation: https://docs.temporal.io/develop/typescript + (primary source, checked 2026-07-17). +- Effect official site and monorepo: https://effect.website/ and + https://github.com/Effect-TS/effect (primary sources; Workflows currently + advertised as alpha). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/` + and `evidence/app/better-auth/docs/workflows-mental-model.md` (observed, + normative, experimental, and counterexample evidence explicitly separated). diff --git a/skills/build-workflows/references/effect-workflow.md b/skills/build-workflows/references/effect-workflow.md new file mode 100644 index 0000000..1e0249f --- /dev/null +++ b/skills/build-workflows/references/effect-workflow.md @@ -0,0 +1,339 @@ +# Effect and `@effect/workflow` + +## Contents + +- [Status and evidence boundary](#status-and-evidence-boundary) +- [Core Effect architecture](#core-effect-architecture) +- [Workflow definition and Layer](#workflow-definition-and-layer) +- [Runtime separation](#runtime-separation) +- [Activities and external effects](#activities-and-external-effects) +- [Durable clocks, waits, and signals](#durable-clocks-waits-and-signals) +- [Typed errors and retry](#typed-errors-and-retry) +- [Scope, interruption, and workers](#scope-interruption-and-workers) +- [Configuration and observability](#configuration-and-observability) +- [Production adapter acceptance](#production-adapter-acceptance) +- [Failure signatures](#failure-signatures) +- [Verification](#verification) + +## Status and evidence boundary + +Official Effect material currently describes Workflows as alpha. The uploaded +repositories pin `effect` 3.21.x and `@effect/workflow` 0.18.x in package +manifests, while Deno import maps use broad major ranges. Pin an exact tested pair +before production use and re-check source/API semantics during upgrades. + +The uploaded code proves: + +- `Workflow.make`, workflow `toLayer`, `WorkflowEngine.layerMemory`, and a + `ManagedRuntime` can execute and poll a test workflow; +- a registration catalog exposes name, version, triggers, and runtime values; +- an adapter seam can start, poll, interrupt, and resume workflows; +- an in-memory control-plane path has focused tests. + +It does not prove durable production execution. The supplied +`effect_sql_adapter.ts` returns explicit not-implemented errors for register, +identity, start, poll, interrupt, and resume. Preserve that honest capability +status until a restart-tested adapter exists. + +## Core Effect architecture + +Use Effect services and Layers for capabilities around workflow execution: + +```text +validated config + -> database / queue / telemetry Layers + -> stores and external service clients + -> activity implementations + -> workflow engine adapter + -> workflow registration Layer + -> API client runtime and worker runtime +``` + +Keep API and worker runtimes independently constructible. They may share Layer +factories but often have different permissions, resources, concurrency, and +lifetime. A public API should not boot worker loops as an import side effect. + +Define external dependencies as `Context.Tag` services. Workflows/activities +request those services; a production Layer supplies real clients and a test +Layer supplies deterministic fakes. Use `Layer.scoped` and +`Effect.acquireRelease` for clients, leases, and subscriptions with cleanup. + +## Workflow definition and Layer + +The uploaded 0.18.x code uses this shape: + +```ts +import { Workflow, WorkflowEngine } from "@effect/workflow" +import { Effect, Layer, ManagedRuntime, Schema } from "effect" + +const ImportPayload = { + organizationId: Schema.String, + uploadId: Schema.String, +} + +const ImportResult = Schema.Struct({ + imported: Schema.Number, + rejected: Schema.Number, +}) + +export const PublishImportWorkflow = Workflow.make({ + name: "PublishImportWorkflow", + payload: ImportPayload, + success: ImportResult, + idempotencyKey: ({ organizationId, uploadId }) => + `${organizationId}:${uploadId}`, +}) + +export const PublishImportWorkflowLayer = PublishImportWorkflow.toLayer( + Effect.fn(function* (payload) { + const imports = yield* ImportService + return yield* imports.publish(payload) + }), +) + +export const WorkflowLive = Layer.mergeAll( + PublishImportWorkflowLayer, +).pipe( + Layer.provideMerge(WorkflowEngine.layerMemory), + Layer.provideMerge(ImportServiceLive), +) +``` + +The in-memory engine is appropriate for unit tests and capability spikes. It is +not a production durability proof. + +Validate payloads at both public transport and workflow runtime boundaries. The +transport schema can accept product-specific syntax; the workflow payload should +be a stable serializable object with explicit schema/version. Do not pass Hono +context, open streams, database clients, class instances, or large file contents. + +## Runtime separation + +Separate four contracts: + +| Contract | Responsibility | +|---|---| +| Definition | Name, payload/result schemas, deterministic identity, implementation Layer | +| Registration | Owning service, semantic version, trigger metadata | +| Runtime adapter | Compute ID, register, start, poll, interrupt, resume | +| Control plane | Durable admission, policy, projections, queues, timelines, public commands | + +An adapter interface prevents HTTP handlers from depending directly on an +engine. It does not create durability. For every operation state whether it is: + +- implemented by the engine; +- implemented by PostgreSQL control-plane state; +- a projection of another authority; +- unsupported and rejected explicitly. + +Do not return success from a placeholder adapter. Disable its configuration or +fail startup before accepting requests. + +Build the Layer/runtime once per API host or worker: + +```ts +const workerRuntime = ManagedRuntime.make(WorkflowWorkerLive) + +try { + await workerRuntime.runPromise(runWorker) +} finally { + await workerRuntime.dispose() +} +``` + +Do not build one runtime per queue item. + +## Activities and external effects + +Use activities or the package's current durable-step primitive for I/O and +retryable effects. Each activity contract needs: + +- stable name/version; +- serializable input/output schema; +- idempotency scope/key; +- timeout and heartbeat/lease behavior if supported; +- typed retryable and permanent errors; +- redacted observability fields; +- compensation/reconciliation path; +- bounded resources and cancellation. + +```ts +export interface SendReceiptEmailInput { + readonly organizationId: string + readonly receiptId: string + readonly operationKey: string +} + +export class EmailService extends Context.Tag("app/EmailService")< + EmailService, + { + readonly sendReceipt: ( + input: SendReceiptEmailInput, + ) => Effect.Effect<ProviderMessageId, EmailRejected | EmailUnavailable> + } +>() {} +``` + +The provider operation key must reach the provider or an atomic local +deduplication record. A workflow-level ID alone does not deduplicate a provider +call after an ambiguous timeout. + +Do not put external network calls directly in replayed orchestration. Do not +retry every tagged error. Convert permanent provider rejection and invalid input +to non-retryable workflow decisions. + +## Durable clocks, waits, and signals + +The reviewed `@effect/workflow` source/handoff describes `DurableClock` for long +sleeps and `DurableDeferred` for suspension/token completion. Confirm exact APIs +against the pinned package before implementation. + +For a durable wait, persist or engine-own: + +- wait ID and execution ID; +- wait/signal name and payload schema version; +- authorization scope; +- resume/completion token; +- timeout/deadline; +- buffered-signal rule; +- unresolved/resolved/timed-out/cancelled status; +- winning completion identity; +- timeline evidence. + +Timeout, signal, and cancellation are a race. The winning transition and resume +publication must be one transaction or repairable by a reconciler. + +The uploaded wait router performs: + +```text +mark wait timed out + -> enqueue timeout signal + -> append timeline + -> mark execution pending +``` + +These are separate calls. A crash after the first call removes the wait from due +scans but never enqueues resume work. Treat this as a concrete atomicity gap to +fix, not a pattern to copy. Prefer a store method that commits the winning wait +transition, resume queue item, timeline event, and projection together. + +## Typed errors and retry + +Separate: + +| Class | Examples | Default policy | +|---|---|---| +| Invalid/permanent | schema invalid, policy denied, unsupported version | Fail without retry | +| Domain decision | insufficient funds, approval rejected | Workflow branch/terminal state | +| Transient infrastructure | provider 503, connection reset | Bounded retry with backoff/jitter | +| Ambiguous effect | timeout after provider may have committed | Lookup/reconcile before retry | +| Defect | invariant violation, impossible state | Fail, diagnose, repair code/data | +| Cancellation/interruption | operator/user/shutdown request | Cooperative cleanup and terminal policy | + +Use Effect `Schedule` at the activity/effect boundary where retry is safe. Record +attempt, delay, and final cause. Workflow/control-plane retries and activity +retries are different budgets; do not multiply them accidentally. + +## Scope, interruption, and workers + +Supervise worker loops. `forkDaemon` without an owned runtime/supervisor can hide +loop failure. Define whether one loop failure stops the worker, restarts with a +budget, or degrades readiness. + +Worker lifecycle: + +1. resolve config and build scoped resources; +2. validate unique workflow/activity catalog; +3. register with production engine/store; +4. verify migrations and queue/task-queue reachability; +5. publish readiness; +6. start supervised claim/poll/scheduler/wait/recovery loops; +7. on shutdown stop claims, interrupt/drain work, release/expire leases, flush, + and dispose runtime. + +Use Scope/finalizers for subscriptions and leased resources, but remember a hard +kill skips finalizers. Durable lease expiry and recovery remain mandatory. + +## Configuration and observability + +Pin engine driver, schema version, queue names, worker identity, concurrency, +lease/heartbeat durations, poll intervals, retry budgets, and observability +policy in a validated config. Do not select memory versus durable driver through +an undocumented fallback. + +Correlate: + +- logical execution and attempt IDs; +- workflow/registration version; +- activity and activity attempt; +- trigger/event/idempotency IDs; +- queue item/lease owner; +- wait/signal/timer; +- trace/correlation/organization IDs; +- replay status where the engine exposes it. + +Avoid duplicate logs on replay. Use the engine/Effect observability path and an +explicit LogTape bridge if LogTape is the repository transport. Redact before +every sink. + +## Production adapter acceptance + +A production `@effect/workflow` adapter is accepted only when it proves: + +- durable engine schema and migrations; +- unique definition registration and compatibility; +- deterministic execution ID behavior; +- start acknowledgement after durable acceptance; +- execution after API/worker restart; +- poll/interrupt/resume across processes; +- durable clock/deferred behavior across restart; +- atomic or reconciled projection gaps; +- multi-worker claim safety; +- old-version replay/upgrade behavior; +- operator inspection and repair; +- resource cleanup and bounded throughput. + +Until then expose the driver as experimental/unavailable, not production. + +## Failure signatures + +| Signature | Defect | Next proof | +|---|---|---| +| `WorkflowEngine.layerMemory` in production graph | Process-local execution | Restart-test durable driver | +| Adapter methods return not implemented | Seam only | Disable config/startup or implement fully | +| Workflow starts but projection stays pending | Engine/control-plane gap | Poll/reconcile authority | +| Wait is timed out with no resume item | Non-atomic router | Transactional completion + recovery scan | +| Worker logs ready with zero active loops | Boot object mistaken for runtime | Reachable dispatch test | +| Same effect runs twice | Missing effect-boundary idempotency | Operation key and provider lookup | +| Loop fiber dies silently | Detached supervision | Worker failure/readiness policy | +| Upgrade fails old executions | No version/replay gate | Frozen history compatibility test | + +## Verification + +- Run workflow definition against memory Layer for fast logic tests. +- Run full production adapter in a real database/engine fixture. +- Kill API after durable start and worker at every activity boundary. +- Poll, interrupt, and resume from a different process. +- Race signal, timeout, cancellation, and duplicate signal. +- Restart across durable sleep/deferred. +- Inject transient, permanent, ambiguous, defect, and cancellation failures. +- Assert resource finalizers on cooperative interruption and lease recovery on + hard kill. +- Replay/version-test representative stored histories before rollout. +- Verify operator status/timeline and reconcile corrupted/missing projections. + +## Sources and freshness + +- Current-version source: pinned `@effect/workflow` 0.19.0 npm tarball and the + Effect site https://effect.website/ (checked 2026-07-17; current product + material labels Workflows alpha). +- Exact example source: `new-finance-app(1).zip:utils/workflows` and + `better-auth.zip:.agents/research/inngest_effect_architecture_handoff.md`, + verified 2026-07-17 as observed tests plus research/design evidence. +- Exact exported values and options including `Workflow.make`, `toLayer`, + `WorkflowEngine.layerMemory`, `Activity`, `DurableClock`, and `DurableDeferred` + are version-sensitive. Uploaded manifests pin `@effect/workflow` `^0.18.2` and + `effect` `^3.21.3`; the Deno import map's broad `@effect/workflow@0` range must + not be treated as a reproducible production pin. +- The uploaded `runtime/effect_sql_adapter.ts` is explicit counterevidence: its + production operations are not implemented. diff --git a/skills/build-workflows/references/failures.md b/skills/build-workflows/references/failures.md index edf16d4..cc370eb 100644 --- a/skills/build-workflows/references/failures.md +++ b/skills/build-workflows/references/failures.md @@ -1,21 +1,184 @@ -# Workflow failure signatures +# Workflow and pipeline failure diagnosis -| Signature | Why it is incomplete | Next proof | +Use this reference when durable work is stuck, duplicated, lost, falsely successful, unrecoverable, or only scaffolded. Prove reachability, atomicity, restart, reconciliation, and operator control. More workflow terminology cannot repair a missing commit protocol. + +## Contents + +- Capability evidence ladder +- Acceptance and start failures +- Queue, lease, and worker failures +- Wait, signal, schedule, and cancellation failures +- Cross-system and pipeline failures +- Runtime adapter failures +- Recovery protocol +- Invariant and failure table +- Failure-injection matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness + +## Capability evidence ladder + +```text +types/definition + -> registry + -> active control-plane path + -> durable state transition + -> deployed worker/runtime dispatch + -> restart/reconciliation + -> operator repair +``` + +Classify each feature independently: starts, event fan-out, queue claims, retries, schedules, waits, signals, cancellation, replay, runtime persistence, pipelines, and observability. A rich type model does not prove any worker calls it. + +The retained finance workflow package demonstrates this distinction. It has detailed definitions, schemas, store interfaces, PostgreSQL tables, control plane, queue workers, waits, schedules, and a memory Effect adapter. Yet `tickEventDispatcher()` returns `0`, the SQL-backed Effect runtime adapter returns `workflow_runtime_not_implemented` for every operation, cron expansion is incomplete, and several multi-write transitions are not transactional. Report that honestly. + +## Acceptance and start failures + +| Signature | Defect | Recovery/proof | +|---|---|---| +| API returns 202 but execution missing | response precedes durable acceptance | idempotent request retry; status immediately readable after accept | +| Execution exists but no queue row | execution/timeline/enqueue split across writes | orphan reconciler or transactional create; crash test | +| Duplicate starts share business effect | idempotency check read-then-insert | unique constraint/atomic insert plus concurrent test | +| Deterministic runtime ID differs from store ID | two identity owners | one explicit mapping and restart/poll test | +| Start endpoint uses legacy control plane | new system unreachable | route-to-control-plane trace | +| Definition exists but runtime registration missing | authored, not deployable | boot-time registry conformance | +| Start fails after external effect | effect occurs before durable intent/idempotency | transactional intent/outbox and effect idempotency | + +For ambiguous client timeouts, query by idempotency/execution ID before retrying. + +## Queue, lease, and worker failures + +| Signature | Defect | Correction evidence | +|---|---|---| +| Two workers execute same row | selection and claim not atomic or lease ignored | concurrent claim oracle; atomic conditional claim/lock | +| Lease expires during healthy long task | no heartbeat/lease extension or duration bound | renewal and ownership-fenced completion | +| Old worker acknowledges after lease stolen | no fencing token/owner condition | ack/update requires current lease owner/version | +| Worker crashes after effect before ack | at-least-once redelivery | effect idempotency and replay test | +| Queue item dead but execution running forever | state/dead-letter update split | reconciler and terminal/operator state | +| Retry count grows without bound | attempt/backoff/dead policy absent | bounded retry and dead-letter action | +| Worker process starts but no loops run | boot/supervision missing | deployed process and tick/heartbeat evidence | +| Shutdown drops leased work | abrupt exit/no drain | stop claims, cancel/drain, release/expire leases | + +The retained PostgreSQL queue first selects eligible rows then conditionally updates each ready row. Its comment acknowledges `FOR UPDATE SKIP LOCKED` as a stronger high-concurrency approach. Conditional update can prevent both callers from returning the same row, but batch fairness, ordering, owner-fenced ack, and transaction behavior still need concurrent executable tests. + +## Wait, signal, schedule, and cancellation failures + +| Signature | Defect | Required repair | +|---|---|---| +| Wait marked timed out but no resume row | status update and enqueue split | transaction/outbox or reconciler for terminal wait without resume | +| Signal stored but execution never resumes | signal, wait update, enqueue, processed marker split | durable transition and repair scan | +| Late signal resumes timed-out/cancelled wait | status change not conditional | compare-and-set active wait and conflict response | +| Same signal wakes multiple waits unexpectedly | matcher/identity ambiguity | explicit target/matcher and uniqueness policy | +| Interval schedule publishes twice | schedule claim/update not atomic | occurrence identity and idempotent start | +| Cron row exists but never fires | calendar/timezone expansion unimplemented | explicit unsupported status or implemented parser tests | +| Cancel says applied while runtime ignores it | request state confused with terminal cancellation | requested/cancelling/applied/terminal distinction | +| Cancel races success | terminal-state policy absent | compare-and-set and allowed-outcome tests | + +In retained code, wait timeout state is updated before resume enqueue; signal flow records signal, updates wait, enqueues, marks processed, appends timeline, and updates execution separately. A crash can expose partial states. Make them repairable/transactional before durability claims. + +## Cross-system and pipeline failures + +| Signature | Defect | Proof | +|---|---|---| +| Runtime starts but DB still pending | cross-system start/status gap | idempotent start plus poll/reconciler | +| DB says running but runtime never received start | status updated before external runtime call | recoverable intent/queue and start retry | +| Workflow succeeds while required sink failed | stage/sink state collapsed | per-sink manifest and completion predicate | +| Checkpoint advances before artifact/sink commit | attempted input treated as progress | committed receipt checkpoint | +| Duplicate retry corrupts sink | activity not idempotent/version-aware | fail-after-effect replay oracle | +| All records kept for profile | nominal stream is unbounded | memory-bound large input test | +| Backfill overwrites newer live change | no authority version compare | snapshot boundary/catch-up and version-aware projector | +| Compensation fails silently | compensation treated as magic rollback | compensation state, retry/operator escalation | + +Workflow runtimes do not make external effects atomic. Each activity needs an effect identity, timeout/cancellation, retry safety, and compensation/repair where appropriate. + +## Runtime adapter failures + +| Signature | Meaning | Action | |---|---|---| -| Definition and types exist | Authored only | Registry and active start path | -| Runtime adapter exists | May be unreachable | Worker boot and dispatch | -| Database tables exist | Persistence schema only | Transactional state transition | -| Dispatcher returns zero | Scaffold | Work claim and execution test | -| HTTP path uses legacy control plane | New system not reachable | Route-to-control-plane trace | -| Sequence is `existing.length + 1` | Concurrent collision | Atomic sequence allocation test | -| Start does read then insert | Idempotency race | Unique constraint and concurrent test | -| Claim reads then updates | Lease race | Atomic claim primitive | -| Wait marked resolved before enqueue | Crash can orphan resume | Transaction or reconciler | -| Engine start and DB update are separate | Cross-system gap | Idempotent start and repair | -| Exceptions are caught and passed | False success and lost evidence | Explicit stage error accounting | -| All records retained for profiling | Unbounded memory | Batch-bounded memory oracle | -| Multiple sinks have no manifest | Partial success is invisible | Per-sink state and resume | -| Retry exists | Process loss still loses state | Restart against durable store | - -The correction is not more workflow terminology. Prove reachability, atomicity, -restart, reconciliation, and operator control with executable failure injection. +| Adapter compiles but every method returns unsupported | seam only, no runtime capability | keep routes/readiness disabled or explicit unavailable | +| Memory adapter passes tests | process-local behavior proven only | run durable backend restart tests | +| Poll returns undefined indefinitely | running/suspended/missing ambiguous | runtime state/heartbeat/deadline classification | +| Registration names differ across deployments | queued state cannot resolve workflow | deployment compatibility/version gate | +| Workflow code changes with in-flight executions | replay/version compatibility unknown | pin version or migration strategy | +| Effect/Temporal API assumed from guide | alpha/version drift | inspect installed exports and executable fixture | + +Do not label an adapter durable because its interface uses `Effect` or its name contains `sql`. The retained `createEffectSqlWorkflowRuntimeAdapter()` is explicitly a placeholder. + +## Recovery protocol + +1. Stop unsafe retry/dispatch if it can duplicate irreversible effects. +2. Capture execution, timeline, queue, lease, wait, signal, schedule, cancellation, runtime, stage/sink, checkpoint, and deployed-version state. +3. Identify the last authoritative transition and any external effect identity. +4. Classify state as safe-to-retry, already-applied, partially-applied, terminal, or unresolved. +5. Apply a named repair: enqueue orphan, release expired lease, reconcile runtime, restore missing projection, resume wait, mark dead, replay from checkpoint, or operator compensation. +6. Make repair idempotent and audit it. +7. Restart workers/runtime and observe through real status/control endpoints. +8. Reconcile final domain and side-effect state, not only workflow status. + +Never delete the timeline/queue/error evidence before repair verification. + +## Invariant and failure table + +| Invariant | Failure oracle | +|---|---| +| Accepted run is durably discoverable | kill after response; status exists after restart | +| One idempotency key has one intended execution/effect | concurrent starts and ambiguous response retry | +| Queue completion is fenced to lease owner/version | lease expiry plus old worker ack | +| Checkpoint never exceeds committed output | kill before sink receipt | +| Required sinks gate completion | fail each sink separately | +| Wait transition produces at most one resume | signal/timeout race | +| Schedule occurrence starts at most intended run | two schedulers claim same due row | +| Cancellation state reflects reality | cancel/success/failure race | +| Runtime/store converge | fail on both sides of start/poll update | +| Operator can repair orphaned partial states | seeded orphan fixtures and repair commands | + +## Failure-injection matrix + +Kill or fault: + +- after execution insert, each timeline append, and queue insert; +- after lease claim, status-running update, external start, ack, and poll persist; +- after signal record, wait update, resume enqueue, processed marker, and timeline; +- after timeout wait update and before enqueue; +- after schedule occurrence start and before next-run update; +- before/after cancellation request, runtime interrupt, and terminal update; +- after every external activity effect and before activity acknowledgement; +- before/after artifact/sink receipt and checkpoint; +- during worker heartbeat/lease renewal; +- during deployment with old queued workflow versions; +- during shutdown and dependency restart. + +Run with two or more workers/schedulers, duplicate messages, out-of-order signals, slow activities, clock skew, and a real durable store/runtime where claims require durability. + +## Executable verification + +Create deterministic failpoints and restart the process between them. Query durable tables/runtime and use public status/timeline/control APIs. Assert: + +- no accepted run vanishes; +- duplicate delivery does not duplicate semantic effects; +- partial states are visible and repaired; +- leases are fenced and recover; +- wait/signal/schedule races produce declared outcomes; +- cancellation reaches a real terminal outcome; +- required sinks gate completion; +- in-flight version/deployment mismatch fails safely; +- shutdown leaves recoverable work; +- operator repair is idempotent and audited. + +Unit tests against a memory store/adapter are useful for control logic but do not establish process-loss durability. + +## Deliberate exclusions + +- Do not force Effect, `@effect/workflow`, Temporal, or the retained finance workflow package. +- Do not call types, tables, adapters, or definitions implemented behavior. +- Do not call a memory adapter durable. +- Do not claim external side effects are atomic because workflow state is durable. +- Do not use retries without idempotency, bounded attempts, and operator visibility. +- Do not collapse requested cancellation into completed cancellation. +- Do not advance pipeline progress from attempted work. +- Do not claim recovery without process restart and real dependency tests. + +## Sources and freshness + +Grounded in the retained finance workflow README, definitions, control plane, store interfaces, PostgreSQL store, schema, workers, memory and placeholder SQL Effect adapters, endpoint helpers, and PopModern multi-stage/multi-sink ETL counterexamples, reviewed 2026-07-17. The retained workflow documentation describes intended behavior more strongly than several code paths currently guarantee; source inspection and failpoint tests take precedence. Runtime and driver behavior is version-sensitive. diff --git a/skills/build-workflows/references/pipelines.md b/skills/build-workflows/references/pipelines.md index c2cbd27..218d1e3 100644 --- a/skills/build-workflows/references/pipelines.md +++ b/skills/build-workflows/references/pipelines.md @@ -1,41 +1,344 @@ -# Resumable data pipelines +# Resumable, bounded, multi-sink data pipelines -## Stage model +Use this reference when work discovers, fetches, parses, normalizes, enriches, loads, aggregates, indexes, or projects data over time. A durable pipeline is not a `for` loop with retries. It has stage contracts, stable identities, committed checkpoints, bounded resources, explicit required/optional sinks, and recovery from every crash window. -For ingestion, separate discovery, fetch, decode, observe, detect, derive, load, -aggregate, index, and project as applicable. Each stage needs input/output schema, -identity, provenance, error policy, checkpoint, and required/optional status. +## Contents -Retain raw evidence when replay, audit, or improved parsing matters. Derive -normalized records and projections reproducibly. +- Pipeline authority and execution owner +- Stage contract +- Identity, provenance, and versions +- Bounded processing and backpressure +- Checkpoints and resume +- Multi-sink commit model +- Error and quarantine policy +- Configuration and capability model +- Worker/runtime integration +- Operational state and controls +- Integration example +- Failure and recovery +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Bounded execution +## Pipeline authority and execution owner -Stream or batch with explicit limits. Do not retain every record for profiling -inside a nominally streaming job. Bound memory, open files, concurrent requests, -queue depth, and sink batches. +Name: -## Checkpoints +- trigger owner: endpoint, event, webhook, schedule, CLI, or operator; +- execution authority: workflow store, job database, artifact manifest, or another durable owner; +- input authority and snapshot/change boundary; +- stage output authority versus rebuildable intermediates; +- worker/lease owner and deployment; +- sink authority and required/optional status; +- operator controls for pause, cancel, retry, replay, repair, and backfill. -A checkpoint represents committed output, not attempted input. Include run ID, -source identity/version, stage version, offset/key, output manifest, and schema -version. Reject incompatible resumes or run an explicit migration. +Do not add Effect, Temporal, or another workflow engine automatically. A short bounded import can use an ordinary job runner with a durable manifest. A long-running pipeline with waits, schedules, external effects, or operator control may benefit from a durable workflow runtime. Preserve the consumer's selected owner and prove its adapter/deployment. -## Multiple sinks +## Stage contract -PostgreSQL, ClickHouse, Typesense, graph stores, JSONL, and Parquet writes are -separate commits unless proven otherwise. Record per-sink state in a completion -manifest. A run succeeds only when every required stage and projection is -complete. Optional failures remain visible. +Each stage needs a contract: -## Error policy +```ts +interface StageContract<Input, Output> { + name: string + version: string + required: boolean + inputSchema: string + outputSchema: string + identityOf: (input: Input) => string + run: (batch: readonly Input[], context: StageContext) => Promise<StageBatch<Output>> + retry: RetryPolicy + resourceLimits: { batchSize: number; concurrency: number; timeoutMs: number } +} -Do not catch broad exceptions and continue without accounting. Classify skip, -retry, quarantine, fail-run, and best-effort outcomes. Persist rejected items and -causes safely enough for inspection and replay. +interface StageBatch<T> { + accepted: Array<{ identity: string; value: T }> + rejected: Array<{ identity: string; code: string; retryable: boolean }> + receipt: { batchId: string; checksum: string } +} +``` -## Verification +Common stages are hypotheses, not mandatory folders: -Interrupt between stages and sink commits, resume from checkpoints, inject -duplicate and malformed inputs, fail one sink, reorder/late-deliver records, and -measure memory against a large synthetic source. +```text +discover -> fetch -> capture raw -> decode -> observe/profile -> normalize + -> detect/change -> derive/enrich -> validate -> load authority + -> aggregate -> search/graph/analytics projections -> reconcile -> publish +``` + +For every transition define input/output schema, stable identity, provenance, version compatibility, error policy, resource bounds, and checkpoint boundary. A mapper mutating an in-memory graph may be useful implementation detail but is not itself a stage commit. + +## Identity, provenance, and versions + +Use identities that survive retries: + +- run ID: one invocation/backfill/retry lineage; +- source identity: provider/system plus object/page/file/event ID; +- source version: revision, ETag, sequence, snapshot, or content digest; +- record identity: deterministic domain/source mapping; +- batch/segment ID; +- stage name/version/config digest; +- artifact/sink target version; +- durable change/checkpoint identity. + +Keep event time, source retrieval time, and processing time distinct. Store the original raw identity on normalized/derived records. A downstream failure should be traceable back to raw evidence without searching log text. + +Schema/code/config changes can make a checkpoint incompatible. Reject resume or run a named migration/replay. Never silently continue a new mapper against old partially normalized output. + +## Bounded processing and backpressure + +Bound every resource: + +- source page/batch size; +- decoded records held at once; +- network concurrency and per-host rate; +- open files/sockets; +- mapper/graph size; +- sink batch size; +- queue depth; +- rejected-record buffer; +- profiling cardinality/sample/sketch memory; +- output line/row size; +- total runtime and per-item timeout. + +Use a bounded producer/consumer pipeline: + +```text +source iterator + -> bounded channel(batch capacity N) + -> worker pool(concurrency C) + -> ordered/unordered result policy + -> sink batcher(size B, timeout T) + -> committed receipt + -> source checkpoint +``` + +Backpressure means the source pauses when downstream capacity is full. “Streaming” code that appends every record to `records_for_profiling` is still unbounded. The retained PopModern MediaWiki ETL does this before profiling and Parquet writing; its Parquet helper also materializes the iterable. Replace global materialization with bounded sketches/samples and chunked writer batches. + +For RDF, the retained importer flushes an RDFLib graph every configured batch, which bounds that graph. However, it writes each sink serially and has no per-record manifest/checkpoint. Keep the bounded idea, add durable receipts and recovery. + +## Checkpoints and resume + +A checkpoint means outputs through a boundary are committed: + +```ts +interface PipelineCheckpoint { + runId: string + stage: string + stageVersion: string + configDigest: string + input: { sourceId: string; sourceVersion: string; position: string } + outputReceipts: Array<{ sink: string; target: string; receipt: string; checksum?: string }> + committedAt: string +} +``` + +Commit order: + +```text +read batch + -> transform + -> write required output(s) + -> inspect receipts and reconcile batch identities + -> atomically commit checkpoint + -> release source/batch +``` + +If the process dies after sink write but before checkpoint, replay occurs. Sinks need deterministic/version-aware idempotency. If checkpoint commits first, output can be permanently skipped; that ordering is invalid unless a stronger transaction spans both. + +Resume preflight: + +1. Load checkpoint and referenced receipts/artifacts. +2. Verify source identity/version and target still exist. +3. Verify stage/schema/config compatibility. +4. Validate last artifact/receipt/checksum. +5. Determine whether replay of boundary is safe. +6. Continue from the last committed position. +7. Reconcile before final completion. + +## Multi-sink commit model + +PostgreSQL, ClickHouse, Typesense, RDF/N-Triples/QLever, JSONL, and Parquet are separate commits unless proven otherwise. Track each sink: + +```ts +type SinkState = { + name: string + required: boolean + state: 'pending' | 'writing' | 'complete' | 'failed' + targetVersion: string + receipt?: string + accepted: number + rejected: number + retryable?: boolean +} +``` + +Choose a policy: + +- required all: run is incomplete until every required sink reconciles; +- optional degraded: run can complete but capability/failed sink stays visible with retry/operator action; +- staged authority then projections: authoritative load commits, run enters `projecting` until required projections complete; +- immutable build/cutover: build all versioned outputs, validate, then publish a final manifest/pointer. + +Never print “complete” after a required sink error. Never use one global error count instead of per-stage/per-sink identity. + +## Error and quarantine policy + +Classify errors: + +| Disposition | Use when | Durable evidence | +|---|---|---| +| retry item/batch | transient and replay-safe | identity, attempt, next time, safe cause code | +| quarantine | record invalid/unsupported but rest may proceed | raw identity, stage/version, safe issues, replay status | +| skip by policy | explicitly irrelevant | policy code and count | +| fail stage/run | contract/infrastructure/required sink compromised | last checkpoint, affected range, operator action | +| best-effort optional | non-required output | visible failed sink, retry/waive decision | + +Broad `except Exception: pass` destroys the distinction. Cleaning failure followed by mapping can produce malformed downstream state; raw capture failure removes replay evidence; profile/Parquet/PostgreSQL failure can make the run incomplete. Decide each independently. + +Do not include secrets or entire sensitive records in quarantine reasons. Store a safe raw-artifact reference and access-control classification. + +## Configuration and capability model + +Validate resolved pipeline configuration: + +```yaml +pipeline: + name: mediawiki-toys + version: 3 + source: + base_url: https://example.invalid/api.php + page_size: 100 + rate_per_second: 1 + snapshot: revision-boundary + execution: + batch_size: 500 + concurrency: 4 + queue_capacity: 8 + item_timeout_ms: 30000 + artifacts: + raw_required: true + normalized_schema: toy.v3 + sinks: + postgres: { required: true } + typesense: { required: true, target: toys_v3 } + parquet: { required: false } +``` + +Keep secret references out of manifests. Record a redacted resolved config digest. Distinguish unsupported capabilities from disabled ones and unavailable dependencies. + +## Worker/runtime integration + +Workers need: + +- atomic lease/claim with owner and expiry; +- heartbeat for long batches; +- bounded attempt and backoff policy; +- idempotent stage/batch execution; +- acknowledgement only after commit receipt; +- dead-letter/quarantine and operator redrive; +- graceful shutdown: stop claiming, cancel/drain bounded work, release/expire leases; +- version compatibility between queued work and deployed worker; +- readiness that checks required store/schema/runtime. + +If integrated with a workflow runtime, store stage/checkpoint facts in the chosen durable owner and use activities for side effects. Do not assume the runtime automatically makes an external bulk API, file append, or database write idempotent. + +The retained finance workflow platform has a useful queue/lease/control-plane model but contains incomplete atomicity and a placeholder SQL-backed Effect adapter. Do not present it as a ready production pipeline runtime without repairing and proving those paths. + +## Operational state and controls + +Expose: + +- run and stage state; +- last committed source/checkpoint; +- required/optional sink state; +- throughput, lag, retry/quarantine counts; +- current worker/lease and heartbeat; +- pause/cancel/retry/replay/repair actions; +- input/schema/config/runtime versions; +- links to safe artifacts and error summaries. + +Cancellation is cooperative. Define whether current batch commits, rolls back, or is abandoned and replayed. Pause should stop new claims without losing committed progress. Replay should create a new run lineage or clearly record the relationship. + +## Integration example + +```text +schedule/operator/API trigger + -> create durable run and freeze source boundary + -> worker leases discover stage + -> fetch page and commit raw artifact + -> decode/normalize bounded batch + -> write authority transaction plus change identities + -> write versioned projection batches + -> inspect per-item receipts + -> commit checkpoint + -> continue until source boundary exhausted + -> reconcile all required sinks + -> publish complete manifest and run status +``` + +## Failure and recovery + +| Crash/failure window | Expected recovery | +|---|---| +| before run record | request may retry with idempotency key; no accepted claim | +| after run record before queue | reconciler enqueues visible orphan or marks failed | +| after raw capture before normalized output | replay parser from raw artifact | +| after sink write before checkpoint | replay same batch; sink returns already-applied/version wins | +| after checkpoint before acknowledgement | redelivery reads committed checkpoint and no-ops/continues | +| after one of several sinks | resume only incomplete sink from batch manifest | +| wait/cancel during batch | apply declared batch commit/abort policy and persist terminal state | +| new code sees old checkpoint | reject or run explicit migration/replay | +| required sink unavailable | remain incomplete with retry/backoff/operator visibility | +| optional sink waived | record operator/policy waiver; do not erase failure | + +## Test matrix + +Test: + +- empty, one-item, batch-boundary, and input much larger than RAM; +- duplicate, out-of-order, late, malformed, and oversized source records; +- source pagination/revision changes and rate limiting; +- each stage schema/version/config mismatch; +- process loss before/after every receipt/checkpoint boundary; +- required and optional sink partial/batch failures; +- per-item bulk rejection; +- duplicate replay and old-version delivery; +- disk full, network timeout/reset, pool exhaustion, dependency restart; +- lease expiry, two workers, heartbeat loss, and redrive; +- cancellation/pause during fetch, transform, and sink write; +- graceful worker shutdown; +- quarantine access/redaction/replay; +- backfill racing live changes; +- final identity/count/hash/domain reconciliation; +- memory, file-descriptor, queue-depth, and latency limits. + +## Executable verification + +Use a disposable source fixture and real sink containers/services where feasible. Record a baseline run, kill the process at deterministic failpoints, restart, and compare authoritative/projection identities with the baseline. Require: + +- no skipped committed input; +- no semantically duplicated effects; +- no final manifest before required sinks complete; +- bounded memory/concurrency; +- visible quarantines and failed optional sinks; +- safe cancellation/shutdown; +- successful targeted repair and full rebuild. + +Run native queries against PostgreSQL/ClickHouse/Typesense/QLever and validate JSONL/Parquet manifests. Unit tests of a stage function are necessary but not sufficient. + +## Deliberate exclusions + +- Do not force Effect, Temporal, a queue, or a workflow engine for every pipeline. +- Do not call iterative code bounded if profiling/artifact writers retain all records. +- Do not advance checkpoints before required output receipts. +- Do not swallow required failures or collapse all sinks into one success bit. +- Do not promise exactly-once; prove idempotent/version-aware replay and reconciliation. +- Do not force Zod, LogTape, Drizzle, or particular sinks when the consumer selected alternatives. +- Do not treat a placeholder adapter/sink as implemented. +- Do not resume across incompatible code/schema/config silently. + +## Sources and freshness + +Grounded in the retained PopModern `DATA_PIPELINE.md`, recipe-driven importer, MediaWiki ETL, raw/JSONL/Parquet/profile/PostgreSQL utilities, RDF/Typesense/QLever sinks, and the retained finance workflow definitions, PostgreSQL store, queue workers, waits/signals, and runtime adapters, reviewed 2026-07-17. PopModern and finance include both useful patterns and explicit prototypes/incomplete durability paths; examples are classified accordingly. Verify selected engine/runtime capabilities and deployed versions before implementation claims. diff --git a/skills/build-workflows/references/recovery.md b/skills/build-workflows/references/recovery.md new file mode 100644 index 0000000..889baee --- /dev/null +++ b/skills/build-workflows/references/recovery.md @@ -0,0 +1,184 @@ +# Recovery, checkpoints, leases, replay, and repair + +## Contents + +- [Recovery model](#recovery-model) +- [Leases and heartbeats](#leases-and-heartbeats) +- [Checkpoints and resume](#checkpoints-and-resume) +- [Retry, replay, resume, and repair](#retry-replay-resume-and-repair) +- [Recovery controller](#recovery-controller) +- [Operator control plane](#operator-control-plane) +- [Recovery drills](#recovery-drills) +- [Failure signatures](#failure-signatures) + +## Recovery model + +For each state answer: + +- What failure can leave work here? +- Is forward progress automatic, timed, or operator-triggered? +- Which durable fact authorizes recovery? +- How is concurrent recovery prevented? +- Can the previous attempt still finish late? +- Which effects are safe to repeat? +- What evidence proves successful recovery? +- When does the run become `requires_attention`? + +Recovery is not “retry on exception.” It includes abandoned leases, process +loss, delayed/missing messages, ambiguous external effects, incompatible +checkpoints, stale projections, partial sinks, poisoned input, and broken +deployments. + +## Leases and heartbeats + +A lease record should contain: + +- item/execution identity; +- owner and worker build/version; +- acquired and expiry timestamps; +- attempt and fencing token/generation; +- heartbeat/progress timestamp; +- last safe error/phase; +- cancellation state. + +Claim atomically. Use a fencing token when a late old owner could still write +after expiry: downstream commits reject stale generations. + +Choose lease duration from maximum heartbeat interval plus operational jitter, +not average runtime. Heartbeat frequently enough to recover promptly but not so +frequently that the store becomes the bottleneck. + +On shutdown stop claims, request cancellation, heartbeat/drain within a deadline, +and release leases only when the activity is definitely stopped. Releasing while +the old work continues creates concurrent effects. + +## Checkpoints and resume + +A checkpoint describes committed progress: + +```ts +export const Checkpoint = z.object({ + run_id: z.string(), + stage: z.string(), + stage_version: z.string(), + input_identity: z.string(), + input_hash: z.string(), + partition: z.string().nullable(), + committed_cursor: z.string(), + output_manifest_id: z.string(), + output_hash: z.string(), + schema_version: z.string(), + committed_at: z.iso.datetime(), +}) +``` + +Resume procedure: + +1. load the latest committed checkpoint for the logical stage/partition; +2. validate input identity/hash and schema/stage compatibility; +3. verify referenced output manifest/artifacts exist and hashes match; +4. reconstruct state from committed output, never process memory; +5. resume strictly after the committed cursor; +6. preserve deduplication for a partially repeated batch; +7. create a new attempt with provenance pointing to the checkpoint; +8. write the next checkpoint only after output commit. + +Never advance a checkpoint before sink commit. Never resume by line count alone +when input can change. For multiple sinks, checkpoint each sink or advance the +stage only after every required sink is committed. + +## Retry, replay, resume, and repair + +| Operation | Meaning | Identity behavior | +|---|---|---| +| Retry | Reattempt a failed effect/stage under same logical operation | New attempt, same idempotency key | +| Resume | Continue after committed checkpoint/wait | Same logical run, new attempt/lease | +| Replay | Reprocess retained input/history under declared code/policy | New replay ID; preserve source run | +| Redrive | Move dead-letter/terminal work back to ready after decision | New attempt, audited operator action | +| Repair | Reconcile authoritative systems and apply targeted correction | Repair ID and before/after evidence | +| Backfill | Process historical range not previously required | New bounded campaign identity | + +Do not use these words interchangeably in APIs or dashboards. A replay may +intentionally use new code; a workflow-history replay often must reproduce old +commands. Name which one. + +## Recovery controller + +Automated recovery scans should be bounded, indexed, leased, observable, and +idempotent. Typical detectors: + +- ready item unclaimed beyond SLA; +- expired lease; +- running execution without live engine/lease evidence; +- cancellation requested beyond grace period; +- due timer/wait without resume item; +- retry scheduled but no ready item; +- completed engine run with nonterminal projection; +- required sink/checkpoint missing; +- outbox unpublished beyond threshold; +- orphaned artifact/temp file. + +Use compare-and-set/state preconditions so a recovery action does not overwrite a +late legitimate transition. Append recovery events to the timeline. + +## Operator control plane + +Provide authorized operations with dry-run/preview where risk warrants: + +- inspect input, history/timeline, attempts, waits, leases, outputs, and errors; +- request cancel and observe completion; +- pause/resume schedule or run; +- retry eligible failed stage/activity; +- redrive poison work after correction; +- replay a bounded set with rate/concurrency controls; +- reconcile one execution or indexed cohort; +- override/skip/compensate with reason and approval; +- quarantine input/artifact; +- terminate only as last resort. + +Every mutation records actor, reason, request/correlation ID, before/after state, +affected version, and resulting operation ID. Scope authorization by tenant and +operator role. Do not expose raw secret-bearing inputs/errors. + +## Recovery drills + +- SIGKILL worker immediately before and after heartbeat. +- Pause network between worker and store/engine. +- Let a lease expire while the old worker later returns. +- Corrupt/delete a checkpoint artifact. +- Change input under the same filename/path. +- Kill after sink commit before checkpoint. +- Make one of several sinks unavailable. +- Deliver a signal at the same time as timeout and cancellation. +- Deploy incompatible worker then rollback. +- Exhaust retry budget and redrive after repair. +- Run reconciler concurrently twice. +- Restore from backup and reconcile platform/domain state. + +Assert one logical outcome, bounded duplicate work, no duplicate external +consequence, visible audit evidence, and a finite terminal/manual state. + +## Failure signatures + +| Signature | Defect | Correction | +|---|---|---| +| Lease expired but two workers commit | No fencing/idempotency | Generation check and effect key | +| Resume duplicates prior output | Checkpoint marks attempted input | Commit output before checkpoint | +| Resume accepts changed input | No input identity/hash | Compatibility preflight | +| Reconciler oscillates state | No authority/CAS | Directional authority and preconditions | +| Replay floods provider | No campaign admission limit | Bounded replay planner | +| Operator “retry” creates new business operation | Logical identity lost | Explicit retry semantics | +| Terminal run has no diagnosis | Poison state stores only string | Safe structured cause and artifacts | +| Cleanup assumes finalizer ran | Hard-kill path ignored | Lease expiry/orphan scan | + +## Sources and freshness + +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/`, + especially store, worker loops, timelines, and tests (observed and + counterexample evidence). +- Temporal TypeScript testing and replay guidance: + https://docs.temporal.io/develop/typescript/testing-suite and + https://docs.temporal.io/develop/typescript/debugging (primary sources; + verify current SDK tooling). +- PostgreSQL locking documentation: https://www.postgresql.org/docs/current/explicit-locking.html + (primary source; fencing and checkpoint schemas are application design). diff --git a/skills/build-workflows/references/streams.md b/skills/build-workflows/references/streams.md new file mode 100644 index 0000000..130fd5c --- /dev/null +++ b/skills/build-workflows/references/streams.md @@ -0,0 +1,189 @@ +# Workflow event streams, cursors, and SSE projections + +## Contents + +- [Separate workflow history from public events](#separate-workflow-history-from-public-events) +- [Event and cursor contracts](#event-and-cursor-contracts) +- [Atomic publication](#atomic-publication) +- [Replay-to-live handoff](#replay-to-live-handoff) +- [Backpressure](#backpressure) +- [Cancellation and authority](#cancellation-and-authority) +- [Retention and compaction](#retention-and-compaction) +- [Tests and failure signatures](#tests-and-failure-signatures) + +## Separate workflow history from public events + +Engine history, operator timeline, domain events, and customer progress streams +serve different consumers: + +| Stream | Authority/purpose | Audience | +|---|---|---| +| Engine history | Deterministic replay and platform operations | Engine/operators | +| Workflow timeline | Stable run inspection and repair | Product operators/support | +| Domain events | Business facts/integration | Services and external consumers | +| Progress stream | Safe product projection | Authorized user/browser | + +Do not expose raw Temporal/Effect/custom-engine history through SSE. It can contain +internal names, payloads, errors, and implementation details and is not a stable +public contract. Project versioned safe events. + +## Event and cursor contracts + +```ts +export const RunEvent = z.object({ + event_id: z.string(), + run_id: z.string(), + organization_id: z.string(), + sequence: z.int().nonnegative(), + type: z.enum([ + "run.accepted.v1", + "run.stage.v1", + "run.warning.v1", + "run.completed.v1", + "run.failed.v1", + "run.cancelled.v1", + ]), + occurred_at: z.iso.datetime(), + data: z.record(z.string(), z.unknown()), +}) +``` + +In a real implementation use a discriminated union so each event type owns a +precise payload. Public payloads exclude raw exceptions, SQL/provider bodies, +secrets, and unauthorized artifacts. + +A cursor identifies a position within an authorized stream scope. Externally it +should be opaque and integrity-protected or server-validated. Internally it may +contain run ID, sequence, event ID, and schema version. + +Cursor rules: + +- resume strictly after cursor; +- stable event IDs make duplicates safely deduplicable; +- wrong-organization/run cursor rejects without existence disclosure; +- expired cursor produces explicit resync/reset behavior; +- cursor never advances for an event not durably committed and delivered/visible + according to the selected client contract. + +## Atomic publication + +When the workflow/control plane and timeline share PostgreSQL, append the public +event in the same transaction as the state transition. Publish live delivery +from an outbox/change feed after commit. + +```text +BEGIN + update execution projection + append operator timeline + append public run event + insert delivery outbox +COMMIT +``` + +If Temporal history is authoritative, derive product events through an Activity, +interceptor/visibility integration, or separate projection process with explicit +idempotency and repair. Do not perform an unrecorded external publish directly +from deterministic Workflow code. + +## Replay-to-live handoff + +Avoid loss between durable replay and live subscription: + +1. authorize stream scope; +2. validate cursor; +3. subscribe or capture durable high-water mark; +4. read committed events after cursor through high-water; +5. begin live delivery strictly after high-water; +6. deduplicate by stable event ID if source overlap is possible. + +An ephemeral pub/sub channel can wake a server, but it cannot be the only source +when replay is promised. On host restart, read from durable events. + +## Backpressure + +The workflow engine/control plane must not block on a browser connection. SSE is +a projection consumer with bounded queues. + +Define: + +- maximum serialized event size; +- per-connection buffer; +- connection/write timeout; +- organization/user connection limit; +- replaceable progress event coalescing; +- disconnect-and-resume policy; +- broker consumer lag limits; +- protection for terminal/audit events; +- maximum replay batch and rate. + +Never hold an unbounded array of timeline history to serve a connection. Page or +stream durable reads. If a client is slow, disconnect with the last durable +cursor or coalesce explicitly replaceable progress; never drop terminal events +without replay. + +## Cancellation and authority + +There are two cancellations: + +- delivery cancellation: browser disconnected; stop subscription/encoding; +- workflow cancellation: authorized durable command; engine/control plane must + record and process it. + +Do not couple them. A customer closing a tab should not cancel an import. A +workflow cancellation should emit requested/observed/terminal events as distinct +states where useful. + +Every stream query applies server-owned organization/resource filters. A client +cannot subscribe to an arbitrary run ID merely because it knows it. Stream +tickets, if used, are short-lived and bound to subject, organization, run, +audience, and expiry. + +## Retention and compaction + +Set separate retention for engine history, operator timeline, public events, and +large diagnostic artifacts. Terminal run snapshots can allow public event +compaction while audit history remains longer. + +If an old cursor falls outside retention: + +- return an explicit resync problem; or +- send a versioned current snapshot/reset event and continue after its high-water. + +Do not silently start from “now”; that makes the client believe it observed a +complete run. + +## Tests and failure signatures + +Test: + +- committed transition and public event atomicity; +- API/publisher restart and durable replay; +- event inserted at replay/live handoff; +- duplicated outbox/broker delivery; +- cursor wrong scope, ahead, malformed, and expired; +- slow consumer and bounded memory; +- cancellation of delivery during replay/write; +- workflow cancellation while stream connected; +- terminal event and reconnect after completion; +- redaction and per-event authorization. + +| Signature | Defect | Correction | +|---|---|---| +| UI status changes without event | Projection/event dual write | Atomic event/outbox | +| Stream loses event on reconnect | Pub/sub used as authority or handoff race | Durable events + high-water | +| Worker slows when client stalls | Workflow writes directly to socket | Async projection and bounded delivery | +| Closing tab cancels work | Delivery/domain cancellation conflated | Separate commands/scopes | +| Another tenant's cursor works | Cursor not scope-bound | Server validation and base filter | +| Replay leaks stack/provider data | Raw operator history exposed | Safe versioned projection | + +## Sources and freshness + +- HTML Living Standard SSE protocol: + https://html.spec.whatwg.org/multipage/server-sent-events.html (primary source, + checked 2026-07-17). +- Temporal visibility/history and TypeScript workflow documentation: + https://docs.temporal.io/develop/typescript and https://docs.temporal.io/visibility + (primary sources; raw history is not treated as a public contract). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/` + and `evidence/web/kaiju-site-scope/` (workflow timeline and product-scope + evidence; public stream projection design remains inferred until tested). diff --git a/skills/build-workflows/references/temporal.md b/skills/build-workflows/references/temporal.md new file mode 100644 index 0000000..8b9852e --- /dev/null +++ b/skills/build-workflows/references/temporal.md @@ -0,0 +1,437 @@ +# Temporal TypeScript architecture and operations + +## Contents + +- [When to choose Temporal](#when-to-choose-temporal) +- [Process and package boundaries](#process-and-package-boundaries) +- [Workflow and activity example](#workflow-and-activity-example) +- [Determinism](#determinism) +- [Clients and identity](#clients-and-identity) +- [Signals, queries, and updates](#signals-queries-and-updates) +- [Retries, timeouts, heartbeats, and cancellation](#retries-timeouts-heartbeats-and-cancellation) +- [Schedules, timers, child workflows, and Continue-As-New](#schedules-timers-child-workflows-and-continue-as-new) +- [Workers and task queues](#workers-and-task-queues) +- [Versioning and deployment](#versioning-and-deployment) +- [Projection and domain-state integration](#projection-and-domain-state-integration) +- [Testing and verification](#testing-and-verification) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) + +## When to choose Temporal + +Temporal is a strong candidate when business coordination must survive host loss +and needs long histories, durable timers, message passing, retries, cancellation, +child workflows, operator inspection, or replay across deployments. + +Do not choose it only to run a short queue consumer. Account for Temporal +Service/Cloud operations, namespaces, task queues, Worker deployment, data +conversion/encryption, history growth, deterministic code constraints, and safe +version rollout. + +## Process and package boundaries + +| Boundary | Package | Owns | Must not do | +|---|---|---|---| +| Client/API | `@temporalio/client` | Connect, start, get handles, signal/query/update/cancel/terminate, schedules | Execute workflow code in request process | +| Workflow | `@temporalio/workflow` | Replay-safe orchestration and state | Direct DB/network/filesystem/Node/DOM I/O | +| Activity | ordinary TS plus `@temporalio/activity` context | Side effects, heartbeats, external systems | Assume exactly-once execution | +| Worker | `@temporalio/worker` | Bundle/poll workflows and execute activities | Serve as public API by accident | +| Testing | `@temporalio/testing` | Time-skipping/integration environment | Replace production replay/upgrade tests entirely | +| Service/Cloud | Temporal platform | Histories, task queues, timers, visibility | Own product domain state automatically | + +Keep code in separate modules: + +```text +workflows/ + publish-import.ts + messages.ts +activities/ + imports.ts +client/ + temporal.ts +worker/ + imports.ts +``` + +Workflow modules may import activity types but not activity implementations. The +Worker receives the implementation object and a `workflowsPath`/bundle. + +## Workflow and activity example + +Activity contract and implementation: + +```ts +// activities/imports.ts +export interface ImportActivities { + readonly parseUpload: (input: ParseUploadInput) => Promise<ParsedArtifact> + readonly publishBatch: (input: PublishBatchInput) => Promise<PublishResult> + readonly finalizeImport: (input: FinalizeInput) => Promise<ImportSummary> +} + +export function makeImportActivities(deps: ImportActivityDeps): ImportActivities { + return { + parseUpload: async (input) => await deps.parser.parse(input), + publishBatch: async (input) => await deps.store.publishIdempotently(input), + finalizeImport: async (input) => await deps.store.finalize(input), + } +} +``` + +Workflow definition: + +```ts +// workflows/publish-import.ts +import { proxyActivities } from "@temporalio/workflow" +import type { ImportActivities } from "../activities/imports.ts" + +const { parseUpload, publishBatch, finalizeImport } = + proxyActivities<ImportActivities>({ + startToCloseTimeout: "10 minutes", + retry: { + maximumAttempts: 5, + initialInterval: "1 second", + backoffCoefficient: 2, + maximumInterval: "1 minute", + }, + }) + +export async function publishImport(input: PublishImportInput): Promise<ImportSummary> { + const artifact = await parseUpload(input) + + for (const batch of artifact.batches) { + await publishBatch({ + organizationId: input.organizationId, + importId: input.importId, + batch, + operationKey: `${input.importId}:${batch.index}`, + }) + } + + return await finalizeImport({ + organizationId: input.organizationId, + importId: input.importId, + artifactId: artifact.id, + }) +} +``` + +This is illustrative. A real parser should usually store a large manifest/blob +outside history and return references, not thousands of batches in a workflow +result. Validate exact SDK option names against the installed version. + +Worker: + +```ts +import { NativeConnection, Worker } from "@temporalio/worker" +import * as activities from "../activities/imports.ts" + +const connection = await NativeConnection.connect({ address: config.address }) +const worker = await Worker.create({ + connection, + namespace: config.namespace, + taskQueue: "imports-v1", + workflowsPath: new URL("../workflows/mod.ts", import.meta.url).pathname, + activities: activities.makeImportActivities(resources), + maxConcurrentActivityTaskExecutions: config.activityConcurrency, +}) + +await worker.run() +``` + +Host/runtime details differ across Node and Deno. Verify the Temporal TypeScript +SDK runtime and bundler support for the selected deployment; do not assume a +Deno service can run the Node Worker unchanged. + +## Determinism + +Temporal Workflow code is replayed against Event History in a deterministic +sandbox. The current TypeScript docs state: + +- external state and side effects belong in Activities; +- workflow parameters/results must be serializable, with one object argument + recommended for evolvability; +- Node.js and DOM APIs cannot be used in workflow code; +- Temporal provides deterministic replacements for time/random APIs within the + sandbox, but unavailable APIs such as `crypto.randomUUID()` must not be used; +- workflow logging should use the SDK logger so replay does not duplicate logs. + +Avoid: + +- direct fetch, SQL, filesystem, environment, or process access; +- branching on mutable config loaded at replay time; +- nondeterministic iteration/order that changes command sequence; +- libraries with hidden native/runtime side effects; +- using replay-status checks to change business logic; +- activity implementation imports in workflow bundles. + +Run replay tests on representative production histories before deploying changed +workflow code. + +## Clients and identity + +The API/composition root owns a long-lived Temporal client connection. Start with +a business-meaningful Workflow ID and one object argument: + +```ts +const handle = await temporal.workflow.start(publishImport, { + workflowId: `import:${organizationId}:${importId}`, + taskQueue: "imports-v1", + args: [{ organizationId, importId, uploadId }], + memo: { organizationId, importId }, + // Application-owned mapper; define it against registered Search Attribute types. + searchAttributes: toSearchAttributes({ organizationId, importId }), +}) +``` + +Define Workflow ID reuse/conflict policy explicitly. A stable Workflow ID is an +admission identity, not automatic idempotency for every Activity. + +Use `start` when the API should return after durable start; `execute` waits for +result and is inappropriate for long background work in an HTTP request. Store +or derive safe status links from product-owned IDs. Do not expose raw Temporal +namespace/task-queue internals as the only product API. + +Differentiate cancellation (cooperative), termination (immediate platform stop), +and workflow failure. Termination is an operator last resort and may skip +workflow cleanup/compensation. + +## Signals, queries, and updates + +The current TypeScript SDK distinguishes: + +| Message | Mutates state | Returns value | History/processing behavior | Use | +|---|---:|---:|---|---| +| Query | No | Yes | Read-only; requires Worker to answer | Inspect live state | +| Signal | Yes | No result from handler | Server acceptance returns before workflow processes it | Approval, notify, wake | +| Update | Yes | Yes | Worker validates/accepts and records accepted update | Command needing accepted result | + +Definitions and handlers: + +```ts +import { + condition, + defineQuery, + defineSignal, + defineUpdate, + setHandler, +} from "@temporalio/workflow" + +export const approve = defineSignal<[ApproveInput]>("approve") +export const progress = defineQuery<Progress>("progress") +export const correctMapping = defineUpdate<MappingVersion, [CorrectMappingInput]>( + "correctMapping", +) + +export async function reviewedImport(input: ReviewedImportInput) { + let approved: ApproveInput | undefined + let mapping = input.mapping + + setHandler(progress, () => ({ approved: approved != null, mapping })) + setHandler(approve, (message) => { approved = message }) + setHandler(correctMapping, (message) => { + const previous = mapping + mapping = message.mapping + return previous + }, { + validator: (message) => MappingSchema.parse(message.mapping), + }) + + await condition(() => approved != null) + return await publishApproved({ ...input, mapping, approved: approved! }) +} +``` + +Query handlers must not mutate state or perform async I/O. Signal handlers do not +return a workflow-processed result. Update validators are synchronous and can +reject before the Update is accepted into history. Define initialization and +concurrency rules for async Signal/Update handlers. Complete handlers before +Continue-As-New; exact SDK constraints must be checked. + +Every public message endpoint still requires product authorization and payload +validation. A Temporal handle is not an authorization decision. + +## Retries, timeouts, heartbeats, and cancellation + +### Activity retry + +Activities are normally the retry boundary for transient I/O. Configure: + +- initial/backoff/maximum interval; +- maximum attempts or expiration budget; +- non-retryable application failure types; +- per-operation idempotency/reconciliation; +- rate limits and provider budgets. + +### Timeouts + +Distinguish: + +| Timeout | Meaning | +|---|---| +| Schedule-To-Start | How long an Activity can wait for a Worker | +| Start-To-Close | One Activity attempt runtime | +| Schedule-To-Close | Total Activity execution including retries/queueing | +| Heartbeat | Maximum silence for a heartbeating long Activity | +| Workflow execution/run/task | Workflow-level platform bounds with different semantics | + +Set at least one appropriate Activity execution timeout as required by the SDK. +Do not use a giant timeout instead of heartbeats for long progress-aware work. + +### Heartbeats + +Long Activities heartbeat progress small enough to resume/retry safely. Heartbeat +details can carry a checkpoint such as last committed batch, not the entire +dataset. A heartbeat does not replace domain idempotency or an output manifest. + +Respond to Activity cancellation through heartbeat/abort-aware calls. If a +provider call ignores cancellation, record ambiguous completion and reconcile. + +### Cancellation + +Workflow cancellation is cooperative. Define cancellation scopes for cleanup +that must run, but avoid pretending cleanup can undo all external effects. +Activities need cancellation behavior; disconnected HTTP clients do not +automatically cancel a durably accepted workflow. + +Test cancellation while queued, running an Activity, waiting on a timer, +handling a signal/update, and during compensation. + +## Schedules, timers, child workflows, and Continue-As-New + +Use Temporal Schedules for platform-owned recurring starts. Define overlap, +catch-up, pause/backfill, timezone, and action policy. Do not put an infinite +cron loop inside a Workflow. + +Workflow timers are durable and replay-safe. Use them instead of wall-clock +sleep APIs. + +Use a Child Workflow when a sub-operation needs its own history, identity, +retry/cancel policy, visibility, or independent lifetime. Use an Activity for a +side effect or bounded computation. Do not create a Child Workflow for every +function call. + +Use Continue-As-New to bound Event History for long-lived workflows. Carry the +minimal versioned state into the new run. Coordinate outstanding handlers and +message deduplication; do not Continue-As-New from an Update handler under the +current documented constraints. + +## Workers and task queues + +Task queues route work; they are not product authorization scopes. Define: + +- which worker build/deployment polls each queue; +- workflow and activity types registered; +- namespace and environment isolation; +- concurrency/tuning and downstream capacity; +- worker identity, health/readiness, graceful shutdown; +- sticky queue/cache implications; +- activity rate limits and fairness; +- build compatibility/version routing. + +Readiness means the worker is connected and polling with the intended catalog, +not merely that the process started. Alert on schedule-to-start latency, task +failures, workflow task failures, activity retry exhaustion, poller absence, +history growth, and stuck/cancel-requested runs. + +## Versioning and deployment + +Temporal supports deployment/worker versioning and replay-safe workflow change +strategies, but APIs and recommendations evolve. Inspect the installed SDK and +current official safe-deployment guide before choosing: + +- Worker Deployment Versioning/routing; +- workflow patch/version markers; +- task-queue migration; +- new workflow type/version; +- Continue-As-New onto a compatible version. + +Release gate: + +1. replay new workflow code against representative existing histories; +2. verify payload/data-converter compatibility; +3. deploy compatible activity implementations; +4. route new starts deliberately; +5. retain workers capable of open histories; +6. observe nondeterminism/workflow task failures; +7. define rollback before rollout; +8. retire old code only when no histories require it. + +Changing Activity implementation without changing workflow command history can +still change business behavior on retry. Treat activity version compatibility as +part of the rollout. + +## Projection and domain-state integration + +Temporal history owns coordination facts; PostgreSQL usually owns product domain +state. Common patterns: + +- Activities commit domain transactions and return stable revisions/IDs. +- An outbox projects domain changes to ClickHouse/search/notifications. +- Workflow memo/search attributes support operations, not full domain state. +- Product API reads a database projection and may query Temporal for live + workflow-only state. + +Never dual-write a domain row and “workflow projection updated” across systems +without idempotency/reconciliation. If start follows a domain transaction, use a +transactional outbox/dispatcher or a reconciler for missing starts. If workflow +start precedes the domain record, make the first Activity idempotently create or +find it. + +## Testing and verification + +- Unit-test pure workflow branches with mocked Activities where useful. +- Use the Temporal test environment/time skipping for timers and messages. +- Integration-test a real Worker, Client, and representative Activities. +- Replay stored histories under candidate workflow bundles. +- Verify activity idempotency after timeout-after-commit and worker kill. +- Test Signal/Query/Update authorization at the API boundary and semantics in the + workflow. +- Exercise every timeout, retry exhaustion, heartbeat timeout, cancellation, and + termination path. +- Kill Worker during Activity and workflow task; restart another Worker. +- Test schedules, overlap/catch-up, child cancellation, Continue-As-New, and + history growth. +- Deploy two compatible Worker versions and execute rollout/rollback drills. +- Reconcile missing/stale PostgreSQL projections. + +## Failure signatures + +| Signature | Likely defect | Correction | +|---|---|---| +| Nondeterminism error after deploy | Workflow command path changed incompatibly | Replay gate and version strategy | +| Activity repeats external charge | Effect not idempotent | Provider/local operation key and reconcile | +| Signal call returns but UI sees no change | Signal acceptance mistaken for processing | Query/projection/event confirmation | +| Query hangs with healthy Temporal Service | No compatible Worker polling | Worker readiness/task queue check | +| Run remains open forever | Missing timeout/cancel/operator policy | Explicit lifecycle bounds | +| History grows without bound | No Continue-As-New/partitioning | History budget and rollover | +| Old run cannot deserialize input | Breaking payload/data converter | Versioned decoder/compatibility worker | +| API user controls arbitrary Workflow ID | Missing product auth/scope mapping | Server-owned identity and policy | +| Activity timeout retries permanent failure | Retry classification absent | Non-retryable failure type | +| Worker works in Node fixture but not Deno host | Runtime/bundling assumption | Dedicated compatible worker deployment | + +## Deliberate exclusions + +- Do not use a Temporal Workflow as the primary store for query-heavy domain + entities. +- Do not call the Temporal Client inside Workflow code. +- Do not put large CSVs, WARC records, or generated artifacts into history. +- Do not use termination as ordinary cancellation. +- Do not assume Workflow ID uniqueness makes external effects exactly once. +- Do not run a Worker in a constrained edge request runtime without explicit SDK + compatibility and lifecycle evidence. + +## Sources and freshness + +- Temporal TypeScript developer guide: https://docs.temporal.io/develop/typescript + (primary source, checked 2026-07-17). +- Workflow basics and deterministic constraints: + https://docs.temporal.io/develop/typescript/workflows/basics +- Message passing: https://docs.temporal.io/develop/typescript/workflows/message-passing +- Activity timeouts: https://docs.temporal.io/develop/typescript/activities#activity-timeouts +- Safe deployment/versioning: https://docs.temporal.io/develop/typescript/workflows/versioning + and https://docs.temporal.io/develop/safe-deployments +- TypeScript SDK API: https://typescript.temporal.io/ (package/API signatures are + version-sensitive; typecheck examples against the installed SDK). +- Attachment, verified 2026-07-17: `evidence/app/new-finance/.agents/research/inngest_effect_architecture_handoff.md` + (comparative design evidence only; the supplied repositories do not implement + a Temporal runtime). diff --git a/skills/build-workflows/references/workers.md b/skills/build-workflows/references/workers.md index 17f05ef..a09fc8a 100644 --- a/skills/build-workflows/references/workers.md +++ b/skills/build-workflows/references/workers.md @@ -1,38 +1,223 @@ -# Workers, queues, timers, waits, and signals +# Workers, queues, schedules, waits, and flow control -## Queue claims and leases +## Contents -Claim work atomically. Use database locking such as `FOR UPDATE SKIP LOCKED`, an -atomic update with returned rows, or the queue engine's claim primitive. A plain -read followed by conditional update can race. +- [Worker process contract](#worker-process-contract) +- [Atomic queue claims and leases](#atomic-queue-claims-and-leases) +- [Retry and poison work](#retry-and-poison-work) +- [Timers and schedules](#timers-and-schedules) +- [Waits, signals, and updates](#waits-signals-and-updates) +- [Backpressure and fairness](#backpressure-and-fairness) +- [Cancellation and shutdown](#cancellation-and-shutdown) +- [Observability](#observability) +- [Tests and failure signatures](#tests-and-failure-signatures) -Store lease owner, expiry, attempt, heartbeat, and last error. Recover expired -leases without duplicating a non-idempotent effect. +## Worker process contract + +The worker entrypoint owns: + +- validated environment-specific config; +- database/queue/engine connections; +- exact workflow/activity catalog; +- queue/task-queue names and build/version identity; +- loop supervision and failure policy; +- concurrency, rate, lease, heartbeat, polling, and batch settings; +- health/readiness and graceful shutdown; +- logs, traces, metrics, and secret redaction. + +Worker readiness requires a compatible catalog and active pollers/loops against +the intended queue/engine. A boot function returning `{ loops: [] }` or a process +blocked before registration is not ready. + +Separate worker deployments by permissions and scaling profile when appropriate: +CPU-heavy parsing, network crawling, payment effects, and lightweight timers +should not necessarily share one queue or credentials. + +## Atomic queue claims and leases + +Use one atomic claim primitive. PostgreSQL example: + +```sql +with candidates as ( + select queue_item_id + from workflow_ready_queue + where status = 'ready' and available_at <= now() + order by priority asc, available_at asc, queue_item_id asc + for update skip locked + limit $1 +) +update workflow_ready_queue q +set status = 'leased', + lease_owner = $2, + lease_expires_at = now() + $3::interval, + attempt = attempt + 1, + fencing_token = fencing_token + 1 +from candidates c +where q.queue_item_id = c.queue_item_id +returning q.*; +``` + +Parameterize safely; SQL is illustrative. Stable ordering needs a tie-breaker. + +Lease semantics: + +- owner uniquely identifies worker process/build; +- expiry is durable and based on database/engine time where possible; +- heartbeat extends only a still-owned generation; +- completion/ack requires matching owner/fencing token; +- expired work becomes recoverable; +- a late worker cannot commit after takeover; +- effect idempotency still protects external consequences. ## Retry and poison work -Define retryable failure classes, attempt limit, exponential/backoff policy, -jitter, timeout, and budget. Separate permanent invalid input from transient -unavailability. Move poison work to a visible terminal/dead-letter state with an -inspect and replay procedure. +Classify before retry: + +| Failure | Examples | Action | +|---|---|---| +| Invalid/permanent | Schema, unsupported version, auth/policy | Quarantine/fail, no retry | +| Transient | 503, connection loss, lock timeout | Backoff/jitter within budget | +| Rate limited | 429/provider quota | Honor reset/retry-after and global budget | +| Ambiguous | Timeout after possible commit | Reconcile by operation key | +| Defect | Invariant/code bug | Fail/alert; retry only after rollout/decision | +| Cancellation | User/operator/shutdown | Cooperative terminal/requeue policy | + +Persist attempt count, next available time, safe structured last error, and +history. Move exhausted/permanent work to a visible dead-letter/quarantine state +with inspect, correct, and redrive commands. Do not retry malformed poison work +forever. + +Coordinate retry budgets across engine, queue, Activity, HTTP client, and +provider SDK. Nested retries can multiply load and exceed business deadlines. + +## Timers and schedules + +Differentiate: + +- durable one-shot delay; +- workflow timer/wait timeout; +- fixed interval; +- calendar/cron schedule; +- backfill/replay campaign. + +Schedule contract includes timezone, daylight-saving behavior, overlap, missed +runs, catch-up window, jitter, pause/resume, backfill, start identity, and +cancellation. + +Do not compute cron behavior with an interval helper. The uploaded scheduler +explicitly declines to advance cron without calendar/timezone expansion; retain +that honesty. Use a tested parser/platform and fixture DST transitions. + +Publishing a due schedule and advancing `next_run_at` must be atomic or +reconciled. Stable schedule-occurrence IDs deduplicate repeat scans. + +## Waits, signals, and updates + +Wait record: + +- wait ID, execution ID, name/type, schema version; +- authorization scope and correlation key; +- created/timeout timestamps; +- resume token/engine identity; +- active/resolved/timed-out/cancelled state; +- winning message/event ID and payload reference; +- buffered signal policy. + +Signal procedure: + +1. authenticate/authorize sender and execution scope; +2. validate message version/payload; +3. deduplicate signal ID; +4. find eligible wait or buffer according to contract; +5. atomically choose winner against timeout/cancel; +6. enqueue/complete engine resume and append timeline; +7. acknowledge acceptance versus processed result accurately. + +Do not lose early signals that arrive before a wait unless the contract explicitly +rejects them. Bound buffered signals and retention. + +Queries are read-only. Signals are accepted asynchronous state changes. Updates +can return an accepted/processed result in engines such as Temporal. A custom +control plane must define its own semantics rather than using the names loosely. + +## Backpressure and fairness + +Control: + +- global and per-worker concurrency; +- per-tenant/workflow/resource slots; +- provider concurrency/rate; +- queue depth and admission shedding; +- claim batch and prefetch; +- memory/open files/connections; +- priority aging/starvation prevention; +- parked runs versus active slots; +- replay/backfill budgets separated from live traffic. + +Concurrency limits active execution; a durable wait normally should not consume +an active worker slot. Store enough state to reacquire admission fairly on resume. + +Avoid prefetch larger than workers can heartbeat/process within lease duration. +Measure queue wait, runnable age, schedule-to-start, active slots, retry rate, +tenant fairness, provider saturation, and oldest poison item. + +## Cancellation and shutdown + +Cancellation states can include requested, observed, compensating/cleanup, and +terminal cancelled. Workers check cancellation before new effects and at safe +points in long work. Heartbeats can deliver cancellation in engines that support +it. + +Shutdown: + +1. mark worker unready; +2. stop claims/polls; +3. request interruption of in-flight work where safe; +4. drain within deadline while heartbeating owned leases; +5. acknowledge completed effects and persist checkpoints; +6. leave/release remaining work according to engine contract; +7. flush telemetry and close resources. + +Never release a lease while its old effect can still commit without fencing. + +## Observability -## Timers, waits, and signals +Per work item record queue wait, lease owner/generation, workflow/run, activity, +attempt, retry delay, deadline, cancellation, worker build, safe cause, and +result/checkpoint identity. Correlate with originating request/event and tenant +without leaking payloads. -Persist timer/wait identity and resolution status. Make timeout, signal, and -cancellation race-safe. Resolving a wait, appending its timeline event, and -enqueueing resume must be atomic or repairable. +Alert on no pollers, expired leases, oldest runnable age, retry storms, dead +letters, stuck cancellation, due timers, unprocessed signals, worker loop exit, +and downstream saturation. -Deduplicate signals and define buffering for signals that arrive before the -workflow waits. Validate signal payloads and authorization. +## Tests and failure signatures -## Schedules and backpressure +- Run two workers claiming the same queue under load. +- Kill after claim, effect commit, checkpoint, and ack. +- Let a late old owner commit after lease takeover; fencing must reject it. +- Test retry classes, `Retry-After`, budgets, and poison redrive. +- Exercise interval/cron timezone, DST, missed/overlap/catch-up behavior. +- Race signal, timeout, cancellation, duplicate, and early signal. +- Saturate one tenant/provider and verify fairness/backpressure. +- Shutdown with idle, queued, running, waiting, and heartbeating work. -Specify missed-run, overlap, catch-up, timezone, daylight-saving, and clock-skew -behavior. Bound queue depth, worker concurrency, per-tenant fairness, downstream -rate, and memory. Admission and worker limits must not contradict each other. +| Signature | Defect | Correction | +|---|---|---| +| Same item leased twice | Read/update race or expiry error | Atomic claim + generation | +| Old worker overwrites new result | No fencing/CAS | Generation-bound commit | +| Cron drifts or doubles at DST | Interval/naive local-time math | Calendar/timezone engine | +| Signal accepted but run never wakes | Non-atomic wait/resume | Transaction/reconciler | +| Queue grows while CPU idle | Admission/worker settings contradict | End-to-end capacity audit | +| One tenant starves others | Global FIFO only | Scoped limits/fair scheduling | +| Worker “ready” with dead loop | Readiness ignores supervision | Loop health policy | -## Worker lifecycle +## Sources and freshness -Prove boot, health, graceful drain, cancellation, active lease handling, logger -flush, and deployment reachability. A worker package not started by deployment -is not an implemented workflow capability. +- PostgreSQL `SELECT ... FOR UPDATE`/`SKIP LOCKED` documentation: + https://www.postgresql.org/docs/current/sql-select.html#SQL-FOR-UPDATE-SHARE + (primary source, checked 2026-07-17). +- Temporal TypeScript Worker documentation: https://docs.temporal.io/develop/typescript/workers + (primary source; exact tuning/versioning options are SDK-version-sensitive). +- Attachments, verified 2026-07-17: `evidence/app/new-finance/utils/workflows/worker/` + and `utils/workflows/env.ts` (observed source with cron and atomicity gaps called out). diff --git a/skills/deliver-software/SKILL.md b/skills/deliver-software/SKILL.md index f694603..4a87ac1 100644 --- a/skills/deliver-software/SKILL.md +++ b/skills/deliver-software/SKILL.md @@ -37,16 +37,19 @@ from imports, configuration, and surrounding code, then load only that framework's reference. When a focused workflow skill is active, let it own its domain references. In -particular, `build-web`, `build-sites`, and `build-web-apps` replace the bundled -general web/framework references for their in-scope decisions. Do not load both -sets by habit. Use this skill for authority, lifecycle, cleanup, and the final -verdict, then use the focused skill for changing contracts and verification. +particular, `build-libraries` owns reusable programming models and package +contracts, while `build-web`, `build-sites`, and `build-web-apps` replace the +bundled general web/framework references for their in-scope decisions. Do not +load both sets by habit. Use this skill for authority, lifecycle, cleanup, and +the final verdict, then use the focused skill for changing contracts and +verification. ## Route the task | Work | Read | | --- | --- | | Code, architecture, API design, refactor, or migration | [general.md](references/general.md) | +| Reusable library, SDK, public package API, tree-shaking, streaming contract, resource-owning API, or application-to-library extraction | Use `build-libraries` when installed; otherwise apply [general.md](references/general.md), [typescript.md](references/typescript.md), [testing.md](references/testing.md), and [benchmarks.md](references/benchmarks.md) as applicable | | Material dependency selection, integration, replacement, or upgrade | Use `explore-ecosystems` when installed; otherwise apply the ecosystem preflight below | | Substantial plan, implementation, refactor, migration, completion audit, or multi-surface change | [workflow.md](references/workflow.md) and [delivery.md](references/delivery.md) | | Refactor, migration, cutover, compatibility window, or legacy removal | [refactors.md](references/refactors.md) | diff --git a/skills/explore-ecosystems/references/evidence.md b/skills/explore-ecosystems/references/evidence.md index c130a6d..6830a55 100644 --- a/skills/explore-ecosystems/references/evidence.md +++ b/skills/explore-ecosystems/references/evidence.md @@ -1,53 +1,298 @@ -# Evidence and provenance +# Evidence, versions, and provenance -## Evidence order +## Contents -Use the strongest available evidence for each claim: +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Evidence strength is claim-specific](#evidence-strength-is-claim-specific) +- [Installed, source, published, and current truth](#installed-source-published-and-current-truth) +- [Claim ledger](#claim-ledger) +- [Source acquisition and integrity](#source-acquisition-and-integrity) +- [Proving relationships and support](#proving-relationships-and-support) +- [Negative evidence and contradictions](#negative-evidence-and-contradictions) +- [Executable evidence](#executable-evidence) +- [Freshness and change control](#freshness-and-change-control) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Sources and freshness](#sources-and-freshness) -1. the consuming repository's resolved lockfile, source imports, tests, and - executable behavior; -2. source and tests at the exact installed revision; -3. canonical manifests, generated API documentation, and release notes; -4. current official documentation and examples; -5. maintainer announcements or issue discussions; -6. community examples and search summaries as discovery leads only. +## When to load this reference -Do not let newer documentation silently override the installed version. Do not -let a local README prove behavior that the code and tasks contradict. +Load this reference for every ecosystem investigation and whenever a decision +depends on version-sensitive, experimental, private, generated, remote, negative, +or conflicting claims. It defines how to prevent a detailed skill or report from +becoming a detailed hallucination. -## Claim record +## Outcome -For every decision-changing claim, record: +Every decision-changing statement has a source, exact identity/version where +relevant, status, freshness date, and verification level. Contradictions remain +visible until resolved. The report never combines facts from incompatible +versions or promotes inference to observed behavior. -- the exact claim; -- source path or URL; -- package/repository version or commit; -- verification date; -- whether it was observed, documented, inferred, or unresolved; -- the decision it affects. +## Evidence strength is claim-specific -Use primary sources at the claim level. A single repository URL is not enough -when package identity, adapter status, and runtime support come from different -places. +Use the strongest evidence that actually proves the claim. -## Installed truth versus current truth +| Claim | Strong evidence | Common insufficient evidence | +|---|---|---| +| Repository resolves package version | lockfile/resolver output | manifest range, current website | +| Public export exists | exact published artifact/export map plus import | README example alone | +| Runtime behavior | source/tests and executable exact-version check | types or similar API | +| Adapter is official | canonical project integration index/maintainer ownership | same org/name/search result | +| Runtime/renderer support | peer/engine metadata plus CI/integration test | generic TypeScript compatibility | +| Configuration precedence | exact version implementation/tests/docs | one config example | +| Package is published | registry artifact and metadata | workspace directory | +| Feature absent | exhaustive relevant exports/source/artifact search | no README mention | +| Performance improvement | controlled raw experiment | anecdote/one fastest sample | +| Archive evolution | normalized content/revision diff | filenames such as old/new | +| Reproducible release | source/input/tool provenance plus matching rebuild | CI success/attestation alone | -Inspect both: +Repository-local evidence is strongest for what this repository currently does. +Official current docs are strongest for current intended usage. Neither replaces +the other; write the boundary. -- installed truth determines what the repository can use now; -- current truth determines available upgrades, deprecations, and migration - paths. +## Installed, source, published, and current truth -When they differ, state the boundary explicitly. Prerelease packages, generated -clients, private adapters, forks, patches, overrides, and vendored source require -revision-specific evidence. +Separate four views: -## Negative evidence +```text +declared truth = manifest ranges/configuration intent +resolved truth = lockfile/artifact actually selected +source truth = code/tests at exact revision +published truth = files/metadata consumers receive +current truth = latest stable/prerelease docs and releases +``` -Absence is a claim too. Before saying a feature or export does not exist, search -the public entrypoints, package exports, source tree, generated artifacts, tests, -and relevant branches or tags. Phrase incomplete searches narrowly. +They can disagree legitimately or reveal a defect. Example: -For example, a README that advertises `stringify` while the published export map -and source contain no implementation proves a contract discrepancy. It does not -prove what a future release will contain. +```text +repository range: ^3.3.0 +resolved c12: 3.3.4 +current research artifact: 4.0.0-beta.5 + +Decision: use 3.3.4 API/behavior for implementation; discuss beta only as a +future migration/status item. Never copy beta source-layer or optional-format +behavior into a stable 3.x claim without exact evidence. +``` + +Inspect patches, overrides, forks, vendored code, workspace aliases, prereleases, +generated clients, and platform-specific optional packages. The lock entry may +resolve a wrapper whose behavior differs from upstream. A package's docs branch +may track main rather than the installed release. + +For artifacts, inspect the archive, not only repository source. Files can be +excluded, generated differently, or mapped through export conditions. The +Wikitext archive's documentation/export discrepancy is a useful negative case: +documentation can advertise an API not implemented/exported in the supplied +revision. + +## Claim ledger + +Use a structured record rather than a references list at the end. + +```json +{ + "claim": "The package exports a browser-safe parser entrypoint", + "identity": "package@1.2.3", + "source": "registry tarball package.json + clean browser import", + "sourceRevision": "sha256:...", + "observedAt": "2026-07-17", + "status": "executable", + "scope": "ESM browser bundle only; filesystem helpers excluded", + "decision": "use ./core in worker adapter", + "unresolved": [] +} +``` + +Allowed status vocabulary should be small and defined. This repository uses: + +- `normative`: desired contract from a guide/policy, not implementation proof; +- `observed-source`: exact source/docs/artifact inspected; +- `executable`: behavior ran with recorded command/result; +- `experimental`: prerelease/research/unstable evidence; +- `counterexample`: evidence that disproves or limits a broad claim; +- `inferred`: reasoning from evidence, not directly observed; +- `unresolved`: identity/behavior/support not established. + +One source can support several claims, but each claim must point to the relevant +path/section/export/test. A repository root URL does not prove package identity, +adapter status, configuration semantics, and host support simultaneously. + +Record decision impact. This makes stale claims reviewable: if an upstream +version changes, only dependent decisions need re-evaluation. + +## Source acquisition and integrity + +For uploaded/local archives: + +- record original artifact name, byte digest, extraction root, and archive + integrity result; +- prevent path traversal/symlink escape during extraction; +- preserve original artifact; analyze a read-only/extracted copy; +- compare duplicate candidates by normalized relative paths/content hashes; +- record claim paths, not just archive name; +- do not redistribute source in deliverables unless authorized/licensed. + +For registry packages: + +- capture exact version, registry URL, integrity/digest, manifest, exports, + files, license, and relevant declarations/source/tests; +- prefer registry tarball over mutable repository main for published behavior; +- verify package/repository ownership links; beware similarly named packages; +- do not execute lifecycle scripts merely to inspect content. + +For Git/source: + +- use an exact commit/tag and verify tag/revision relationship where material; +- record submodules/LFS/generated artifacts/patches; +- distinguish default branch, release tag, and installed source; +- preserve line/path references or local evidence extracts. + +For web/docs: + +- prefer versioned official docs, generated API docs, changelogs, and release + notes; record URL/date/version; +- mutable `latest` pages require freshness recheck and cannot silently define + an older installed version; +- search snippets/community posts are discovery leads; follow to primary source. + +Do not expose credentials while acquiring private sources. Record access/blocker +status without copying private content into a public source ledger. + +## Proving relationships and support + +Relationship claims need separate evidence from feature claims. + +To call a sibling/adapter official, find at least one canonical maintainer-owned +surface and preferably compatibility tests/release ownership. Same scope, +organization, contributor, or package keyword is insufficient. + +To claim compatibility, record exact dimensions: + +```text +package A version + x adapter version + x framework/renderer version + x runtime/platform + x configuration/feature +``` + +Types can prove structural compatibility at compile time. They do not prove SQL +dialect behavior, runtime globals, serialization, lifecycle, error propagation, +SSR/hydration, browser bundling, native ABI, or service protocol compatibility. +Use source/tests and executable integration. + +Examples/starters prove that a combination was intended/tested at their own +revision. Resolve their lockfile and date before treating them as current. + +## Negative evidence and contradictions + +Absence is an expensive claim. Before saying a feature/export/adapter does not +exist, search: + +- public export maps/root/subpaths/bin; +- source modules, generated files, types, tests, examples; +- workspace packages and separate official repositories; +- current and installed release branches/tags; +- integration/plugin indexes and deprecation/migration notes; +- registry artifacts for the exact version. + +Phrase bounded conclusions: "No `stringify` export was found in the supplied +Wikitext revision's `mod.ts`, source modules, tests, or package config, despite +README language" is supportable. "Wikitext has never supported stringify" is +not proven. + +Do not average contradictions away. Record: + +| Claim | Source A | Source B | Resolution | +|---|---|---|---| +| Export exists | README | published export map lacks it | contract discrepancy; do not use | +| Archives are old/new | filenames | identical normalized hashes | no evolution evidence | +| Adapter supports host | types compile | integration fails | unsupported/unresolved runtime behavior | + +Normative guides describe what software should do. Code proves what the inspected +revision does. A guidebook's detailed desired architecture cannot be presented +as already implemented without source/tests. + +## Executable evidence + +Prefer the smallest check that distinguishes the claim from plausible alternatives: + +- clean import for public export/package contents; +- type matrix for declaration/resolver claims; +- config fixture for precedence/merge/provenance; +- generated SQL/protocol trace for dialect/driver behavior; +- SSR/client build for renderer integration; +- failure/cancellation/disposal test for lifecycle; +- clean pack/install for packaging; +- raw controlled benchmark for performance; +- check/write/check/hash for generation. + +Record command, cwd/fixture, exact versions, environment, exit, stdout/stderr +summary, produced artifact/digest, and whether the result was observed or blocked. +A blocked registry/credential/hardware check stays blocked; it does not become +documented success. + +A smoke test proves only its path. Pair with source/contract inspection for +unexercised options and failure modes. Do not use a mock to claim a real service, +driver, registry, or cross-platform workflow passed. + +## Freshness and change control + +Set freshness based on volatility: + +- exact archived artifact/revision: stable for that identity; +- registry version: stable artifact, mutable deprecation/channel metadata; +- current docs/package index: recheck at decision time; +- prerelease/alpha: pin and recheck exports frequently; +- service API/pricing/quota/deployment support: recheck immediately; +- user memory/private source: unresolved until located. + +When refreshing, do not overwrite old claim records invisibly. Record prior/new +version, changed claim, affected decisions, migration, and verification. Preserve +source digests and evaluation fixtures for released skill guidance. + +Skill references should state version boundaries and source status near fragile +examples. Avoid exact API code for an unresolved/private surface; provide a +protocol/interface placeholder labelled local and require source inspection. + +## Failure signatures + +| Signature | Evidence defect | Next inspection | +|---|---|---| +| Detailed API is not in installed package | versions merged or invented | lock, tarball exports/source | +| Root GitHub URL cited for every claim | provenance too coarse | package/path/test-level records | +| README says yes, runtime says no | docs/artifact contradiction | published exports and executable test | +| Missing feature claimed after one search | negative evidence too weak | full public/workspace/integration surfaces | +| "Official" based on scope/name | relationship unproven | canonical index/maintainer/release evidence | +| Current docs applied to old lock | installed/current truth collapsed | versioned docs/tag source | +| Similar archives described as versions | identity names trusted | normalized hashes/diffs | +| Attestation treated as correctness | provenance scope misunderstood | behavior/semver/consumer gates | +| Mock result presented as integration | evidence level inflated | real connected check or blocked status | +| Private package gets plausible imports | memory promoted to API | consuming source/exports or unresolved | + +## Deliberate exclusions + +- Do not cite search result snippets as implementation evidence. +- Do not execute untrusted packages or lifecycle scripts to inspect metadata. +- Do not combine stable and prerelease capabilities into one API. +- Do not label inference, normative policy, mock behavior, or type compatibility + as observed runtime behavior. +- Do not claim universal absence from a bounded search. +- Do not copy private source/credentials into public reports or skill bundles. +- Do not provide a long bibliography without claim-level mapping. + +## Sources and freshness + +- `evals/sources.json` in this repository, source identity/status/digest/claim + path registry for all retained uploads and current primary sources; validated + against 16 distinct uploaded artifacts on 2026-07-17. +- Retained `old-finance`/`new-finance` duplicate archives and Wikitext + documentation/export discrepancy, observed counterexamples to name- and + README-based inference; verified 2026-07-17. +- Pinned registry source records for current deep package references, including + exact integrities and explicit prerelease boundaries; verified 2026-07-17. + +Re-run source verification and exact-version inspection when the source registry, +installed graph, or upstream release changes. diff --git a/skills/explore-ecosystems/references/failures.md b/skills/explore-ecosystems/references/failures.md index ca4ccb8..b3c3475 100644 --- a/skills/explore-ecosystems/references/failures.md +++ b/skills/explore-ecosystems/references/failures.md @@ -1,30 +1,212 @@ -# Failure signatures and counterexamples +# Ecosystem failure signatures and anti-hallucination recovery -| Signature | Likely cause | Next evidence | +## Contents + +- [How to use this reference](#how-to-use-this-reference) +- [Identity and provenance failures](#identity-and-provenance-failures) +- [Topology and selection failures](#topology-and-selection-failures) +- [Version and API failures](#version-and-api-failures) +- [Integration and ownership failures](#integration-and-ownership-failures) +- [Runtime, data, and operational failures](#runtime-data-and-operational-failures) +- [Research-process failures](#research-process-failures) +- [Counterexamples that must remain in the model](#counterexamples-that-must-remain-in-the-model) +- [Recovery protocol](#recovery-protocol) +- [Verification](#verification) +- [Sources and freshness](#sources-and-freshness) + +## How to use this reference + +Use this table when research or implementation produces a contradiction, +surprising absence, incompatible behavior, or a response that sounds complete +but lacks exact evidence. A signature suggests the next discriminating evidence; +it is not itself a diagnosis. + +Anti-hallucination rule: + +```text +observe signature + -> restate the exact bounded claim + -> list plausible competing explanations + -> inspect the source/check that distinguishes them + -> lower status to inferred/unresolved until proven + -> update selection/integration or stop +``` + +## Identity and provenance failures + +| Signature | Plausible causes | Next discriminating evidence | Unsafe shortcut | +|---|---|---|---| +| Package name cannot be resolved | typo, private package, unpublished workspace, different registry | consuming imports/manifests/lock/local source | invent an npm/JSR API | +| Repository link and package owner differ | fork, transfer, malicious/similar name, stale metadata | registry provenance, canonical docs, release signer | trust name/README badge | +| Two archives called old/new look alike | duplicate exports or repackaging | normalized path/content hashes | narrate evolution from filenames | +| Binary has no source/version | editor download, cache, vendor artifact, malware | file type/digest, installer/config, provenance manifest | execute it for `--version` | +| Current docs have no version selector | mutable main/latest docs | installed tag source, tarball declarations/tests | apply latest to lock version | +| Registry integrity differs from retained artifact | repack, wrong registry, corruption | registry metadata/download/digest/source revision | ignore because version matches | +| Guidebook describes feature absent in code | normative target, partial implementation, stale code | exact source/tests/tasks | say repository already supports it | +| Source exists but published import fails | omitted file/export/build transform | packed artifact/export map/clean import | cite repository path as public API | + +## Topology and selection failures + +| Signature | Plausible causes | Next discriminating evidence | Unsafe shortcut | +|---|---|---|---| +| Core package appears too small | capability split into siblings/adapters or docs stale | workspace/packages/exports/integration index | invent methods on core | +| Many same-scope packages discovered | package family or incidental organization adjacency | capability/dependency/official relationship map | install all packages | +| Same maintainer or org | first-party sibling or unrelated project | canonical project docs/source edges | call it an official integration | +| Similar names/APIs across packages | alternative, fork, adapter, coincidence | ownership, dependency direction, spec/conformance | treat as drop-in compatible | +| Official example uses another library | supported recipe at one revision, illustrative only | example lock/CI/current integration docs | make it mandatory companion | +| One tool already owns the concern | candidate is alternative or migration target | existing wrapper/config/tests/consumers | stack duplicate owners | +| Agent researches every transitive package | materiality/stopping rule absent | decision-capability path | produce exhaustive org catalog | +| No siblings found | genuinely standalone or discovery incomplete | workspace/org/docs/registry bounded search | manufacture an ecosystem | + +## Version and API failures + +| Signature | Plausible causes | Next discriminating evidence | Unsafe shortcut | +|---|---|---|---| +| Type/method exists only in docs | beta/newer line or docs bug | exact installed declarations/source/export | copy current example | +| Stable and beta capabilities both appear | versions merged in notes | claim ledger by version | present a superset API | +| Types compile, runtime throws | host/global/module/dialect/lifecycle mismatch | exact runtime integration and emitted trace | call structural typing support | +| Root import works, subpath fails | export omitted/condition mismatch | packed export map and resolver trace | deep-import source path | +| CJS example fails in ESM package | module-system boundary | package exports/type and target runtime | add `require` shim blindly | +| Config option ignored | wrong version/source layer/name or loader not active | exact config schema/implementation/effective provenance | assume default applied | +| Optional peer becomes required at runtime | feature path eagerly imported/build bundled | source graph and absence test | add every optional peer | +| Private adapter resembles Drizzle/Effect API | local design inferred from public analogue | actual source/export/tests | invent matching methods | + +## Integration and ownership failures + +| Signature | Plausible causes | Next discriminating evidence | Unsafe shortcut | +|---|---|---|---| +| Two tools both load config | duplicate authority or unbounded bridge | composition root/effective config provenance | merge outputs implicitly | +| Diagnostics appear twice | logger bridge plus inherited parent sink | category/sink ownership and recorder test | filter duplicate strings | +| JSON stdout has prefixes | diagnostics/result channels collapsed | sink routing/parent inheritance/subprocess bytes | write directly to console | +| Arrays duplicate after layering | generic default concatenation used for operation semantics | custom merge orientation and fixtures | dedupe after merge | +| Framework plugin builds but hydration fails | wrong renderer/SSR/client boundary | exact adapter peer matrix and hydration test | blame app code only | +| Generated config loses comments | value serializer used on authored syntax | AST/format-aware transform and unsupported-shape test | overwrite from parsed object | +| Package works only in workspace | hoist/alias/unpublished file/undeclared dep | packed clean consumer | add root dependency without ownership | +| Upgrade leaves old behavior | old adapter/config/cache/generated artifact still owns path | complete consumer/provenance/removal map | delete random cache | + +## Runtime, data, and operational failures + +| Signature | Plausible causes | Next discriminating evidence | Unsafe shortcut | +|---|---|---|---| +| Retry duplicates side effect | non-idempotent activity or missing idempotency key | persisted operation/attempt records | increase retries | +| Workflow restarts from beginning | ordinary Effect service mistaken for durable engine | workflow history/checkpoint/store contract | say Effect is automatically durable | +| Temporal workflow nondeterminism | I/O/time/random/library in workflow code | replay test and workflow/activity split | catch retry exception | +| ClickHouse projection differs | delivery gap, mutable aggregate, dedup/order assumption | source-of-truth reconciliation/query semantics | treat it as OLTP backup | +| Drizzle-like ClickHouse adapter emits wrong SQL | relational dialect inferred, AST/session incomplete | generated SQL + ClickHouse integration | rely on query-builder types | +| Auth plugin exists but schema/session fails | one-sided registration/migration/host cookie config | server/client/plugin/adapter matrix | only add client plugin | +| Shutdown loses logs/jobs | async sinks/workers not disposed or deadline too short | lifecycle trace and pending queue | call process exit | +| Offline/cache returns wrong template | cache key omits ref/subdir/integrity | provenance manifest and immutable acquisition | trust `--offline` result | +| Release succeeded in one registry only | per-target state collapsed | target ledger and exact artifact | rerun entire pipeline/rebuild | +| Benchmark microcase wins, users regress | protected workflow/measurement boundary absent | E2E, memory, error, stress reports | advertise fastest number | + +## Research-process failures + +| Signature | Process defect | Correction | |---|---|---| -| Core package appears too small for documented features | Capability lives in siblings or adapters | Workspace packages, exports, integration index | -| Example imports a package absent from the repository | Version drift, unpublished example, or separate repo | Example revision, registry metadata, release notes | -| Types pass but runtime fails | Host, native, bundler, or lifecycle incompatibility | Exact runtime matrix and executable reproduction | -| Two tools both configure the same concern | Duplicate ownership | Composition root and configuration provenance | -| Adapter API resembles another dialect | Unsupported inferred behavior | Adapter source, tests, generated SQL or protocol trace | -| Documentation promises an export that source lacks | Aspirational or stale contract | Export map, publish include list, registry artifact | -| Same organization contains many packages | Brand adjacency mistaken for required stack | Capability map and deliberate exclusions | -| Community integration is called official | Provenance collapsed | Maintainer and canonical documentation evidence | -| Upgrade changes behavior despite matching names | Version or prerelease boundary | Changelog, exact imports, lockfile, migration tests | -| Agent researches every incidental dependency | Trigger is too broad | Restate the decision and materiality threshold | - -## Counterexamples - -- A standalone package may have no meaningful siblings. Classify it as - standalone after checking; do not manufacture an ecosystem. -- Two archives named “old” and “new” may contain identical source. Compare - content hashes before inventing an evolution narrative. -- A repository named for a product scope may not implement every surface implied - by the name. Search entrypoints and manifests before deriving guidance. -- A guidebook is normative evidence, not proof that its companion codebase - implements the contract. -- A private package remembered by the user may be misspelled, unpublished, or - only present in another workspace. Find source before writing imports. - -Stop and report uncertainty when identity, ownership, or compatibility would -otherwise be guessed. State the exact source or executable check needed next. +| Answer lists package names and slogans | no capability/decision model | owner table, exact versions, failures, proof | +| Large reference has no sources/version | detail not grounded | claim-level provenance and freshness | +| Code example uses plausible unknown API | source status ignored | exact exports or labelled local interface | +| Same guide repeated in many skills | progressive disclosure/ownership failure | one canonical reference, targeted routing | +| Evals accept one keyword | grader shortcut | multi-part decision/behavior assertions | +| Prompt variants differ only by suffix | duplicate smoke cases | distinct boundary/failure scenarios | +| Held-out case was visible to optimizer | data leakage | immutable split/export gate | +| Agent reads every reference | routing/selectivity failure | decision-specific required references | +| Blocked command reported passed | evidence status inflation | pass/fail/blocked/not-run report | +| Markdown diff is mostly wrapping | formatter violated reviewability | revert whitespace churn, scope formatting | + +## Counterexamples that must remain in the model + +### Standalone can be the correct topology + +After workspace, canonical docs, export/dependency, integration, and organization +searches, a package may have no meaningful siblings. Record standalone. The +ecosystem hypothesis prevents premature stopping; it does not require a positive +ecosystem claim. + +### Brand membership does not imply composition + +UnJS packages can compose, but selecting c12 does not require Unbuild, unstorage, +or Citty. Citty can be an alternative to Optique even if other selected UnJS +tools use it internally. + +### Minimal interoperability does not imply optional metadata + +Standard Schema conformance can expose validation while completion choices, +JSON Schema, transformations, codecs, and errors remain implementation-specific. + +### A guide is not implementation evidence + +The detailed production CLI guidebook defines a normative architecture. A +retained CLI codebase may implement only part or contradict it. Inspect code and +tests before claiming compliance. + +### Archive names do not prove change + +The retained old/new finance archives normalize to identical contents. Describe +one evidence body with duplicate provenance, not a migration history. + +### Documentation can outrun exports + +The retained Wikitext README/export source contains a contract discrepancy. +Bound claims to the inspected revision and do not fabricate the missing export. + +### Types are not dialect or durability + +A Drizzle-shaped query builder is not a correct ClickHouse dialect. Effect +services/Layers are not automatically a durable workflow history. Verify the +specialized runtime contract. + +### Tooling is not collected evidence + +The Wikitext archive contains stress runners for scenarios that were not all +collected. Report implemented tooling and collected artifacts separately. + +## Recovery protocol + +When a likely hallucination or contradiction is found: + +1. stop propagation: remove the claim from recommendations/code until resolved; +2. state exact claim, version, and scope; +3. mark current status `inferred` or `unresolved`; +4. identify at least two plausible explanations; +5. inspect exact lock/artifact/export/source/test/official relationship evidence; +6. run the smallest behavior check that discriminates explanations; +7. update topology, capability owner, exclusion, and integration plan; +8. add a regression eval that requires decision behavior, not the package word; +9. report remaining uncertainty and next evidence; do not fill it with analogy. + +If code was already written against an invented API, preserve user changes, +inspect the diff, replace it with an exact-version API or local interface plus +explicit unimplemented adapter, and run connected checks. Do not hide the error +behind `any`, casts, broad catches, or fake mocks. + +If wrong data/schema/public release escaped, follow the system's recovery plan: +stop writes/publishing, preserve evidence, reconcile/forward-fix/deprecate, and +communicate exact affected versions. Do not overwrite immutable history. + +## Verification + +- Validate every selected package and API against exact installed/published + exports/source. +- Check every official/sibling/adapter edge against canonical evidence. +- Search for stable/prerelease/private boundaries in references and examples. +- Run negative tests: missing optional peer, wrong host, unavailable dependency, + invalid config, cancellation, disposal, and rollback. +- Ensure evals require ownership, version/status, failure/exclusion, and + executable proof—not one keyword. +- Verify optimizer training and held-out splits are isolated. +- Review Markdown diffs without formatting unrelated text. +- Report blocked external-model/service/platform checks honestly. + +## Sources and freshness + +- All retained uploaded artifacts and their source registry, including duplicate + archive, normative-guide/implementation, Wikitext export, generated artifact, + private adapter, and cross-host counterexamples; verified 2026-07-17. +- Exact-version current primary source records for the ecosystems covered by the + deep skill suite; verified 2026-07-17. +- Existing repository evaluation schema and SkillOpt isolation contract, + observed at the working revision on 2026-07-17. + +Update signatures when new real failures appear. Do not add speculative failure +facts without exact source or executable evidence. diff --git a/skills/explore-ecosystems/references/integration.md b/skills/explore-ecosystems/references/integration.md index a5e3362..87fa660 100644 --- a/skills/explore-ecosystems/references/integration.md +++ b/skills/explore-ecosystems/references/integration.md @@ -1,54 +1,292 @@ -# Integration procedure +# Ecosystem integration procedure -## Before editing +## Contents -- locate the nearest owning manifest and lockfile; -- classify the repository as native, package-manager-first, or hybrid where the - runtime makes that distinction relevant; -- inspect existing imports, wrappers, utilities, adapters, and conventions; -- identify generated files, publish metadata, and deployment consumers; -- establish the exact target versions and peer requirements; -- capture the current verification commands and a failing or missing behavior. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Preflight](#preflight) +- [Integration contract](#integration-contract) +- [Implementation sequence](#implementation-sequence) +- [Configuration and provenance](#configuration-and-provenance) +- [Lifecycle, errors, and observability](#lifecycle-errors-and-observability) +- [Data, generated artifacts, and packaging](#data-generated-artifacts-and-packaging) +- [Verification matrix](#verification-matrix) +- [Migration, rollback, and removal](#migration-rollback-and-removal) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Sources and freshness](#sources-and-freshness) -## Integrate the ecosystem, not just the import +## When to load this reference -Trace the capability through its complete path: +Load this reference once component selection is evidence-backed and the task +authorizes implementation or a detailed implementation plan. It applies to +package adoption, replacement, adapter/plugin addition, framework integration, +service/database connection, or ecosystem migration. + +## Outcome + +Integrate a capability through every connected surface while preserving one +authority, explicit lifecycle, observable failures, package/deployment validity, +and rollback. A working import is the first checkpoint, not completion. + +## Preflight + +Before edits: + +- confirm requested mutation scope and prohibited external actions; +- capture Git status and preserve user changes; +- locate nearest owning manifest, workspace root, package-manager/lock authority, + runtime config, and repository instructions; +- record exact selected package/adapter versions, peers, integrity, and status; +- inspect existing imports/wrappers/adapters/config/schema/tests/generators; +- identify public API/config/data/deployment/package consumers; +- capture current passing/failing behavior and verification commands; +- write capability ownership before/after and rollback point. + +Do not change lock/package manager, format Markdown, regenerate unrelated files, +or upgrade sibling packages simply because the integration touches a manifest. +If the selected exact versions cannot coexist, stop and revisit selection. + +## Integration contract + +Trace the full path: ```text -manifest and version - -> public import or adapter - -> configuration and schema - -> runtime owner - -> observable behavior - -> tests and generated surfaces - -> build, package, and deployment +manifest + lock + integrity + -> public import/export and local adapter + -> authored config/source layers + -> sparse merge/provenance + -> schema validation/defaults + -> runtime lifecycle owner + -> domain/service call + -> success/error/cancel/retry/recovery + -> LogTape/metrics/traces or other observability owner + -> generated/data/package/build/deploy surfaces + -> clean consumer and operational verification ``` -Update only the packages and connected surfaces the chosen capability requires. -Preserve framework and package-manager metadata that remains authoritative. +Write a local boundary when it owns project policy or isolates volatility: + +```ts +export interface ArtifactStore { + put(key: string, bytes: Uint8Array, signal: AbortSignal): Promise<void>; + get(key: string, signal: AbortSignal): Promise<Uint8Array | undefined>; + close(): Promise<void>; +} +``` + +The interface is not proof of any package API. Implement it from exact source +and translate upstream errors/status/lifecycle deliberately. Avoid speculative +methods for private or experimental adapters. + +Define boundary ownership: + +- schema validates external/config/data shapes; +- adapter translates library/host mechanics; +- service/domain owns business policy; +- composition root creates/configures resources; +- lifecycle coordinator disposes/flushes; +- executable/deployment maps outcomes to host behavior. + +## Implementation sequence + +### 1. Add exact dependencies at the owning package + +Use repository package-manager policy and frozen lock semantics. Classify runtime, +peer, optional, development, platform/native, and build-only dependencies. Do not +add a dependency to the root when only one workspace package imports it. Review +install scripts and package contents before execution when material. + +### 2. Establish imports and local boundary + +Import only public subpaths verified for the selected version. Prefer a narrow +adapter over upstream types throughout domain code when configuration, error, +lifecycle, or experimental status needs isolation. Reexport only intentional +public contracts. + +### 3. Implement one vertical success slice + +Connect config -> validation -> resource -> service -> output for the smallest +representative workflow. Use real observable behavior, not only mocks. This +exposes missing siblings/adapters before broad migration. + +### 4. Implement failure/lifecycle slice + +Add invalid config, unavailable dependency, timeout, cancellation, retry limit, +partial result, cleanup, and shutdown. Translate errors without discarding cause, +code, retryability, or provenance. Await asynchronous sink/client/worker disposal. + +### 5. Update all connected surfaces + +Types, tests, generated files, tasks, permissions, manifests, build/export maps, +docs examples, completion/config inspection, database migrations, deploy config, +health/readiness, observability, package contents, and release checks. + +### 6. Remove duplicate owner only after cutover + +Keep compatibility paths only for a bounded migration. Prove no consumers remain +before removing dependencies/config/tasks/data. Update lockfile and generated +artifacts through authoritative commands. + +## Configuration and provenance + +Keep source adapters sparse: CLI/env/config layers return only explicitly present +values. Merge using a documented order and field policy, then validate/apply +defaults exactly once. Track origin for diagnostics and `config show` behavior. + +Example ownership: + +```text +CLI patch > environment patch > project config > user config > defaults +``` + +This is an example; c12's exact source/layer order and custom merger orientation +must come from the selected version. Arrays often require explicit replace, +append, and prepend authoring objects rather than defu's generic concatenation. +No `$append`/`$replace` operation should reach runtime services. + +For plugins/adapters: + +- configuration belongs to the application owner, not process-global import + side effects; +- secret values come from named secret providers/environment and are redacted; +- optional features have explicit enablement, capability check, and fallback; +- watchers/reload define atomicity, validation, previous-good-state, and disposal; +- effective config/provenance is inspectable without exposing secrets. + +Do not add two configuration packages for the same concern merely because an +ecosystem offers its own config helper. + +## Lifecycle, errors, and observability + +Construct long-lived clients/sinks/workers in the composition root. Share or +scope according to upstream contract. Define startup ordering, readiness, +shutdown deadline, cancellation propagation, flush/dispose, and behavior after +partial startup. + +Error table: + +| Upstream condition | Local contract | Retry/recovery | Observable fields | +|---|---|---|---| +| invalid config/input | typed validation error before resource use | no retry | path/source/issues | +| auth/permission | typed authorization/dependency error | refresh/escalate by policy | target/status/correlation, redacted | +| transient network/service | typed unavailable/timeout | bounded idempotent retry | attempt/deadline/target | +| cancellation | cancelled outcome preserving reason | cleanup only | operation/duration | +| partial publish/write | partial-state result | reconcile/resume/compensate | completed/pending IDs | +| incompatible version/protocol | startup/preflight failure | upgrade/downgrade | exact versions | + +Use one observability transport owner. Bridges/adapters should map upstream logs +into categories/properties without configuring a second global logger. Stable CLI +results remain isolated from diagnostics. Instrument retries, queue depth, +backpressure, recovery, and disposal where operationally meaningful. Do not log +credentials, cookies, tokens, raw personal data, or full config. + +## Data, generated artifacts, and packaging + +### Data and services + +Define system of record, schema/migration owner, transaction boundary, idempotency, +delivery, ordering, cursor/checkpoint, retention, backup/restore, reconciliation, +and destructive-operation authorization. An analytics projection is not a +transactional backup. A workflow service does not make arbitrary side effects +durable unless activities/idempotency/recovery are designed. + +Test fresh schema, upgrade, rollback/forward-fix, concurrent operations, failure +after each durable boundary, and recovery from persisted state. + +### Generated code/config/docs + +Record source/generator/output ownership, exact version, check/write command, +provenance, and consumer test. Integrating Automd/Magicast/Giget or another +generator must not format unrelated Markdown or execute untrusted config. +Regenerate only owned outputs and inspect semantic diffs. + +### Build/package/deployment + +Update export maps, declarations, externals/peers, assets, permissions, engine +requirements, native/platform packages, container images, server adapters, +environment/secret declarations, health/readiness, and release manifests. Verify +from clean packed artifacts and actual target host. Workspace resolution can hide +missing dependencies and files. + +## Verification matrix + +| Level | Required proof | +|---|---| +| Static | exact imports, types, schemas, manifests, lock, exports, generated metadata | +| Unit/contract | adapter translations, config merge, errors, lifecycle, invariants | +| Targeted integration | selected exact components in a real fixture | +| Connected workflow | actual command/request/job/data path end to end | +| Consumer | clean pack/install/import/build/execute outside workspace | +| Operational | unavailable/timeout/cancel/retry/shutdown/recovery/rollback | +| Compatibility | supported runtime/renderer/driver/platform/version matrix | +| Security | permissions, trust, redaction, secret and destructive boundaries | + +Use representative assertions, not package keywords. Verify output/protocol/data +semantics. Record commands/results and distinguish passed, failed, blocked, and +not run. A registry delay, unavailable service credential, or absent target +platform is blocked evidence, not success. + +Composition tests matter. When multiple skills/tools activate, assert one +repository discovery, one plan, one lifecycle/report owner, targeted reference +loading, and specialist verification. When multiple libraries compose, test the +seam, not only each in isolation. + +## Migration, rollback, and removal + +Define before cutover: + +- compatibility window and public/data/config translation; +- primary authority during coexistence; +- backfill/dual-read/dual-write policy and reconciliation; +- feature flag/channel and stop condition; +- rollback revision/artifact/config/data and time limit; +- irreversible steps and required backup/authorization; +- removal inventory across code, deps, lock, config, secrets, tasks, docs, CI, + data, deploy, alerts, and packages. + +Rollback can mean restoring the previous application while leaving a forward- +compatible schema, not reversing every migration. Test the declared action. For +published packages, recovery normally means deprecate/supersede rather than +overwriting an immutable version. -## Version and host matrix +## Failure signatures -Check the actual combinations the project claims: +| Signature | Likely integration gap | Next inspection | +|---|---|---| +| Import works, production fails | host/deploy/lifecycle not connected | exact target matrix | +| Config differs from expectation | layer order/array/default authority | sparse layers and provenance | +| Tests pass only in monorepo | undeclared dep/file/workspace alias | packed clean consumer | +| Logs duplicate or JSON corrupts | two transports/sink inheritance | observability/result ownership | +| Shutdown loses events/work | async dispose/flush not awaited | lifecycle coordinator | +| Adapter throws untyped library errors | translation boundary missing | upstream failure contract | +| Analytics diverges from OLTP | delivery/checkpoint/reconciliation absent | authority and repair workflow | +| Generated diff rewrites docs | generator/formatter scope too broad | owned markers and check mode | +| Old dependency remains after cutover | connected surfaces not inventoried | full removal search/lifecycle | +| Rollback plan cannot restore state | irreversible boundary discovered late | migration backup/forward-fix | -- runtime and operating system; -- package and adapter versions; -- framework renderer and server adapter; -- bundler, compiler, or packaging target; -- database dialect and driver; -- development, test, CI, and production hosts. +## Deliberate exclusions -Type compatibility is not runtime compatibility. API resemblance is not dialect -support. A package existing in the ecosystem is not proof that its adapter works -in the selected host. +- Do not stop after installing/importing. +- Do not bypass existing wrappers/config owners without migration evidence. +- Do not call types or mocks an integration test. +- Do not run broad Markdown formatting or unrelated generation. +- Do not use duplicate log/config/data/migration/workflow owners indefinitely. +- Do not infer a private/experimental adapter API from a similar library. +- Do not mutate production, publish, rotate secrets, or destroy data without + explicit authorization. +- Do not remove the old path before connected consumers and rollback are proven. -## Verification levels +## Sources and freshness -1. Static: imports, types, schemas, manifests, and generated metadata agree. -2. Targeted: the changed package and adapter tests pass. -3. Integration: the real connected workflow executes. -4. Consumer: a clean project can install, import, build, or connect. -5. Operational: failure, cleanup, rollback, and observability behave as claimed. +- Attached production CLI guidebook v1.1 and config-resolution handoff, normative + portable boundary, sparse source, provenance, lifecycle, output, package, and + verification patterns; verified 2026-07-13. +- Retained uploaded CLI, finance, site, data, workflow, Better Auth, Undent and + Wikitext codebases, observed connected integration/failure/generator/release + behavior; verified 2026-07-17. +- Current exact-version primary source records for deep ecosystem references; + verified 2026-07-17. -Report which levels ran. Never convert a registry delay or missing credential -into a passing result. +Recheck installed packages and target deployment/runtime before implementation. +Examples encode ownership patterns, not universal package APIs. diff --git a/skills/explore-ecosystems/references/method.md b/skills/explore-ecosystems/references/method.md index a154619..db8ce7b 100644 --- a/skills/explore-ecosystems/references/method.md +++ b/skills/explore-ecosystems/references/method.md @@ -1,53 +1,329 @@ -# Ecosystem investigation worksheet +# Ecosystem investigation method -Use this worksheet for a material dependency decision. Omit empty sections only -when they cannot affect the decision. +## Contents -## Decision +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [The ecosystem hypothesis](#the-ecosystem-hypothesis) +- [Plan at three levels](#plan-at-three-levels) +- [Phase 1: frame the decision](#phase-1-frame-the-decision) +- [Phase 2: establish identity and installed truth](#phase-2-establish-identity-and-installed-truth) +- [Phase 3: map topology and capabilities](#phase-3-map-topology-and-capabilities) +- [Phase 4: inspect behavior and operations](#phase-4-inspect-behavior-and-operations) +- [Phase 5: select and prove](#phase-5-select-and-prove) +- [Investigation worksheet](#investigation-worksheet) +- [Stopping rules](#stopping-rules) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Sources and freshness](#sources-and-freshness) -- Task and capability needed: -- Existing owner in the repository: -- Candidate under investigation: -- Decision deadline and acceptable uncertainty: +## When to load this reference -## Identity +Load this reference whenever a package, framework, service, tool, database, +protocol, or remembered project materially affects architecture or implementation. +Material means it can change public contracts, capability ownership, security, +data, deployment, build/package output, runtime support, operations, or migration. +Do not run this full investigation for an incidental leaf dependency that the +task does not touch. -- Exact package, import, product, or protocol: -- Canonical repository and owner: -- Installed and latest relevant versions: -- Runtime, host, license, and maturity: -- Identity confidence and unresolved ambiguity: +## Outcome -## Topology +Reach a decision-complete, claim-level evidence record. Another engineer should +be able to answer: -- Classification: -- Workspace packages: -- First-party sibling repositories: -- Official adapters and plugins: -- Community or experimental integrations: -- Specifications and adjacent systems: +- what exact thing was investigated at what version/revision; +- whether it is standalone, a monorepo, a multi-repository ecosystem, a + specification ecosystem, or unresolved; +- which sibling packages, adapters, plugins, presets, templates, integrations, + and alternatives were considered; +- which capability each selected component owns and what remains excluded; +- how installed truth differs from current documentation/releases; +- configuration, runtime, generated, security, data, packaging, deployment, + failure, migration, and rollback consequences; +- which statements were observed, documented, inferred, or unresolved; +- what executable verification ran and what remains blocked. -## Capability ownership +The output is not a package catalog. It is a capability and boundary decision. -| Capability | Current owner | Candidate owner | Decision | Evidence | -|---|---|---|---|---| +## The ecosystem hypothesis -Record explicit exclusions and why they remain excluded. Organization membership -alone is not an inclusion reason. +Start every material dependency as if the named package may be only one visible +node in a larger system. Search for: -## Compatibility and operations +- workspace siblings split by core, runtime, framework, adapter, driver, plugin, + testing, configuration, UI, build, docs, or deployment concern; +- first-party sibling repositories and official integration indexes; +- protocol/specification peers that interoperate across owners; +- examples/starters/templates that reveal intended composition; +- optional and peer dependencies that expose adapters; +- related personal/private packages visible only in consuming repositories. -- Supported runtimes and version boundaries: -- Configuration and generated artifacts: -- Security and secret boundaries: -- Failure modes and recovery: -- Build, bundle, publish, and deployment implications: -- Migration and rollback: +This is a mandatory investigation hypothesis, not permission to assert that +every dependency is literally a monorepo or that every sibling belongs in the +solution. After searching, classify the actual topology and record exclusions. -## Verification +Examples: -- Source/tests inspected: -- Minimal workflow run: -- Clean-consumer or integration check: -- Claims still inferred rather than observed: -- Freshness date and primary sources: +- LogTape is a package family: core transport, pretty/file/redaction/testing and + integrations have distinct roles. Selecting only the core may miss required + result routing or tests; installing every adapter creates duplicate ownership. +- Optique separates parser core, runners/discovery, config/env/default sources, + schema integrations, completion, man output, and LogTape integration. A task + concerning completions cannot be decided from one parser package README. +- UnJS is intentionally multi-repository. c12, defu, jiti, rc9, std-env, ofetch, + unstorage, pkg-types, nypm, unbuild, changelogen, Automd, Giget, Magicast, and + others can compose, but brand membership is not an inclusion reason. +- Standard Schema is a specification ecosystem: implementations may interoperate + through a minimal contract while optional metadata remains library-specific. +- A standalone package may have no meaningful ecosystem. That is a valid result + after discovery, not a reason to invent relationships. + +## Plan at three levels + +Keep three synchronized plans so breadth does not erase the task. + +### Decision plan + +State the user/repository decision, deadline, reversibility, required confidence, +and acceptance criteria. Example: "Choose the configuration owner for one CLI; +preserve current precedence and provenance; no publication in this task." + +### Capability plan + +List capabilities and connected surfaces before naming packages: + +```text +config discovery -> layer provenance -> merge algebra -> schema validation + -> runtime object -> config inspection -> watcher/reload -> package/deploy +``` + +Assign current owner, candidate owner, risk, evidence needed, and verification. +This reveals missing siblings without creating a shopping list. + +### Evidence plan + +For each decision-changing claim identify the preferred source and fallback: + +```text +installed export -> lock/source/tests at revision -> published artifact + -> versioned official docs/release notes -> issue/discussion -> unresolved +``` + +Keep independent unknowns visible. Do not wait to write one monolithic summary; +update the claim ledger as evidence arrives. + +## Phase 1: frame the decision + +Write: + +- requested outcome and authorization mode (research, recommend, implement); +- repository/runtime/package-manager/workspace context; +- current owners and observed problem/failure; +- hard requirements and non-goals; +- security/data/public compatibility constraints; +- target versions/hosts or the evidence needed to choose them; +- required verification and rollback. + +Search the repository first. Existing imports, wrappers, task names, lockfiles, +patches, generated files, tests, adapter modules, comments, deployment code, and +local guides are stronger context than an internet recommendation. Do not replace +a mature wrapper because an upstream package has a similar API; establish which +policy the wrapper owns. + +Define materiality. Investigate direct owners, sibling adapters needed by the +capability path, and alternatives that could replace those owners. Stop chasing +transitive utility packages unless they change the decision, risk, or failure. + +## Phase 2: establish identity and installed truth + +Names are ambiguous. Resolve: + +| Field | Evidence | +|---|---| +| Import/package name | source import plus owning manifest/lock entry | +| Registry and version | lockfile/resolution metadata, not range alone | +| Repository/owner | package metadata and canonical project links | +| Workspace package path | root workspace declarations and package manifest | +| Artifact integrity | registry integrity/digest or revision | +| Runtime/host/platform | manifest engines/peers plus source/tests | +| License/maturity | packaged license, releases/status, not badge alone | + +Inspect exact installed exports and source before current docs. Then inspect +current stable/prerelease lines and release notes to learn migration/deprecation. +Write the boundary: "repository resolves c12 3.3.4; source record also examined +4.0.0-beta.5 for future capability; beta APIs are not available to current +code." Never merge versions into a fictional superset API. + +For a private, personal, forked, patched, vendored, generated, or unpublished +package, record the source path/revision and lower confidence. User memory is a +discovery lead, not an import API. A repository named after a product may not +contain every implied package; search manifests/entrypoints. + +## Phase 3: map topology and capabilities + +Use [topology.md](topology.md) for the discovery algorithm. Produce a bounded +node/edge ledger: + +| Node | Relationship | Capability | Version/status | Evidence | Decision | +|---|---|---|---|---|---| +| core | workspace sibling | command grammar | stable installed | exports/tests | include | +| log adapter | official integration | verbosity mapping | stable compatible | official index | include | +| React binding | official adapter | React renderer | stable | peer deps | exclude: Solid host | +| experimental store | community | persistence | prerelease | repo only | exclude | + +Edges need semantics: depends on, adapts, implements spec, alternative to, +generates, publishes, consumes, or merely related. Organization membership is +not a semantic edge. + +Build a capability owner table. Split packages only where their concerns differ. +For a CLI, Optique can own command grammar while LogTape owns output transport, +c12 owns discovery/layers, defu/custom merge owns fallback algebra, Zod owns the +application schema, and Standard Schema owns an interoperability boundary. If +two tools own the same concern, select one or write an explicit composition rule. + +## Phase 4: inspect behavior and operations + +For each selected candidate inspect the complete lifecycle: + +1. public exports/types and minimal usage; +2. configuration shapes, defaults, precedence, environment, and provenance; +3. runtime behavior, concurrency, lifecycle/disposal, errors, retries, timeout, + cancellation, and recovery; +4. framework/runtime/driver adapters and exact version/peer boundaries; +5. security/trust/secrets/permissions and supply-chain concerns; +6. persistence/schema/migration/data authority where applicable; +7. build/bundle/tree-shaking/generated/package/publication behavior; +8. tests/examples that cover real integrations and failure cases; +9. upgrade/downgrade/rollback and deprecated/experimental surfaces; +10. cost: dependencies, binaries, services, latency, memory, operations, context. + +Read implementation and tests where documentation is incomplete or a claim is +fragile. Do not generalize an example beyond its host: a React adapter is not a +Solid adapter, a PostgreSQL dialect is not ClickHouse, Node compatibility is not +edge support, and matching TypeScript shapes are not behavioral interoperability. + +Create a failure table before integration. If you cannot say what an adapter +does when unavailable, misconfigured, incompatible, cancelled, or partially +successful, research is not decision-complete. + +## Phase 5: select and prove + +Use [selection.md](selection.md) to assign owners and [integration.md](integration.md) +to implement/verify. A recommendation must include: + +- chosen version/components and exact roles; +- rejected plausible siblings/alternatives and reasons; +- config/runtime/generated/deploy migration sequence; +- compatibility and rollback boundary; +- minimal proof and real connected workflow; +- unresolved claims and evidence needed next. + +Prefer a small spike in an isolated fixture when documentation cannot establish +behavior. Install exact versions, assert types and observable behavior, test one +failure, remove the fixture afterward or retain it as an eval. Do not turn a +research spike into an unreviewed production dependency. + +## Investigation worksheet + +```text +Decision + task/outcome: + current owner/problem: + constraints/non-goals: + authorization/reversibility: + +Identity + exact package/import/product/protocol: + installed version/revision/integrity: + canonical owner/repository/license/status: + current stable/prerelease boundary: + +Topology + actual topology classification: + workspace siblings: + repository siblings: + official adapters/plugins/presets: + specification peers: + community/experimental/alternatives: + +Capability ownership + required capability -> current owner -> candidate -> decision -> evidence: + duplicate-owner resolution: + exclusions: + +Operations + config/defaults/provenance: + runtime/lifecycle/errors/recovery: + security/secrets/permissions: + data/migrations: + build/package/deploy: + upgrade/rollback: + +Proof + exact sources and status: + executable checks: + connected workflow: + unresolved/blocked: + freshness date: +``` + +## Stopping rules + +Stop when all decision-changing capabilities have one owner, every selected +relationship has evidence, material compatibility/failure/operational risks have +a test or explicit unknown, plausible alternatives/exclusions are recorded, and +the verification plan can accept or reject the integration. + +Also stop and report uncertainty when: + +- identity or source cannot be established; +- installed artifact/source is unavailable; +- only mutable/current docs exist for an older installed version; +- an adapter is private/experimental and lacks observable contract tests; +- credentials/hardware/production authority are required; +- further siblings are incidental and cannot change the decision. + +Do not stop merely after finding a familiar package, a working import, an official +example, or a long list. Do not continue until every organization repository has +been read. + +## Failure signatures + +| Signature | Method defect | Correction | +|---|---|---| +| Only the named package appears | ecosystem hypothesis skipped | inspect workspace/org/integration/export surfaces | +| Dozens of siblings recommended | topology confused with inclusion | capability owner and exclusion test | +| API combines old stable and beta | installed/current truth collapsed | versioned claim ledger | +| Familiar package replaces local wrapper | repository ownership ignored | inspect wrapper policy/tests/consumers | +| Research never ends | no materiality or stopping rule | restate decision and evidence gaps | +| Example copied but host differs | integration context generalized | version/renderer/runtime matrix | +| "Official" community adapter | relationship status unverified | canonical integration/maintainer evidence | +| Types accepted, behavior failed | structural typing treated as contract | executable connected workflow | +| Private API invented from memory | discovery lead treated as source | locate code/export or mark unresolved | +| Long report has no decision | information not mapped to capability | owner table, selection, proof | + +## Deliberate exclusions + +- Do not research every dependency in the lockfile. +- Do not assume monorepo literal topology or manufacture an ecosystem. +- Do not install every sibling, adapter, or UnJS package discovered. +- Do not equate same organization, maintainer, naming, or API resemblance with + compatibility or official support. +- Do not copy current documentation into an older installed version. +- Do not call an inferred/private/experimental surface stable. +- Do not mutate production or publish merely to test an ecosystem decision. + +## Sources and freshness + +- Attached production CLI guidebook v1.1 and CLI audit, normative examples of + capability-owner composition across Optique, LogTape, c12, defu, schema, + prompts, and selective UnJS adapters; verified 2026-07-13. +- Retained uploaded codebases, observed monorepo, multi-package, duplicate + archive, private/personal, generated, adapter, and stale-doc counterexamples; + source registry verified 2026-07-17. +- Current pinned source records for Optique, LogTape, c12/defu/jiti, UnJS, + Effect/Temporal, ClickHouse/Drizzle, Astro icons/fonts, Better Auth, Solid + Primitives, and Okikio packages; verified 2026-07-17. + +Use those records as examples of claim discipline. Recheck mutable official docs, +installed versions, and registry artifacts at the time of a new decision. diff --git a/skills/explore-ecosystems/references/selection.md b/skills/explore-ecosystems/references/selection.md index 0db5d15..f7c23dc 100644 --- a/skills/explore-ecosystems/references/selection.md +++ b/skills/explore-ecosystems/references/selection.md @@ -1,48 +1,293 @@ -# Selection and capability ownership +# Capability ownership and ecosystem selection -## Start from the capability graph +## Contents -List the capabilities the task actually needs, then map each to its current and -candidate owner. Typical owners include command grammar, configuration loading, -schema validation, logging transport, rendering, routing, server state, -authentication, authorization, persistence, migrations, and durable execution. +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Select capabilities before packages](#select-capabilities-before-packages) +- [Owner, companion, adapter, and alternative](#owner-companion-adapter-and-alternative) +- [Decision model](#decision-model) +- [Version, maturity, and portability](#version-maturity-and-portability) +- [Operational and supply-chain cost](#operational-and-supply-chain-cost) +- [Migration and reversibility](#migration-and-reversibility) +- [Worked ecosystem selections](#worked-ecosystem-selections) +- [Exclusion records](#exclusion-records) +- [Failure signatures](#failure-signatures) +- [Deliberate exclusions](#deliberate-exclusions) +- [Verification](#verification) +- [Sources and freshness](#sources-and-freshness) -Prefer one canonical owner per concern. Two tools may coexist only when the -boundary is explicit, for example: +## When to load this reference -- Zod owns application schemas while Standard Schema is an interoperability - boundary; -- LogTape owns process results and diagnostics while a stage writer owns durable - JSONL artifacts; -- PostgreSQL owns transactions while ClickHouse owns analytical projections; -- Astro owns page rendering while Solid owns an interactive island. +Load this reference after identity/topology evidence exists and before adding, +removing, replacing, or recommending ecosystem components. It is required when +several sibling packages look useful, two tools overlap, or the implementation +already has an owner for the concern. -## Inclusion test +## Outcome -Include a package when all are true: +Choose the smallest coherent set of exact components that covers required +capabilities with one canonical owner per concern, explicit adapter boundaries, +understood operations, and executable proof. Record plausible exclusions so +future agents do not rediscover and install them reflexively. -1. a required capability has no adequate existing owner; -2. the package's identity and relationship are verified; -3. its version and runtime fit the repository; -4. its operational cost is understood; -5. its integration can be verified; -6. it does not create an unexplained second source of truth. +## Select capabilities before packages -## Exclusion record +Start with observable needs: -Record plausible siblings and alternatives that were considered but excluded. -Common valid reasons include duplicate ownership, wrong runtime or renderer, -experimental status, unsupported dialect, redundant configuration, excess -deployment cost, or no current use case. +```text +Need: typed command grammar, generated help/completion/man +Need: stable stdout results and structured diagnostics +Need: authored/project/user config with provenance and array policy +Need: package build, registry artifact, clean consumer +``` -Do not install every sibling to demonstrate ecosystem awareness. The goal is a -coherent system, not maximal package count. +Then map ownership: -## Alternative versus companion +| Capability | Current owner | Candidate | Gap/overlap | Decision evidence | +|---|---|---|---|---| +| command grammar | hand parser | Optique | replacement | grammar/help/completion tests | +| output transport | LogTape | another logger | duplicate | keep LogTape | +| config discovery/layers | c12 | c12 | none | retain | +| config merge algebra | generic defu | custom c12 merger | partial | explicit arrays/provenance tests | +| package build | dnt | unbuild | target-dependent | Deno-to-Node output contract | -Distinguish these deliberately. Consola and LogTape may be alternative logging -owners; Citty and Optique may be alternative command parsers. An official -Optique-to-LogTape adapter is a companion because it connects different owners. +Do not let package availability redefine the task. A sibling may have excellent +capabilities that are unnecessary here. -If two candidates overlap, compare behavior, maturity, ecosystem fit, migration -cost, failure modes, and existing repository ownership before selecting one. +## Owner, companion, adapter, and alternative + +Use these roles consistently: + +- **Owner:** canonical source of behavior/policy for one concern. +- **Companion:** owns a different required concern and composes at a documented + boundary. +- **Adapter:** translates between owners/hosts without becoming an independent + policy source. +- **Alternative:** could own the same concern instead; normally choose one. +- **Optional extension:** adds a non-required feature behind an explicit enable, + capability check, lifecycle, and fallback. +- **Source/generator:** produces another selected node; needs provenance. +- **Incidental dependency:** implementation detail, not application architecture. + +Examples: + +- Optique and LogTape are companions; `@optique/logtape` is their adapter. +- Optique and Citty are command-grammar alternatives for most applications. +- Zod can own application schemas while Standard Schema is an adapter/spec + boundary. Standard Schema does not replace optional Zod metadata automatically. +- c12 owns layer loading; defu/custom merger implements merge mechanics. Neither + should apply application defaults a second time after final validation. +- PostgreSQL and ClickHouse can be transactional owner plus analytical projection, + provided delivery/reconciliation is explicit. They are not interchangeable. +- Astro and Solid can be page-rendering owner plus interactive-island owner; + renderer-specific packages remain in their host. + +If two selected nodes both configure/log/cache/migrate/persist the same concern, +write an ordering/authority rule or remove one. Bridges should not create a +third configuration source. + +## Decision model + +Score only after hard constraints pass. A weighted score cannot compensate for +wrong runtime, incompatible license, absent required capability, or unverified +identity. + +### Hard gates + +1. Exact identity/source and selected version are established. +2. Required capability is actually present in that version/artifact. +3. Runtime, framework/renderer, driver/dialect, platform, and license fit. +4. Security/data/deployment constraints can be met. +5. The component does not create an unresolved second authority. +6. Integration/failure/rollback can be verified. + +### Comparative dimensions + +| Dimension | Questions | +|---|---| +| Capability fit | Does it cover the full required behavior and failures? | +| Existing fit | Does it compose with current owners/wrappers/conventions? | +| Maturity | Stable, prerelease, experimental; release/maintenance evidence? | +| Portability | Runtime/platform/renderer/bundler/service coupling? | +| Operational cost | service, storage, binaries, credentials, observability, recovery? | +| Supply chain | dependency graph, install scripts, integrity, update owner? | +| Migration | data/config/API/public compatibility and rollback? | +| Verification | Can real behavior be tested locally/CI/staging? | +| Context cost | Will instructions/reference loading be proportionate? | + +Write decisive evidence beside each score. Avoid false precision such as 8.3/10 +when unknown behavior dominates. + +### Inclusion rule + +Include when a required capability has no adequate owner, identity and +relationship are verified, compatibility/operations are acceptable, integration +is testable, and ownership stays coherent. Retain an existing adequate owner +unless replacement has material benefit that exceeds migration/risk. + +## Version, maturity, and portability + +Select an exact version line, not a timeless package name. Record installed, +target, latest stable, and relevant prerelease separately. Do not use beta docs +for stable code. + +For prerelease/experimental packages: + +- pin exact version/integrity; +- isolate behind a local interface/adapter; +- avoid making its data format or API an irreversible public contract; +- add export and behavior fixtures that fail on upgrade; +- define removal/fallback and upgrade owner; +- report status near examples. + +Portability claims require host matrices. A library that uses Web APIs may still +depend on Node package resolution or filesystem. A typed ORM adapter may still +emit unsupported SQL. A framework plugin may support Vite but not the selected +SSR adapter. Test exact combinations. + +## Operational and supply-chain cost + +Selection includes lifecycle, not only code ergonomics: + +- additional service/account/region/quota/cost and availability; +- schema/migration/backup/restore/reconciliation; +- secrets, permissions, network hosts, filesystem and subprocesses; +- binary/native install and supported targets; +- logging/metrics/tracing/diagnostics and redaction; +- retries/timeouts/cancellation/backpressure/disposal; +- update/deprecation/security response and maintainer capacity; +- build/package size, startup/memory/performance; +- license/notices and transitive/lifecycle-script exposure. + +Prefer existing ecosystem components when they reduce duplicated policy, but +do not add a dependency for a trivial stable operation. A new abstraction that +only renames an upstream API increases ownership without isolation value. + +## Migration and reversibility + +Separate adoption phases: + +1. prove in isolated fixture/spike; +2. introduce local boundary/adapter and compatibility tests; +3. dual-read/compare only when data migration requires it and authority is clear; +4. move one capability owner at a time; +5. verify connected consumers/deployment; +6. remove compatibility path after proven cutover; +7. retain rollback data/config/artifacts for the declared window. + +Avoid dual-write authority by default. If both systems must run, name the primary, +replication direction, idempotency, failure/reconciliation, and cutover condition. +For package-manager or lockfile trials, keep one authoritative lock until a +separate cutover. + +## Worked ecosystem selections + +### Production CLI + +Required: typed structural grammar, source-aware config, stable output transport, +schema validation, package build. + +Selection: + +- Optique packages needed for grammar/runner/help/completion/man and exact schema + integration; +- LogTape core plus only needed pretty/file/redaction/testing sinks/adapters; +- c12 for project/user/package/environment/extends layers; custom merger using + defu only where defaults semantics match, with explicit array operations; +- Zod as application schema; Standard Schema only at library interoperability; +- dnt for a Deno-source Node artifact when its transform matches, or unbuild for + a Node library build. Do not layer both over the same output without purpose. + +Exclusions: Citty as duplicate grammar owner; `@logtape/config` when c12/application +schema already owns config; every UnJS utility not on the capability path; remote +presets by default when installed versioned packages are safer. + +### Durable work + +Required: resumable orchestration, durable state, external side effects, recovery. + +Choose among local workflow runtime, `@effect/workflow` experimental line, and +Temporal based on durability authority and deployment. Effect services/Layers +provide typed capability composition but do not by themselves persist workflow +history. `@effect/workflow` is alpha/version-sensitive and must be pinned/isolated. +Temporal introduces a service, deterministic workflow constraints, workers, +activities, versioning, and operations. Do not select both durable engines as +co-equal owners. + +### Data and ORM + +Required: transactional state plus analytics. Keep PostgreSQL/Drizzle as OLTP +schema/query/migration owner and ClickHouse as projection. A custom Drizzle-like +ClickHouse adapter must be derived from actual driver/dialect/session/query +contracts and ClickHouse semantics; never infer compatibility from API shape. +Exclude a generic SQL adapter that claims no ClickHouse dialect support. + +## Exclusion records + +For every plausible sibling/alternative record: + +```text +Candidate: @logtape/config +Relationship: first-party package +Capability: logging configuration +Decision: exclude +Reason: application config is already owned by c12 + Zod; adding it would create + two config authorities. Reconsider only if LogTape-specific dynamic + configuration cannot be expressed through the composition root. +Evidence/version: official LogTape docs, selected package versions, repository config +``` + +Good reasons: duplicate ownership, no required capability, wrong host/dialect, +unsupported peer/version, experimental maturity, excessive operations/security +cost, license, absent failure/rollback, or existing owner is adequate. "Not +popular" or "did not appear in first search" is not sufficient. + +## Failure signatures + +| Signature | Selection error | Correction | +|---|---|---| +| Every sibling installed | topology mistaken for selection | required owner/exclusion table | +| Two configs/loggers/ORMs | alternatives stacked | one authority plus adapter/migration | +| Tool chosen for brand consistency | capability fit missing | hard gates and behavior proof | +| Beta API leaks into public contract | maturity/reversibility ignored | pin, isolate, fixture, fallback | +| Types fit but integration fails | portability reduced to types | exact host matrix and behavior | +| New wrapper adds no policy | abstraction without ownership value | use upstream or define real boundary | +| Migration needs indefinite dual write | cutover/authority absent | primary, reconciliation, stop condition | +| Alternative score looks precise but evidence missing | false numeric confidence | hard gates/unknowns first | +| Existing mature wrapper discarded | repository ownership ignored | compare policy and migration benefit | +| Selected package cannot be tested | verification gate skipped | spike or keep unresolved/exclude | + +## Deliberate exclusions + +- Do not maximize package count or ecosystem purity. +- Do not replace an adequate owner for novelty or a nicer isolated API. +- Do not stack alternatives without explicit bounded migration. +- Do not adopt prerelease/private surfaces without pinning, isolation, and exit. +- Do not let a generic score override hard compatibility/security/data gates. +- Do not select a service/library whose required workflow cannot be verified. +- Do not omit exclusion reasoning; absence invites future hallucinated inclusion. + +## Verification + +1. Validate identity/version/relationship claims against exact source/artifact. +2. Build capability ownership table and assert no unexplained duplicate owners. +3. Run a minimal exact-version spike for required success and failure behavior. +4. Test runtime/renderer/dialect/platform/package/build matrix. +5. Exercise security/lifecycle/disposal/recovery and package/deployment effects. +6. Compare retained baseline/existing owner against candidate behavior and cost. +7. Prove rollback/cutover on a fixture for data/config/public migrations. +8. Review exclusions with triggers for reconsideration. +9. Report blocked checks and experimental/inferred claims without promoting them. + +## Sources and freshness + +- Attached production CLI guidebook v1.1 and CLI audit, normative owner/adaptor + decision model and selective ecosystem package map; verified 2026-07-13. +- Current official/pinned Optique, LogTape, c12/defu/UnJS, Effect/Temporal, + ClickHouse/Drizzle and framework ecosystem records; verified 2026-07-17. +- Retained uploaded implementations, observed existing wrappers, service modules, + custom adapters, runtime constraints, and counterexamples; verified 2026-07-17. + +Re-evaluate selected versions, package status, service operations, and existing +repository ownership at implementation time. diff --git a/skills/explore-ecosystems/references/topology.md b/skills/explore-ecosystems/references/topology.md index 37d696d..979699f 100644 --- a/skills/explore-ecosystems/references/topology.md +++ b/skills/explore-ecosystems/references/topology.md @@ -1,48 +1,296 @@ -# Ecosystem topology - -## Discovery surfaces - -Inspect more than the package landing page: - -- workspace declarations and package directories; -- root and nearest package manifests; -- export maps and registry metadata; -- organization repositories and pinned projects; -- documentation navigation and integration indexes; -- examples, starters, templates, presets, and test fixtures; -- source imports and peer/optional dependencies; -- release workflows, changelogs, and deprecation notices; -- protocol or specification implementations outside the owner organization. - -## Relationship classes - -Use one of these labels for each related project: - -| Label | Required evidence | -|---|---| -| First-party workspace sibling | Same verified workspace and owner | -| First-party repository sibling | Same verified owner with an explicit relationship | -| Official adapter or plugin | Listed and maintained through official project surfaces | -| Specification peer | Implements the same documented interoperability contract | -| Community integration | Third-party project with no first-party guarantee | -| Alternative | Overlaps ownership and normally replaces rather than complements | -| Incidental adjacency | Similar domain or organization, no task-relevant contract | -| Unresolved | Identity or relationship could not be established | - -Never turn adjacency into compatibility. Similar names, shared maintainers, -organization membership, mirrored APIs, or an ecosystem directory are leads, -not guarantees. - -## Ecosystem patterns - -- Monorepos often split core, adapters, integrations, testing, configuration, - and presentation across packages. Search exports and package manifests before - assuming the core package is the entire product. -- Multi-repository ecosystems such as UnJS can be cohesive without one - monorepo. Select packages by capability ownership, not by brand membership. -- Specification ecosystems such as Standard Schema or unified connect projects - through contracts. Confirm conformance and version boundaries independently. -- Framework ecosystems may offer renderer-specific bindings. A React example - does not prove a Solid, Vue, server, or native binding exists. -- Private and personal ecosystems may only be observable through consuming - repositories. Treat remembered names as discovery hints until source is found. +# Ecosystem topology and relationship discovery + +## Contents + +- [When to load this reference](#when-to-load-this-reference) +- [Outcome](#outcome) +- [Topology classes](#topology-classes) +- [Discovery algorithm](#discovery-algorithm) +- [Relationship taxonomy](#relationship-taxonomy) +- [Monorepo package discovery](#monorepo-package-discovery) +- [Multi-repository ecosystem discovery](#multi-repository-ecosystem-discovery) +- [Specification and framework ecosystems](#specification-and-framework-ecosystems) +- [Private, personal, and generated ecosystems](#private-personal-and-generated-ecosystems) +- [Capability graph and boundaries](#capability-graph-and-boundaries) +- [Stopping and completeness](#stopping-and-completeness) +- [Failure signatures](#failure-signatures) +- [Sources and freshness](#sources-and-freshness) + +## When to load this reference + +Load this reference after a material dependency is identified and before assuming +the named package represents its full capability surface. It is especially +important for monorepos, organizations such as UnJS, packages with many official +adapters/plugins/drivers, framework bindings, specifications, and remembered +private/personal libraries. + +## Outcome + +Produce a bounded, evidence-backed graph in which nodes have exact identity and +status, edges have named semantics, capability ownership is visible, and +incidental adjacency is excluded. The graph should reveal missing siblings or +adapters without pressuring the implementation to install all of them. + +## Topology classes + +Classify actual topology after investigation: + +| Class | Defining evidence | Common discovery risk | +|---|---|---| +| Standalone | no task-relevant first-party/spec siblings found | inventing an ecosystem from naming | +| Monorepo | verified workspace root and package manifests | inspecting only the flagship package | +| Multi-repository ecosystem | canonical owner/docs connect separate repositories | treating all org repos as one product | +| Package family | separately published core/adapters/plugins under one project | installing core without needed adapter | +| Framework integration ecosystem | bindings by renderer/runtime/deployment | copying a binding for the wrong host | +| Specification ecosystem | independent implementations share a versioned contract | assuming optional capabilities interoperate | +| Service/platform ecosystem | API/SDK/CLI/plugins plus remote service | ignoring credentials, quotas, region, lifecycle | +| Private/personal ecosystem | relationships observed in source/consumers, not public registry | inventing unpublished APIs | +| Hybrid | more than one of the above with verified edges | flattening status/version across layers | +| Unresolved | identity or relationships cannot be established | asserting topology from memory/search snippets | + +"Ecosystem" is a capability relationship, not a compliment or brand category. + +## Discovery algorithm + +### 1. Seed exact identities + +For each known import/product, record package name, resolved version, registry, +repository, owner, manifest path, export map, and integrity/revision. Expand aliases, +workspace protocols, patches, overrides, forks, and generated clients. + +### 2. Inspect local graph + +Search: + +- root/nearest manifests and workspace declarations; +- lockfile package and peer/optional dependency edges; +- import graph, wrappers, reexports, lazy/dynamic imports; +- config/plugin/adapter registration; +- examples/tests/fixtures and generated code; +- tasks/build/release/deployment/docs; +- patches and vendored source. + +Local use can reveal an ecosystem package absent from current docs, but it does +not establish that the relationship remains supported upstream. + +### 3. Inspect canonical project surfaces + +Search exact-version source first, then current: + +- repository workspace/package directories and manifests; +- root README/docs navigation/API/package index; +- examples, starters, templates, recipes, integrations; +- package exports, peers, optional dependencies, plugin registries; +- changelog/release/deprecation/migration notes; +- CI matrices and tests connecting packages; +- ownership/maintainer/security policy. + +### 4. Inspect organization and registry carefully + +Organization repositories, scopes, keywords, download pages, and search results +generate candidates only. Confirm each relationship through canonical docs, +source edges, maintenance, or a specification. Do not mark every `@scope/*` +package a companion. + +### 5. Expand only decision-changing edges + +Follow an edge when it owns a required capability, adapts a selected owner to the +repository's host, changes version/security/deployment risk, or is a plausible +alternative. Do not recursively inventory the entire ecosystem. + +## Relationship taxonomy + +Assign one primary label and evidence/status to each edge. + +| Label | Required evidence | Inclusion implication | +|---|---|---| +| Workspace sibling | same verified workspace/revision and package manifest | candidate only; inspect capability | +| First-party repository sibling | same owner plus explicit project relationship | candidate only | +| Official adapter/driver/plugin | canonical docs/index and maintained compatibility | include if host/capability requires | +| Optional integration | peer/optional/source edge with conditional behavior | include only with feature and fallback policy | +| Implements specification | named/versioned contract and conformance evidence | verify optional behavior separately | +| Generates | one node deterministically produces another | generator/source ownership required | +| Publishes | source/build maps to registry artifact | artifact/version integrity required | +| Alternative/replacement | overlapping owner boundary | choose; do not stack silently | +| Community integration | third party, no first-party support promise | higher verification/maintenance burden | +| Experimental | explicitly unstable/prerelease/research | isolate and version-pin | +| Deprecated/superseded | canonical deprecation/migration evidence | avoid new adoption; plan migration | +| Incidental adjacency | name/domain/org similarity only | exclude | +| Unresolved | insufficient identity/relationship evidence | no API or support claim | + +Edges can be asymmetric. A community adapter may support an upstream package +without upstream support. One repository can list an example without committing +to the example library's versions. Record direction and source. + +## Monorepo package discovery + +Do not assume the root manifest lists the public packages or that directory names +equal published names. + +1. Find actual workspace config (`package.json`, `pnpm-workspace.yaml`, + `deno.json`, Cargo/Go/Python equivalents, custom generation). +2. Expand workspace globs with exclusions; detect missing/ignored/nested roots. +3. Read every material package manifest: name, version strategy, private flag, + exports/bin/files, peers/optional/runtime dependencies, engines, publish config. +4. Map internal dependency and peer edges plus package categories. +5. Inspect root docs/integration index and per-package tests/examples. +6. Identify generated packages, compatibility shims, deprecated packages, and + packages not published despite living in the workspace. +7. Record versioning/release policy: lockstep, independent, canary, prerelease. + +Common capability splits: + +```text +core protocol/types + -> runtime implementation + -> framework bindings + -> platform/driver adapters + -> configuration/loaders + -> observability/testing + -> build/devtools +``` + +Inspect public exports. A package named `core` may contain runner/discovery +features; a directory named `adapter` may be internal. Names are hints. + +## Multi-repository ecosystem discovery + +UnJS demonstrates a cohesive multi-repository ecosystem. Its package index and +cross-dependencies connect focused tools, but each package owns a narrow concern: + +- c12 discovers/resolves config layers; +- defu supplies defaults-oriented merge primitives; +- jiti loads/transforms modules at runtime and executes code; +- rc9 owns user RC persistence; +- std-env reports runtime/CI/TTY/provider signals; +- ofetch owns an HTTP client adapter; +- unstorage owns portable key/value storage drivers; +- pkg-types inspects package/workspace metadata and exports; +- nypm detects/operates through project package managers; +- unbuild builds libraries; changelogen assists release notes/version actions; +- Automd owns bounded generated Markdown; Giget acquires templates; Magicast + edits supported static-ish JS/TS shapes. + +They can work together because capability boundaries align, not because all +come from UnJS. Select by required owner. For example, using c12 does not require +unbuild or unstorage. Citty and Optique normally compete for command grammar; +they are not companions merely because Citty is used by other UnJS tools. + +For a multi-repository ecosystem, record package-level versions independently. +The same attached codebase can resolve c12 stable 3.x while research also finds +4.x beta; do not assign the beta's combined capability to stable. + +## Specification and framework ecosystems + +A specification edge is narrower than a package-family edge. Standard Schema can +let consumers validate with Zod, Valibot, or other conforming libraries through +a minimal interface. It does not guarantee enum introspection, JSON Schema, +labels, transformations, codecs, completion choices, or identical error shapes. +Build a required/optional conformance table and test the exact implementations. + +Framework ecosystems require host dimensions: + +```text +framework version x renderer x server adapter x bundler x runtime x deployment +``` + +An icon loader's Astro integration and its React/Solid compiler can have different +imports, SSR output, styling, and bundler hooks. A database ORM's PostgreSQL +dialect cannot be inferred to support ClickHouse because SQL methods look alike. +A Better Auth plugin may require both server and client registration and an +adapter/schema migration. Map exact peer ranges and integration tests. + +Official examples are topology evidence, not universal compatibility proof. + +## Private, personal, and generated ecosystems + +For `@okikio/*`, custom adapters, or unpublished packages: + +- search consuming workspaces, lockfiles, local registries, import maps, source + directories, docs, tests, and package metadata; +- distinguish exact spelling and version (`observables` versus a remembered + misspelling); +- inspect actual exports and maturity; do not fill gaps from a similar public API; +- record unavailable source as unresolved and provide the next required path; +- never publish or expose private implementation details just to complete a map. + +Generated clients/schemas/packages have two identities: generator/source schema +and generated artifact. Connect both. A generated API client's version may not +match the service version; capture protocol/schema compatibility. + +Duplicate archives or mirrors require content comparison. Same paths/digests +mean one evidence body with multiple archive identities, not two evolutionary +stages. Forks require base revision plus patch set and update policy. + +## Capability graph and boundaries + +Prefer a table for review and a graph only when topology is genuinely complex. + +```text +required capability + -> current owner + -> candidate node + -> relationship status + -> version/host compatibility + -> include/exclude/unresolved + -> verification +``` + +Each selected capability needs one canonical owner. Companions connect different +owners; alternatives overlap. Explicit examples: + +- Optique owns grammar, LogTape owns output transport, `@optique/logtape` + connects verbosity/configuration: companion. +- LogTape and another process-wide logger both owning diagnostics: alternative + unless one is a bounded bridge. +- PostgreSQL owns transactional state, ClickHouse owns analytical projections: + companion with a delivery/reconciliation contract. +- Two ORMs both writing the same schema/migrations: duplicate authority. + +Include connected systems: build, generated code, package manager, CI, runtime, +data store, service, authentication, observability, deployment, and docs. + +## Stopping and completeness + +Topology is sufficiently complete when: + +- the required capability path has no unexplained gap; +- each selected edge has primary evidence and version/status; +- likely official adapters/siblings for the actual host were inspected; +- overlapping alternatives and plausible exclusions are recorded; +- unresolved nodes cannot be included without more source; +- expanding more nodes would not change ownership, risk, or verification. + +Report coverage, not omniscience: "inspected workspace packages, official +integration index, export/peer edges, and current releases; did not enumerate +unrelated organization repositories." + +## Failure signatures + +| Signature | Likely topology error | Next evidence | +|---|---|---| +| Core looks too small for docs | capability in sibling/adapter | workspace, exports, integration index | +| Package exists in scope but is private | workspace mistaken for publication | manifest/private/publish workflow | +| Community project called official | ownership edge collapsed | canonical docs/maintainer policy | +| Same org treated as required stack | adjacency treated as capability | owner table and dependency edges | +| React example proposed for Solid | framework binding dimension omitted | peer ranges, compiler/renderer docs | +| Similar dialect API proposed | structural resemblance treated as support | dialect/driver source and generated SQL | +| All UnJS packages recommended | ecosystem discovery confused with selection | material capability path | +| Private package API invented | unresolved node promoted | consuming source/export evidence | +| Two archives yield fake chronology | names used instead of hashes | normalized content comparison | +| Graph is huge but decision unclear | unbounded recursive discovery | stopping rule and decision-changing edges | + +## Sources and freshness + +- Current UnJS package index and pinned published source records for c12, defu, + jiti, Unbuild, Pkg-types, Automd, Changelogen, Giget, Magicast and related + focused packages; verified 2026-07-17. +- Current Optique and LogTape official full documentation, observed package-family + and integration separation; verified 2026-07-17. +- Retained uploaded monorepos/codebases and source registry, observed package, + framework, private/personal, duplicate-archive, generated and stale-contract + topologies; verified 2026-07-17. + +Organization membership and mutable package indexes are discovery surfaces. +Recheck exact versions, maintainers, package manifests, and current integration +status before making an implementation decision. diff --git a/skills/use-okikio/SKILL.md b/skills/use-okikio/SKILL.md index 6043af1..9a75601 100644 --- a/skills/use-okikio/SKILL.md +++ b/skills/use-okikio/SKILL.md @@ -1,6 +1,6 @@ --- name: use-okikio -description: Research, select, integrate, review, or debug Okikio-maintained libraries and recurring project patterns without inventing private APIs. Use for @okikio/undent, @okikio/wikitext, @okikio/sparql, remembered observables packages, backend endpoint/query/response/database utilities, service modules, workflow control-plane utilities, custom ClickHouse Drizzle work, package generation, and related personal repositories. +description: Research, select, integrate, review, or debug Okikio-maintained libraries and recurring project patterns without inventing private APIs. Use for @okikio/undent, @okikio/wikitext, @okikio/sparql, @okikio/observables, backend endpoint/query/response/database utilities, service modules, workflow control-plane utilities, custom ClickHouse Drizzle work, package generation, and related personal repositories. --- # Use Okikio libraries and patterns @@ -39,8 +39,10 @@ Do not claim availability above the strongest evidence. experimental code, unexported helpers, and executable behavior. 4. Match the local repository's schema, logging, configuration, resource, and composition contracts. Reuse a pattern only when those contracts align. -5. Preserve Zod v4 schema-first boundaries, LogTape process output, explicit - resource lifetime, and source-grounded verification. +5. Preserve the consuming repository's validated schema, logging, configuration, + and resource-lifetime owners. Reuse Zod v4 or LogTape when the repository has + selected them; do not introduce either merely because an Okikio example uses + it. 6. If source is unavailable, state the exact inspection needed and offer an interface or discovery plan, not invented imports. @@ -50,6 +52,8 @@ Do not claim availability above the strongest evidence. alignment, newline, and Unicode display-width choices. - [wikitext.md](references/wikitext.md): token/event/tree cost ladder, diagnostics, sessions, maturity, and missing exports. +- [observables.md](references/observables.md): exact published 1.4.0 Observable, + operator, error-mode, EventBus, backpressure, teardown, and interop contracts. - [backend.md](references/backend.md): service modules, endpoint, validation, query, response, server, database, and auth patterns. - [workflows.md](references/workflows.md): control-plane and durable-workflow diff --git a/skills/use-okikio/references/backend.md b/skills/use-okikio/references/backend.md index da87cac..59657ae 100644 --- a/skills/use-okikio/references/backend.md +++ b/skills/use-okikio/references/backend.md @@ -1,42 +1,319 @@ -# Backend utility and service-module patterns +# Okikio backend utilities and service-module architecture -## Endpoint definitions +Use this reference when working inside a retained Okikio finance-style repository or evaluating whether its private/workspace backend utilities should be reused. These are observed codebase capabilities, not presumed public packages. Verify manifests, exports, versions, and consumer imports before using any symbol. -The reviewed utilities include a `defineEndpoint(...)` pattern that preserves -literal method, route, and schema types, and `matchSchema(...)` over Standard -Schema's `~standard.validate`. A reusable definition can accept validator-neutral -schemas while the application remains Zod v4-first. +## Contents -Register matching Hono validator middleware before reading validated request -values. Map expected validation failures to stable 422 problems and unexpected -validator defects to internal diagnostics. +- Status and package identity +- Complete service-module model +- Endpoint contracts +- Validation and Standard Schema seam +- Response and Problem Details +- Query utilities +- Database and execution utilities +- Auth and middleware utilities +- Server composition +- Integration sequence +- Known counterexamples and repair +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Service modules +## Status and package identity -The documented pattern assigns route contracts to `definition.ts`, behavior to -`handler.ts`, aggregate definitions to `mod.ts`, and root dependencies, handlers, -one server, and registration to `index.ts`. Treat this as a target pattern only -where the repository actually instantiates the services. +Observed private/workspace packages under the retained finance codebase include: -Use `workflows/` for business coordination and `runtime/` for worker/engine -bootstrap. Avoid nested server factories and silently skipped handlers. +```text +@utils/endpoint +@utils/response +@utils/query +@utils/execution +@utils/middleware +@utils/server +@utils/db +@utils/auth +@utils/workflows +@utils/env +``` -## Query and response utilities +Do not install these names from a registry based on this guide. Before reuse: -The reviewed query utilities separate source parsing, filters, sorts, fields, -cursor/offset pagination, count mode, and execution. Server-owned base filters -must precede user filters. Exact, planned, estimated, and no-count modes have -different contracts. +1. Inspect the consumer's workspace manifests and lockfile. +2. Resolve the actual package path/version. +3. Read its `deno.jsonc`/`package.json` export map and `mod.ts`. +4. Confirm whether the code is source-imported, built, or published. +5. Run import/type/runtime tests at the exact consumer revision. -Problem utilities support stable OpenAPI responses and problem-type registries. -Describe complete result/response variants and never expose raw database errors. +The current reference describes the uploaded new/old finance source as of 2026-07-17. Some files contain active repairs, experimental workflow work, and counterexamples. -## Database and server cautions +## Complete service-module model -Central Drizzle re-export can prevent duplicate incompatible class instances. -Database factories should expose underlying lifetime/close ownership. Avoid the -reviewed counterexamples: LogTape configuration at module import, wildcard CORS -inside “production” defaults, literal patch markers in docs, and raw database -messages in public failures. +The retained guides define this target architecture: -Compose with `build-apis` and `build-data` for full implementation and tests. +```text +services/<service>/ + endpoints/ + <group>/ + <action>/ + definition.ts + handler.ts + mod.ts + index.ts optional, plain route module only + workflows/ durable business orchestration only when real + activities/ side-effecting workflow units + runtime/ engine/layer/worker/registration wiring + domain/ service domain schemas/rules/errors + data/ repositories/storage-specific operations + mod.ts contract registry + index.ts runtime registry and one service composition root +``` + +Ownership: + +| File/folder | Owns | Must not own | +|---|---|---| +| endpoint `definition.ts` | name, route, methods, request-source schemas, response contract | DB clients, auth sessions, workflow runtime | +| endpoint `handler.ts` | route middleware, validated input, domain/workflow call, response mapping | root framework setup, import-time resources | +| group/service `mod.ts` | aggregated `EndpointDefinitions` contract objects | server construction/side effects | +| service `index.ts` | `EndpointHandlers`, one server, dependency middleware, route registration | reusable pure contract exports only | +| `workflows/` | durable step ordering/retry/wait/compensation | HTTP details or worker bootstrap | +| `activities/` | side effects with idempotency/retry contract | orchestration state machine | +| `runtime/` | engine Layers/adapters, registration, worker host | product route copy | + +Do not create `workflows/activities/runtime` as empty architecture theatre. Add them only when durable behavior and deployment exist. `orchestrator/` is intentionally avoided because it can confuse business orchestration with runtime/client ownership. + +## Endpoint contracts + +The observed `defineEndpoint(...)` pattern preserves literal name, method, route, schema, and response types. Conceptual shape: + +```ts +export default defineEndpoint({ + Name: 'list-accounts', + Route: '/organizations/:organization_id/accounts', + Methods: ['GET'] as const, + Schemas: { + Param: ParamSchema, + Query: QuerySchema, + Header: HeaderSchema, + }, + Response: ResponseSchema, +}) +``` + +Observed source kinds include JSON, Query, Param, Header, Form, and Cookie through endpoint/validation utilities. Verify the exact generic names and schema keys in current source. + +Handler shape: + +```ts +export const Middleware = [ + createAuthOptionalContextMiddleware(auth), + createAuthRequiredContextMiddleware(auth), + createOrganizationRequiredContextMiddleware(), + createValidator('param', Definition.Schemas.Param), + createValidator('query', Definition.Schemas.Query), +] + +export const Handler = async (c) => { + const params = c.req.valid('param') + const query = c.req.valid('query') + // call domain/data/workflow capability +} +``` + +Middleware must match every `c.req.valid(source)` call. A definition alone does not install validators. Route definition/handler/middleware mapping must be checked at startup and by a real request. + +## Validation and Standard Schema seam + +The retained endpoint utilities include a `matchSchema(...)` pattern over Standard Schema's `~standard.validate`. This can keep endpoint definitions validator-neutral while an application uses Zod v4 or another Standard Schema implementation. + +Operational requirements: + +- confirm the object implements the installed Standard Schema contract; +- handle sync and async validation results; +- normalize issue paths/messages without losing source location; +- keep transport decoding separate from domain validation; +- map expected validation to a stable 400/422 contract; +- treat validator exceptions/invalid implementations as internal defects; +- infer types through the selected schema library's supported helper, not guessed properties. + +Do not force Zod when a consumer selected another Standard Schema implementation. Conversely, do not claim any object with a `parse` method is Standard Schema compatible. + +## Response and Problem Details + +Observed `@utils/response` areas include: + +- success helpers and schemas; +- status codes; +- error/problem helpers; +- RFC 9457-style Problem Details; +- problem type registries/docs/localization/trace extensions in current server integration. + +Model full response behavior: + +```text +status + headers + payload/envelope +``` + +not payload alone. Stable problems should separate machine fields (`type`, `status`, stable extension codes) from localized display text (`title`, `detail`). Resolve request-specific `instance` and trace at the HTTP adapter. Do not return raw database/provider/stack messages. + +Verify actual exports before naming helpers such as `ok`, `accepted`, `badRequest`, `gone`, or `internalServerError`; these names are observed in uploaded source but may change or remain private. + +## Query utilities + +Observed `@utils/query` capabilities: + +- bracket/JSON/form filter adapters; +- per-field operator/type/enum registries; +- operators including equality/range/set/string/null; +- filter count caps; +- sort adapters/defaults/allowlists/tiebreakers; +- simple and JSON:API-style field selection; +- offset and signed HMAC cursor pagination; +- query component disable flags; +- exact/planned/estimated/no-count metadata; +- composite endpoint query schemas; +- extensive tests and benchmarks. + +Important caveats: + +- exact exports are workspace-private until confirmed; +- empty allowlist semantics must be checked and should not expose arbitrary storage fields; +- cursor schema inspected does not bind filter/resource context visibly; +- query compilation remains storage-specific; +- server-owned organization/resource filters must precede user filters; +- type/schema success does not prove query plans or stable pagination. + +Compose with `build-apis/references/queries.md` and `build-data/references/queries.md` rather than copying these utilities blindly. + +## Database and execution utilities + +Observed `@utils/db` provides: + +- lazy `DATABASE_URL` validation; +- `postgres.js` client construction; +- Drizzle `postgres-js` wrapper with shared schema; +- auth, finance, and workflow schema exports; +- optional Drizzle query logger integration; +- a central Drizzle re-export surface to avoid duplicate class identity; +- Drizzle Kit config/migrations. + +Observed default choices include pool `max: 10` and `prepare: false`; derive consumer settings from actual topology. The current `createDatabase()` hides the underlying client inside the returned Drizzle wrapper, so close/drain ownership needs repair or an alternate factory before claiming graceful shutdown. + +Observed `@utils/execution` contains storage-specific execution helpers for Drizzle/database and SPARQL. The SPARQL implementation maps query filters/sorts/cursors into an `@okikio/sparql` builder and transport. Inspect it carefully: some special operators combine raw expression strings, date cursors may reduce precision, and term comparison semantics depend on the engine. + +## Auth and middleware utilities + +Observed auth utilities include server/client factories, framework-specific clients (React/Solid/Svelte/Vue/Lynx paths in source), environment parsing, routes, and Better Auth integration. Confirm exact plugins/options at installed Better Auth version. + +Observed middleware areas include: + +- authentication and optional/required context; +- organization optional/required context; +- correlation/trace/logger access; +- DB context; +- validation. + +Policy order: + +```text +session identification + -> current user requirement + -> current organization membership/permission + -> resource belongs to authorized organization + -> validated public query/body + -> server-scoped data operation +``` + +Do not treat active organization stored in a session as timeless authorization. Revalidate according to sensitivity/long-lived stream policy. + +## Server composition + +The intended pattern is one `createServer(...)` call per service root. Root middleware can own request ID, correlation, diagnostics, CORS, security headers, timing, trailing slash policy, health, and final error handling. Service middleware adapts long-lived dependencies. Route middleware owns auth/policy/validation. + +Hono does not deduplicate middleware. Calling the root factory in nested groups can execute logging/correlation/CORS/timing twice. + +Current retained server counterexamples to repair before reuse: + +- default `cors: true` described as all origins; +- module-scope `await configure(...)` for LogTape; +- global logger configuration from a reusable package; +- pretty JSON enabled by default in a production-optimized object; +- literal `+` patch markers in a comment block; +- import-time boot log. + +These violate import safety/configuration ownership and should not be normalized as “Okikio style.” If LogTape is selected, the application entrypoint configures it once and injects/uses categories; otherwise preserve the selected logger. + +## Integration sequence + +```text +definition.ts declares public contract + -> mod.ts aggregates definitions without side effects + -> composition root parses config and constructs DB/auth/providers/runtime + -> index.ts maps definition names to handler modules + -> startup fails or marks unavailable for mismatches + -> root middleware installed once + -> route middleware establishes auth/policy/validation + -> handler calls domain/data/workflow capability + -> response helper emits declared status/headers/body + -> error boundary maps one stable problem and correlated diagnostic + -> shutdown drains and closes owned resources +``` + +## Known counterexamples and repair + +| Observed/candidate pattern | Why unsafe/incomplete | Repair | +|---|---|---| +| definition/handler exists | not necessarily registered/reachable | conformance plus real request | +| nested `createServer()` | root middleware duplicates | one composition root | +| `c.req.valid()` without validator | no runtime guarantee | matching middleware | +| import-time LogTape configure | reusable module mutates global state | app entrypoint owns config | +| wildcard CORS default | unsafe with credentials/private APIs | explicit origin/method/header policy | +| DB wrapper hides client | shutdown handle missing | return/accept owned resource | +| raw DB message in Problem | internal/security leak | stable mapping, redacted cause | +| empty 200 stub | false capability | unregister/501/implement | +| workflow folder without worker/runtime | architecture scaffold | prove durable path or remove | + +## Test matrix + +Test: + +- import `mod.ts`, definitions, schemas without env/network/global logger mutation; +- all definition names/methods/routes unique; +- every definition has one matching handler and declared middleware; +- OpenAPI and runtime route inventories agree; +- each request source validator and normalized output; +- positive/negative auth and organization/resource policy; +- middleware exact invocation count/order; +- stable success and every problem response variant; +- query filters/sorts/fields/cursors/counts and semantic compatibility; +- DB/schema migration and pool close/drain; +- SPARQL generated query/term behavior against the selected engine; +- dependency timeout/cancellation/restart; +- standalone boot, readiness, actual HTTP requests, and shutdown; +- workflow trigger/status/cancel only when runtime capability is real. + +## Executable verification + +Use workspace-native tasks. At minimum: + +```bash +deno check services/<service>/mod.ts services/<service>/index.ts +deno test -P utils/endpoint utils/query utils/response utils/middleware utils/db +``` + +Adapt to actual task ownership and permissions. Boot the service with disposable dependencies, issue real requests for every route and negative policy, inspect route/OpenAPI registries, run migration and query integration, and confirm process exit after shutdown. Package-level type tests alone do not prove a service module. + +## Deliberate exclusions + +- Do not claim `@utils/*` packages are public/installable without registry/repository evidence. +- Do not invent exports from remembered names; inspect `mod.ts` and export maps. +- Do not force Zod, LogTape, Drizzle, Hono, Better Auth, Effect, or SPARQL into a consumer that selected another owner. +- Do not copy known counterexamples as architecture. +- Do not add workflow folders when work is synchronous/bounded and no durable runtime is selected. +- Do not treat generated OpenAPI, definitions, or types as reachability proof. +- Do not hide client lifetimes or configure process globals at import. + +## Sources and freshness + +Grounded in the retained new/old finance service-module structure and authoring guides and the complete observed `utils/{endpoint,response,query,execution,middleware,server,db,auth,workflows,env}` sources, tests, benchmarks, schemas, manifests, and migrations, reviewed 2026-07-17. These are private/workspace observations unless a specific public package is independently verified. Several files contain experimental or incomplete paths; the status and counterexamples above are source-level findings, not accusations about a deployed service. diff --git a/skills/use-okikio/references/observables.md b/skills/use-okikio/references/observables.md new file mode 100644 index 0000000..9b48619 --- /dev/null +++ b/skills/use-okikio/references/observables.md @@ -0,0 +1,266 @@ +# `@okikio/observables` 1.4.0 + +## Contents + +- Status and runtime support +- Core Observable contract +- Creation and consumption +- Operator families +- Error model +- EventBus and EventDispatcher +- Backpressure and pull consumption +- Resource lifetime and cancellation +- Custom operators and interop +- Selection guide +- Failure signatures +- Verification +- Sources and freshness + +## Status and runtime support + +JSR reports `@okikio/observables` 1.4.0 as the latest release on 2026-07-17. It is a TC39-inspired, Web-Streams-backed Observable implementation for Deno, Node, Bun, browsers, and workers. It has no runtime dependencies according to the published package page. + +Pin the version before using exact behavior: + +```ts +import { Observable, filter, map, pipe } from "jsr:@okikio/observables@1.4.0"; +``` + +The npm-compatible package is also documented as `@okikio/observables`. Verify which registry produced the installed lockfile. + +## Core Observable contract + +An `Observable<T>` is cold by default: the subscriber body runs independently for each subscription. The observer receives `next`, `error`, and `complete`; subscription teardown runs once on unsubscribe, completion, error, abort, or disposal according to the source contract. + +```ts +const ticks = new Observable<number>((observer) => { + let value = 0; + const id = setInterval(() => observer.next(value++), 1_000); + return () => clearInterval(id); +}); + +using subscription = ticks.subscribe({ + next: (value) => consume(value), + error: (error) => report(error), + complete: () => finish(), +}); +``` + +`observer.start(subscription)` runs before the subscriber body. If an attached signal is already aborted, `start()` still sees a closed subscription and the subscriber body is skipped. + +Do not confuse cold Observables with an `EventBus`, which is hot and multicasts one event source. + +## Creation and consumption + +Confirmed creation/consumption surfaces: + +- `new Observable(subscriber)` for an owned producer; +- `Observable.of(...values)` for fixed values; +- `Observable.from(...)` for supported promises, iterables, async iterables, array-like inputs, and objects implementing `Symbol.observable`; +- `subscribe(observer)`; +- `subscribe(next, error?, complete?)`; +- subscribe options with `AbortSignal` cancellation; +- `for await ... of observable`; +- `observable.pull(...)` for backpressure-aware async iteration; +- `using`/`Symbol.dispose` and async disposal. + +Verify exact overloads from 1.4.0 types before wrapping third-party subscribables; `Observable.from()` is intentionally narrower than the interop helper. + +## Operator families + +Operators are tree-shakeable pipeline stages used through `pipe(source, ...operators)`, not prototype methods. + +Published and documented families include: + +| Concern | Confirmed operators/helpers | +|---|---| +| transform/filter | `map`, `filter`, `scan` | +| bounds/search | `take`, `drop`, `find`, `findIndex`, `first`, `elementAt` | +| flatten/concurrency | `mergeMap`, `concatMap`, `switchMap` | +| combination | `withLatestFrom`, `combineLatestWith`, `zipWith`, `raceWith` | +| identity/change | `changed`, `unique` | +| scheduling | `debounce`, `throttle` | +| errors | `catchErrors`, `ignoreErrors`, `mapErrors`, `tapError` | +| custom stages | `createOperator`, `createStatefulOperator` | +| native/foreign interop | `fromStreamPair`, `fromObservableOperator` | + +`pipe()` in 1.4.0 supports up to 19 operators per call at the type level. Split longer chains into named, testable helpers. + +Choose flattening semantics deliberately: + +- `switchMap`: later input supersedes/cancels prior inner work; good for search; +- `concatMap`: preserve order and run one inner operation after another; +- `mergeMap`: permit concurrency; define a bound and output ordering expectation. + +```ts +const results = pipe( + searchInput, + debounce(250), + filter((query) => query.length >= 3), + switchMap((query) => + pipe( + Observable.from(fetch(`/search?q=${encodeURIComponent(query)}`)), + map((response) => response.json()), + catchErrors([]), + ) + ), +); +``` + +Confirm that the underlying fetch receives a usable abort signal when cancellation must stop network work; switching an Observable is not proof that an unrelated Promise was aborted. + +## Error model + +The package's operator system has four modes: + +| Mode | Behavior/intent | +|---|---| +| `pass-through` | thrown transformation failures become `ObservableError` values for later recovery; documented default | +| `ignore` | skip the failure/value path | +| `throw` | fail fast | +| `manual` | custom operator owns all error behavior; lowest overhead/higher responsibility | + +This differs materially from RxJS-style terminal error assumptions. Built-in 1.4.0 operators use pass-through behavior by default. A failed mapping can continue as an `ObservableError` value until `catchErrors`, `ignoreErrors`, `mapErrors`, or `tapError` handles it. + +Do not silently choose `ignore` for data ingestion. Decide whether one bad record is rejected, emitted as a typed result, stops the stream, or enters a dead-letter path. Preserve correlation and redacted diagnostics. + +Custom operator: + +```ts +const decode = createOperator<string, Record<string, unknown>>({ + name: "decode-json", + errorMode: "throw", + transform(value, controller) { + controller.enqueue(JSON.parse(value)); + }, +}); +``` + +## EventBus and EventDispatcher + +Use `EventBus<T>` for one hot typed channel: + +```ts +using bus = new EventBus<ProgressEvent>(); +bus.events.subscribe(updateProgress); +bus.emit({ stage: "parse", completed: 10 }); +``` + +Use `createEventDispatcher<EventMap>()` for distinct typed event names: + +```ts +interface AppEvents { + signedIn: { userId: string }; + listUpdated: { listId: string }; +} + +const events = createEventDispatcher<AppEvents>(); +events.on("listUpdated", ({ listId }) => refreshList(listId)); +events.emit("signedIn", { userId: "user_123" }); +``` + +Published docs also describe waiting for events. Inspect exact `waitForEvent` signature in the pinned types before use. + +An in-memory bus is not durable, replayable, cross-process, transactional, or guaranteed delivery. Do not use it as a Temporal/queue/outbox replacement. It is appropriate for process-local fan-out and UI coordination. + +## Backpressure and pull consumption + +The library uses Web Streams internally. `pull()` exposes an async generator backed by a `ReadableStream`, allowing a slow consumer to apply backpressure rather than accumulating an unbounded callback queue. + +```ts +for await ( + const batch of source.pull({ strategy: { highWaterMark: 8 } }) +) { + await persist(batch); +} +``` + +Backpressure is only end-to-end when the source observes demand or its buffering is bounded. DOM events, WebSockets, and external callbacks may continue producing. Define overflow policy: drop, latest, bounded buffer, pause upstream, spill, or fail. + +Use callback subscription for low-cost push reactions; use `pull()`/async iteration when sequential async processing or producer pacing matters. + +## Resource lifetime and cancellation + +Teardown may be a cleanup function, subscription `unsubscribe`, `Symbol.dispose`, or `Symbol.asyncDispose`. It must run exactly once. + +Every source must handle: + +- synchronous subscriber throw; +- observer cancellation before setup completes; +- abort during async work; +- completion/error race; +- inner `switchMap` cancellation; +- early `for await` break; +- disposal of EventBus/dispatcher; +- timers/listeners/socket/readable stream cleanup. + +Prefer scoped `using` when target runtimes/toolchains support it. Otherwise use `try/finally` or explicit unsubscribe. Test teardown counts; do not infer cleanup from types. + +## Custom operators and interop + +`createStatefulOperator` creates per-subscription state: + +```ts +function movingAverage(size: number) { + return createStatefulOperator<number, number, number[]>({ + name: "moving-average", + createState: () => [], + transform(value, state, controller) { + state.push(value); + if (state.length > size) state.shift(); + controller.enqueue(state.reduce((sum, item) => sum + item, 0) / state.length); + }, + }); +} +``` + +Use `fromStreamPair(() => new CompressionStream("gzip"))` to adapt readable/writable platform transforms. + +Use `fromObservableOperator()` for RxJS operator functions. Standard RxJS operators need a `sourceAdapter`, commonly RxJS `from(source)`. Alias overlapping imports. The wrapper accepts a wider direct-subscribable result than `Observable.from()` and wraps synchronous subscription failures as `ObservableError` values. + +Interop has cost and semantic risk. Verify cancellation, error, scheduling, hot/cold behavior, backpressure, and teardown across the boundary. + +## Selection guide + +| Need | Choose | +|---|---| +| one eventual value | Promise/Effect, not Observable | +| multiple push values with transformations | Observable pipeline | +| sequential async consumption with pacing | `pull()` / async iteration | +| process-local one-to-many channel | `EventBus` | +| typed named process-local events | EventDispatcher | +| durable/replayable cross-process event | queue/workflow/event log, not this package | +| existing Web TransformStream | `fromStreamPair` | +| isolated reuse of an RxJS operator | `fromObservableOperator` | + +## Failure signatures + +| Signature | Likely defect | Correction | +|---|---|---| +| interval/listener survives unsubscribe | source omitted teardown | return cleanup and assert once | +| old search response wins | merge/independent promises used | `switchMap` plus real abort propagation | +| memory grows under ingestion | push source outruns consumer | `pull`, bound, pause/drop/spill policy | +| error appears as ordinary value | pass-through mode not handled | add error operator or select `throw` | +| two subscribers duplicate request | cold source assumed hot | share through deliberate bus/cache owner | +| EventBus loses restart events | in-memory bus used durably | durable workflow/queue/outbox | +| RxJS stage leaks | interop teardown not propagated | boundary lifecycle test | +| typing fails at giant chain | over 19 operators/inference depth | split named pipelines | + +## Verification + +- test each cold subscription gets independent setup/state; +- assert cleanup once on complete, error, unsubscribe, abort, dispose, and early iterator return; +- test all four error modes with sync/async failures; +- test `switchMap`, `concatMap`, and bounded `mergeMap` ordering/cancellation; +- test backpressure/overflow under a fast producer and slow consumer; +- test EventBus/EventDispatcher hot multicast and disposal; +- test native stream and RxJS interop both directions; +- benchmark representative pipelines and retained memory in fresh processes; +- run Deno, Node, and browser compatibility checks required by the consumer. + +## Sources and freshness + +- Primary: [JSR `@okikio/observables@1.4.0`](https://jsr.io/@okikio/observables/1.4.0), inspected 2026-07-17 for the documented lifecycle, operators, error modes, events, streams, interop, and runtime support. +- Attachment status: no `@okikio/observables` source archive was provided in this evidence set; uploaded consumers are not treated as package API authority. + +Version 1.4.0 is the verified boundary. Undocumented exports, exact scheduler/backpressure internals, and APIs from other versions remain unverified until the target export map, declarations, and source are inspected. diff --git a/skills/use-okikio/references/packages.md b/skills/use-okikio/references/packages.md index 4d980d0..1104cee 100644 --- a/skills/use-okikio/references/packages.md +++ b/skills/use-okikio/references/packages.md @@ -1,39 +1,235 @@ -# Personal packages, releases, and unknowns +# Okikio package map, evidence, release, and integration protocol -## Packaging patterns +Use this reference to decide whether and how to use an Okikio-owned package or private workspace utility. Names from memory are discovery leads. Only inspected manifests, public registry metadata, export maps, source, tests, and consuming imports establish identity and capability. -The reviewed undent build demonstrates clean-output generation, root plus -`./unicode` entrypoints, Deno and generated Node checks, explicit exclusions for -Deno-only tests, `sideEffects: false`, a Node engine floor, semver validation -from one release version, and deterministic license/README/changelog copying. +## Contents -Verify source-runtime tests, generated package type/runtime behavior, every -export, package contents, version propagation, and clean consumers separately. +- Evidence classes +- Current package map +- Selection and verification workflow +- Public package integration +- Private workspace integration +- Release and generated-package contracts +- Cross-package ecosystem analysis +- Unknown or missing packages +- Test matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -## Generated data +## Evidence classes -The reviewed Unicode data synchronizer uses check/write modes, explicit Deno -permissions, an immutable upstream version behind a mutable latest URL, SHA-256 -comparison, AST-based update, and no rewrite in check mode. Preserve those -properties and avoid unrelated formatting. +Label every claim: -## Performance experiments +| Class | Meaning | Allowed claim | +|---|---|---| +| verified public | official registry/repo/export evidence inspected | exact version/export/capability at verified date | +| observed uploaded source | code exists in retained archive | capability/status of that snapshot only | +| private workspace | local package/export map in an app repo | usable only in that workspace/revision unless published separately | +| experimental/unreleased | version/status or docs show prototype | do not recommend as stable production dependency | +| remembered/unresolved | name supplied from memory, source not established | search/ask; never invent APIs | -The wikitext event-shape study keeps baseline/candidate source and harness -snapshots, runs timing and retained-memory in fresh processes, preserves raw -samples, uses deterministic ordering, applies statistical/effect-size gates, and -protects parse/session workflows from microbenchmark regressions. +Do not convert an observed `deno.json` name into proof that a package was published. Do not convert a registry package into proof that the consumer uses it. -## Unverified package protocol +## Current package map -For remembered observables, backend helpers, custom adapters, or misspelled -package names: +As reviewed 2026-07-17: -1. search consuming manifests, lockfiles, imports, registries, and owner repos; -2. establish exact identity and status; -3. inspect public exports and tests; -4. state what remains unavailable; -5. define the needed interface without assigning invented exports; -6. request or locate source before implementation. +| Identity | Evidence/status | Observed scope | Detailed reference | +|---|---|---|---| +| `@okikio/undent` | uploaded source `0.3.3`; public identity known from retained project | indentation removal/alignment plus `./unicode`; Deno and generated npm release | `references/undent.md` | +| `@okikio/observables` | public JSR evidence previously verified at `1.4.0` | observable/reactive primitives; inspect exact exports/operators/adapters | `references/observables.md` | +| `@okikio/sparql` | public JSR evidence previously verified at `0.0.2` | SPARQL builder/terms/execution/result helpers; early version | `references/sparql.md` | +| `@okikio/wikitext` | uploaded source version `0.0.0` | tokenizer/parser/events/tree/filter/stringify/session experiments | `references/wikitext.md` | +| `@utils/endpoint` | private finance workspace `1.0.0` | endpoint definitions/schemas/types, Standard Schema seam | `references/backend.md` | +| `@utils/response` | private finance workspace `1.0.0` | success/error/problem/response schemas | `references/backend.md` | +| `@utils/query` | private finance workspace `1.0.0` | filters/sorts/fields/cursors/count/query configs | `references/backend.md` | +| `@utils/execution` | private finance workspace `1.0.0` | Drizzle and SPARQL query execution adapters | `references/backend.md` | +| `@utils/db` | private finance workspace; manifest version inconsistent between files | PostgreSQL/Drizzle/schema/migration/client | `references/backend.md` | +| `@utils/workflows` | private finance workspace `1.0.0`, incomplete durable runtime | control plane/store/workers/Effect seam | `references/workflows.md` | +| `@utils/auth`, middleware, server, env | private finance workspace | Better Auth, Hono middleware/server/config | `references/backend.md` | -Names from memory are useful discovery leads, never API contracts. +Potential remembered names such as `@okikio/obserables` are misspelled until proven otherwise; do not install them. “Backend utils,” “service modules,” or “custom ClickHouse Drizzle library” describe code/architecture and do not establish a public package name. + +## Selection and verification workflow + +For each dependency request: + +1. Normalize the remembered name but preserve possible spellings. +2. Search the consumer's manifests, lockfiles, import maps, source imports, vendored code, and workspace config. +3. Search official JSR/npm/GitHub owner sources when public identity matters. +4. Record exact selected version/source and whether it is public/private/experimental. +5. Inspect the export map and relevant source at that version. +6. Inspect sibling packages, subpaths, adapters, tests, benchmarks, examples, release tooling, and consumers—the ecosystem hypothesis. +7. Build a capability matrix including unsupported/unknown behavior. +8. Choose the smallest package/subpath that matches the use case. +9. Run an import/type/runtime integration fixture. +10. Record sources/freshness and unresolved questions. + +Never answer “what API does this have?” from name familiarity. If official source cannot be inspected, define the required interface generically and state that integration is blocked. + +## Public package integration + +### `@okikio/undent` + +Observed uploaded `deno.json`: + +```json +{ + "name": "@okikio/undent", + "version": "0.3.3", + "exports": { ".": "./mod.ts", "./unicode": "./unicode.ts" } +} +``` + +The package owns two explicit entrypoints. Inspect their exported functions and use `references/undent.md`; do not invent additional subpaths. The release verifies Deno lint/doc/test/bench, generated npm build, dry-run publish, and Unicode data synchronization. Consumer tests must cover whitespace/newline/tab/Unicode width semantics relevant to output. + +### `@okikio/observables` + +Use `references/observables.md` for the verified 1.4.0 capability map. Inspect actual JSR exports and avoid importing internal source paths. Establish subscription, teardown, error/completion, scheduling, reentrancy, async iteration, and interop semantics. Do not treat it as RxJS or Solid signals by resemblance. + +### `@okikio/sparql` + +Use `references/sparql.md`. At verified public version 0.0.2, treat it as early and pin deliberately. Separate term construction, query building, transport, parsing, and domain mapping. Verify escaping/datatypes/prefixes/engine compatibility and never use raw fragments with untrusted input. + +### `@okikio/wikitext` + +Uploaded manifest version is `0.0.0`; do not represent it as a stable public release. Observed export is only root `./mod.ts`, while published include candidates list AST, events, text source, tokenizer, parsers, parse/tree/stringify/filter/session source. Inclusion does not create subpath exports. Import only from the declared root unless the manifest changes. + +## Private workspace integration + +Private `@utils/*` packages use workspace/import-map resolution. Preserve their boundary: + +- do not publish accidentally because a manifest has a name/version; +- use declared export subpaths only; +- align Deno/npm dependency versions across workspace; +- keep import-time code environment/network/global-side-effect free; +- run each package's tasks with the repository's permissions/tool wrapper; +- verify service-level composition, not only package unit tests; +- decide whether extraction to a public package is actually beneficial. + +Observed export maps matter. For example, `@utils/workflows` exports only `.`; its internal `runtime/*` paths are not declared public. `@utils/db` declares client/env/drizzle/schema/types subpaths. `@utils/execution` declares `./db` and `./sparql`. + +Before importing an internal file to access a missing symbol, decide whether it should become an explicit supported export or remain internal. Do not bypass the export map casually. + +## Release and generated-package contracts + +The uploaded Undent project demonstrates a thorough cross-runtime release pattern: + +- one Deno source package; +- explicit JSR include/exclude; +- clean generated npm output; +- root and `./unicode` entrypoints; +- generated Node checks; +- Deno-only tests excluded from npm artifact where necessary; +- `sideEffects: false`/Node engine/package metadata in generated artifact (inspect generator); +- version validation/propagation; +- deterministic license/README/changelog copying; +- release verification/dry run. + +For any dual JSR/npm package verify separately: + +```text +source Deno import/type/test + -> clean generation + -> generated package exports/types/runtime + -> package contents/metadata + -> packed install into clean Node consumer + -> ESM/CJS policy and Node floor + -> version/license/readme/changelog parity +``` + +Generated files should be reproducible and checked for drift. Never format unrelated handwritten Markdown as a side effect of generation. + +Unicode/generated-data update also needs: + +- pinned immutable upstream version behind mutable discovery URL; +- download checksum/content validation; +- check versus write modes; +- AST/targeted update instead of broad formatting; +- explicit Deno permissions; +- no rewrite when content is unchanged; +- tests covering the new table/data behavior. + +## Cross-package ecosystem analysis + +Assume every package may have an ecosystem, then verify relationships: + +- owner monorepo/workspace siblings; +- root/subpath entrypoints; +- framework/runtime adapters; +- testing/benchmark/build packages; +- examples and real consumers; +- shared schemas/types/protocols; +- version compatibility and peer dependencies; +- generated artifacts and release channels. + +Do not force every discovered sibling. Example integration hypotheses: + +- `@okikio/undent` can improve multiline stable CLI/docs output, but should not transform machine JSON or arbitrary user content; +- `@okikio/sparql` can back graph query construction, but the API query contract and engine remain separate owners; +- Observables can model event streams, but do not replace durable event storage, SSE replay, or Solid's own reactivity without evidence; +- Wikitext parsing can feed pipelines, but its `0.0.0` status and performance/memory experiments require deliberate adoption; +- private query/execution utilities can compose, but public package extraction would need stable contracts, exports, fixtures, docs, and release work. + +## Unknown or missing packages + +When source/API is unavailable: + +```text +Requested capability: ClickHouse Drizzle-like adapter +Identity: unresolved/private code architecture +Verified exports: none +Do not assume: package name, transactions, migrations, RETURNING, Drizzle compatibility +Required interface: schema -> dialect -> driver/session -> result mapping -> migration artifacts +Next evidence: owner repo/imports/manifest/source/conformance fixtures +``` + +This is more useful than fabricating a code example. A generic interface example must be labeled conceptual and must not use invented import paths. + +## Test matrix + +For every selected package: + +- identity/source/version resolution and lockfile pin; +- declared root and subpath exports; +- typecheck and runtime import in each supported Deno/Node/browser host; +- documented primary use case; +- edge/error/cancellation/resource-lifetime behavior; +- framework/engine integration at installed versions; +- unsupported import/subpath/API rejection; +- clean consumer without monorepo path leakage; +- duplicate dependency/class identity where relevant; +- generated package tarball/JSR dry run and contents; +- version/license/readme/changelog provenance; +- regression benchmark only where it protects an actual workflow; +- source freshness and security/advisory review. + +For private utilities, also run whole-service boot, requests, migrations, worker loops, and shutdown. + +## Executable verification + +Inspect exact exports: + +```bash +deno info jsr:@okikio/undent@0.3.3 +deno info jsr:@okikio/observables@1.4.0 +deno info jsr:@okikio/sparql@0.0.2 +``` + +Network/registry access may be unavailable; use the consumer lockfile/cache/source and record the limitation. In uploaded source, run repository tasks such as `deno task release:verify` only with required dependencies/permissions, inspect the generated npm package, pack/install it in a clean consumer, and compare exports. For `@utils/*`, use workspace tasks and real service integration. + +## Deliberate exclusions + +- Do not invent package names, exports, subpaths, or maturity from memory. +- Do not claim a private workspace package is public. +- Do not claim Wikitext `0.0.0` is a stable release. +- Do not import internal files past an export map without an explicit ownership decision. +- Do not force Okikio packages where the consumer has selected another owner or the package adds no value. +- Do not equate Observables with durability, SPARQL builders with engine safety, or Drizzle-shaped APIs with Drizzle parity. +- Do not publish generated packages without clean-consumer verification. +- Do not format unrelated Markdown during generation/release. + +## Sources and freshness + +Grounded in uploaded `@okikio/undent` 0.3.3 source/build/release tooling, uploaded `@okikio/wikitext` 0.0.0 source/experiments, retained public-registry verification for `@okikio/observables` 1.4.0 and `@okikio/sparql` 0.0.2, and complete private finance `@utils/*` manifests/export maps/source, reviewed 2026-07-17. Public registry state can change; reverify before release or upgrade. Private/workspace identities remain local unless independently published. diff --git a/skills/use-okikio/references/sparql.md b/skills/use-okikio/references/sparql.md index 9d88247..09aaa4d 100644 --- a/skills/use-okikio/references/sparql.md +++ b/skills/use-okikio/references/sparql.md @@ -1,33 +1,232 @@ -# `@okikio/sparql` and SPARQL execution +# `@okikio/sparql` 0.0.2 -The reviewed consumer uses `@okikio/sparql` through query construction, -execution, error mapping, a query specification, filter/sort/pagination mapping, -result transforms, and safe query preview. Inspect the actual package source and -exports at the installed revision before writing API calls. +## Contents -## Boundary model +- Status and layers +- Values and expressions +- Graph patterns +- Query construction +- Updates +- Executor boundary +- Safe public query mapping +- Federation and engine differences +- Failure signatures +- Verification +- Sources and freshness + +## Status and layers + +JSR reports `@okikio/sparql` 0.0.2 as the latest release on 2026-07-17. It is a type-safe SPARQL 1.1 builder for Deno, Node, Bun, browsers, and workers. Version `0.0.2` is early: pin it and inspect exports before upgrades. + +The public documentation describes three layers: + +1. values/terms and expressions; +2. composable graph patterns; +3. complete fluent query/update builders and execution. + +The uploaded Kaiju/finance utilities additionally import executor symbols from `@okikio/sparql/executor`, establishing that subpath for 0.0.2 consumers: + +```ts +import { raw, select, triple, v } from "@okikio/sparql"; +import { + executeSparql, + QueryError, + transformResults, +} from "@okikio/sparql/executor"; +``` + +Do not assume the executor subpath or these exports exist unchanged in another release. + +## Values and expressions + +Use `v("name")` for variables and fluent value expressions. Confirmed fluent/standalone capabilities include: + +- comparisons: `eq`, `neq`, `gt`, `gte`, `lt`, `lte`; +- boolean composition: `and`, `or`; +- arithmetic: `add`, `sub`, `mul`, `div`, `mod`; +- math: `abs`, `round`, `ceil`, `floor`; +- strings: `concat`, `strlen`, `ucase`, `lcase`, `contains`, `startsWith`, `endsWith`, `regex`, and standalone `substr`; +- conditionals: `ifElse`, `coalesce`; +- checks: `isNull`, `isNotNull`, `isIri`, `isBlank`, `isLiteral`, `bound`; +- aliasing: `.as("name")`; +- aggregates demonstrated through `count()` and `avg(...)`. + +```ts +const finalPrice = ifElse( + v("inStock").eq(true), + v("basePrice").mul(0.9).round(), + v("basePrice").add(10), +).as("finalPrice"); +``` + +Core conversion escapes string literals and formats numbers/dates according to the package. Verify exact RDF datatype/language behavior for application values; JavaScript string conversion does not prove the desired RDF term. + +Treat `raw(...)` as unsafe trusted SPARQL. The uploaded `QuerySpec` adapter builds `IN`, `NOT IN`, `BETWEEN`, and cursor predicates by concatenating `.value` fragments into `raw(...)`. This is a counterexample requiring review: prefer package-native composition/term serialization and allowlisted fields. Never put untrusted values into raw query text. + +## Graph patterns + +Confirmed pattern styles compose: + +```ts +triple("?person", "foaf:name", "?name"); +``` + +```ts +node("product", "schema:Product", { + "schema:name": v("name"), + "schema:publisher": node("publisher", "schema:Organization", { + "schema:name": v("publisherName"), + }), +}); +``` + +```ts +match( + node("person", "foaf:Person", { "foaf:name": v("name") }), + rel("person", "foaf:knows", "friend"), + node("friend", "foaf:Person", { "foaf:name": v("friendName") }), +); +``` + +The `cypher` template tag provides visual ASCII-like path syntax that compiles to SPARQL triples. Use it only with verified pattern objects and static relation syntax; it is not a Cypher database query and must not accept arbitrary user interpolation. + +Property-path helpers demonstrated by primary docs: + +- `zeroOrMore(predicate)`; +- `sequence(...predicates)`; +- `alternative(...predicates)`. + +Also confirmed: named graph `graph(...)`, federated `service(...)`, and nested `subquery(...)` patterns. + +## Query construction + +Basic select: + +```ts +const adults = select(["?name", "?age"]) + .where(triple("?person", "foaf:name", "?name")) + .where(triple("?person", "foaf:age", "?age")) + .filter(v("age").gte(18)) + .orderBy("?name") + .limit(100); + +const text = adults.build(); +const result = await adults.execute({ endpoint }); +``` + +Confirmed fluent clauses/examples include: + +- repeated `.where(...)`; +- `.filter(...)`; +- `.bind(expression.as(...))`; +- `.groupBy(...)` and `.having(...)`; +- `.orderBy(variable, direction?)`; +- `.limit(...)` and consumer evidence for `.offset(...)`; +- `.fromNamed(...)`; +- subqueries, named graphs, and services; +- `.build()` and `.execute({ endpoint })`. + +The package page claims full SPARQL 1.1 support and exposes hundreds of symbols. Do not infer an exact factory name for ASK, CONSTRUCT, DESCRIBE, dataset clauses, or every update form from that claim alone; inspect 0.0.2 symbols/tests before use. + +## Updates + +Primary docs demonstrate `modify()` with delete/insert/where and `.done()`: + +```ts +const incrementAge = modify() + .delete(triple("?person", "foaf:age", "?oldAge")) + .insert(triple("?person", "foaf:age", v("oldAge").add(1))) + .where(triple("?person", "foaf:age", "?oldAge")) + .where(filter(v("oldAge").gte(0))) + .done(); + +await incrementAge.execute({ endpoint: updateEndpoint }); +``` + +Updates require separate authorization, endpoint capability, graph ownership, idempotency, timeout, audit, and partial-failure semantics. A builder prevents some syntax defects; it does not make arbitrary graph mutation safe. + +## Executor boundary + +The uploaded consumer uses: + +- `BindingMap` and `QueryResult<T>` types; +- `executeSparql(...)`; +- `transformResults(...)`; +- `QueryError` with `kind`, message, query, and optional status. Separate: -- domain/query specification; -- approved fields, predicates, sorts, and pagination; -- RDF term and parameter serialization; -- SPARQL text construction; -- HTTP transport and credentials; -- result media type and parsing; -- domain result transform; -- engine/protocol failure mapping; -- redacted preview and diagnostics. +```text +domain QuerySpec + -> approved fields/predicates/operators + -> SPARQL builder + -> safe query text + bounded preview + -> executor (endpoint, timeout, cancellation, credentials) + -> media type/status parsing + -> binding transform + -> domain result +``` + +Map errors by stable kind/status. The uploaded tests map timeout to gateway timeout, upstream authorization to bad gateway/configuration, and rate limiting to a retryable response. Do not expose the full query when it can contain sensitive literals; keep a redacted, length-bounded preview. + +Verify whether `.execute(...)` and standalone `executeSparql(...)` share configuration and result behavior. Pick one owner per application layer. -Do not concatenate untrusted values or identifiers into query text. Use the -package's verified term/parameter builders and allowlists. Engine-specific -features and update semantics require deployed-engine evidence. +## Safe public query mapping + +For API filters/sorts: + +1. validate transport input with Zod/Standard Schema; +2. map public fields to approved variables/predicates; +3. apply server-owned tenant/graph/base patterns first; +4. map operators to builder expressions; +5. serialize values through package term/value APIs; +6. add a stable sort tie-breaker; +7. encode a cursor bound to sort/filter context; +8. bound limit, timeout, result bytes, and graph complexity; +9. keep updates unavailable unless explicitly authorized. + +Never accept a predicate, service endpoint, graph IRI, variable name, raw filter, or update fragment directly from an untrusted request. + +The uploaded cursor builder converts dates to a manually typed `xsd:date` string and interpolates fragments. Replace this with verified typed-literal/package helpers when available; otherwise isolate and rigorously escape/test the serializer. + +## Federation and engine differences + +`service(endpoint, pattern)` emits federated SPARQL, but the target engine controls support, timeouts, credentials, result limits, and SSRF risk. Never let a browser/user choose arbitrary service endpoints. + +QLever, Blazegraph, Virtuoso, Fuseki, and other engines differ in extensions, update support, inference, full-text search, optimizer behavior, and protocol details. Inspect the deployed engine, dataset, and endpoint. A repository README naming Blazegraph does not prove a current QLever deployment behaves the same. + +Define namespace/prefix ownership and collision behavior. Preserve RDF term identity, datatype, language tags, blank-node scope, and graph provenance through result mapping. + +## Failure signatures + +| Signature | Likely defect | Correction | +|---|---|---| +| filters allow injected syntax | `raw` concatenates untrusted value/field | native term builder plus allowlist | +| cursor repeats/skips | order lacks deterministic tie-breaker | compound order/filter and context-bound cursor | +| query works on one engine only | extension/inference assumed standard | engine compatibility fixture | +| result loses datatype/language | binding flattened to string | typed binding/domain transform | +| authorization filter missing | base patterns applied after/optionally | server-owned mandatory scope | +| timeout leaves request alive | AbortSignal not propagated | executor cancellation test | +| logs expose secrets/query literals | raw query logged | redacted bounded preview | +| update modifies wrong graph | graph/endpoint authority unclear | explicit graph and permission contract | +| service clause reaches internal URL | user-controlled federation | endpoint allowlist/disable federation | ## Verification -Inspect generated SPARQL, test Unicode/language/datatype values, optional and -missing bindings, pagination/order stability, HTTP status and malformed results, -timeout/cancellation, credential redaction, and representative engine behavior. +- assert exact `.build()` output for triples, nodes, paths, filters, binds, groups, subqueries, graph/service, and updates; +- test string escaping, Unicode, IRIs, dates, datatypes, language tags, blank nodes, and malicious values; +- test every approved filter/operator without `raw` interpolation; +- execute against the actual QLever/Blazegraph/etc. versions; +- test empty/optional/missing bindings and transform behavior; +- test timeout, abort, 401/403, 429, 5xx, malformed media, and huge result; +- test cursor stability under concurrent changes; +- test tenant/graph isolation and federation endpoint allowlists; +- test update authorization, idempotency, and audit; +- pin 0.0.2 and rerun export/SQL goldens before upgrade. + +## Sources and freshness + +- Primary: [JSR `@okikio/sparql@0.0.2`](https://jsr.io/@okikio/sparql/0.0.2), inspected 2026-07-17 for documented value, expression, pattern, query, update, and executor surfaces. +- Attachments: `new-finance-app(1).zip` and `kaiju-site-scope(17).zip`, inspected 2026-07-17 for concrete consumer imports, query mapping, executor use, and raw-interpolation counterexamples. -PopModern's older manual SPARQL implementations are historical/counterexample -evidence, not automatically the current package contract. +Version 0.0.2 is experimental and version-sensitive. Symbols not present in the verified docs or consumers remain unverified; endpoint-specific SPARQL support must be tested against the actual engine. diff --git a/skills/use-okikio/references/undent.md b/skills/use-okikio/references/undent.md index ce3c0ff..f0eb862 100644 --- a/skills/use-okikio/references/undent.md +++ b/skills/use-okikio/references/undent.md @@ -1,43 +1,365 @@ # `@okikio/undent` -The reviewed package is version `0.3.3` with root and `./unicode` exports. Verify -the installed version before depending on exact behavior. +## Contents -## Selection table +- [When to load this reference](#when-to-load-this-reference) +- [Evidence and version boundary](#evidence-and-version-boundary) +- [Capability model](#capability-model) +- [Choose the API by intent](#choose-the-api-by-intent) +- [Indent detection and trimming](#indent-detection-and-trimming) +- [Interpolation and alignment](#interpolation-and-alignment) +- [Explicit indentation anchors](#explicit-indentation-anchors) +- [Line-ending ownership](#line-ending-ownership) +- [Unicode and terminal columns](#unicode-and-terminal-columns) +- [Configured instances](#configured-instances) +- [Integration patterns](#integration-patterns) +- [Incorrect patterns](#incorrect-patterns) +- [Failure signatures](#failure-signatures) +- [Verification](#verification) +- [Sources and freshness](#sources-and-freshness) -| Need | API | +## When to load this reference + +Load this reference for readable multiline literals, generated source, SQL or +GraphQL snippets, CLI help, snapshots, nested code generation, interpolation +alignment, line-ending preservation, or terminal-width-sensitive output. + +Do not load it merely because a repository depends on `@okikio/undent`. + +## Evidence and version boundary + +The reviewed source is `@okikio/undent` `0.3.3`. It exposes the root module and +an opt-in `@okikio/undent/unicode` entry point. Verify the installed version and +export map before copying an exact import or relying on implementation details. + +The package solves source indentation and interpolation layout. It is not a +general code formatter, terminal table engine, escaping system, SQL builder, or +security boundary. + +## Capability model + +The root API separates five decisions that shallow usage often conflates: + +1. which indentation is structural; +2. which blank wrapper lines are removed; +3. whether source line endings are preserved or normalized; +4. whether interpolated multiline values are aligned; +5. how the insertion column is measured. + +The important exports in the reviewed version are: + +| Capability | Export | |---|---| -| Readable multiline template or string | `undent` | -| Runtime-loaded string | `undent.string(...)` or `dedentString(...)` | -| Align a multiline interpolation at its insertion column | `align(...)` | -| Dedent an already-indented snippet, then align it | `embed(...)` | -| Mark an explicit template baseline | `indent` | -| Test whether a value is an alignment wrapper | `isAligned(...)` | -| Create configured behavior | `createUndent(...)` | -| Terminal columns with tabs/emoji/CJK/combining marks | `createUnicodeColumnOffset(...)` | - -`dedent` is an alias and `outdent` is a configured variant in the reviewed -source. Inspect public exports before relying on either in another version. - -## Important distinctions - -- `align(...)` aligns the supplied value; it does not first dedent it. -- `embed(...)` provides dedent-plus-align behavior for pre-indented snippets. -- Newline normalization is opt-in. LF, CRLF, and CR sequences are preserved by - default. -- Core indentation counts raw characters. Terminal visual width requires the - Unicode column-offset export. -- `strategy: "common"` and `"first"` are different behavior choices. -- Left/right trim modes support `"all"`, `"one"`, and `"none"` independently. -- Internal bounded caches are performance details, not correctness or security - guarantees. +| Default tagged-template processor | `undent` | +| Alias for the default processor | `dedent` | +| Classic outdent-compatible policy | `outdent` | +| Build an independent configured tag | `createUndent()` | +| Process a runtime string | `undent.string()` or `dedentString()` | +| Align an interpolation without dedenting it | `align()` | +| Dedent and then align an interpolation | `embed()` | +| Set an explicit indentation baseline | `undent.indent` or `indent` | +| Recognize an alignment wrapper | `isAligned()` | +| Resolve configuration | `resolveOptions()` | +| Low-level line operations | `splitLines()`, `rejoinLines()`, `alignText()` | +| Raw insertion-column measurement | `columnOffset()` | +| Visual terminal measurement | `createUnicodeColumnOffset()` and related `./unicode` exports | + +Prefer the high-level tag, factory, and wrapper functions. Low-level exports +are useful for integration or conformance work, but reassembling the algorithm +from them creates more surface for semantic drift. + +## Choose the API by intent + +### A readable static literal + +```ts +import { undent } from "@okikio/undent"; + +const query = undent` + SELECT account_id, sum(amount) AS total + FROM ledger_entries + WHERE posted_at >= {start:DateTime64} + GROUP BY account_id +`; +``` + +This strips source-code indentation. It does not validate or parameterize SQL. + +### A runtime-loaded string + +```ts +const source = await Deno.readTextFile("template.sql"); +const normalizedIndent = undent.string(source); +``` + +Use `.string()` rather than pretending a dynamic value is a template segment. + +### A multiline value that is already left-aligned + +```ts +import { align, undent } from "@okikio/undent"; + +const items = "- inspect\n- plan\n- verify"; + +const result = undent` + phases: + ${align(items)} +`; +``` + +`align()` stringifies the value and pads later lines to the insertion column. +It does not remove indentation already present inside the value. + +### A snippet with baked-in indentation + +```ts +import { embed, undent } from "@okikio/undent"; + +const fragment = ` + SELECT id + FROM users +`; + +const result = undent` + query: + ${embed(fragment)} +`; +``` + +`embed()` first applies the package's string dedent behavior and then aligns the +result. This is the correct distinction when snippets originate in separately +indented constants or files. + +## Indent detection and trimming + +`UndentOptions.strategy` controls the structural baseline: + +- `"common"` scans content lines and strips the smallest shared indentation; +- `"first"` uses the first content line as the reference and matches the + package's documented classic `outdent` behavior. + +The default is `"common"`. Do not select `"first"` only because it sounds +faster: it changes semantics when later lines are less indented. + +`trim` accepts one policy for both ends or separate leading and trailing +policies. Each side supports: + +- `"all"`: remove all edge blank lines; +- `"one"`: remove at most one edge blank line; +- `"none"`: preserve wrapper lines. + +```ts +const preserveLeading = undent.with({ + trim: { leading: "none", trailing: "all" }, +}); +``` + +Whitespace-sensitive formats need exact-string tests for the selected policy. +A snapshot that visually hides the first or last blank line is insufficient. + +## Interpolation and alignment + +Alignment is opt-in by wrapper, or global per configured instance through +`alignValues: true`. + +```ts +const generator = undent.with({ alignValues: true }); + +generator` + command: + ${"first line\nsecond line"} +`; +``` + +Prefer explicit `align()` or `embed()` when only selected values need layout +treatment. Global alignment is useful for a dedicated generator, but can alter +unrelated interpolations and should not be enabled casually on a shared helper. + +Interpolation is stringification, not syntax-aware composition. Values still +need the correct escaping or parameterization for their destination: + +- SQL values belong in driver parameters; +- HTML values belong behind the renderer's escaping contract; +- shell arguments belong in argument arrays rather than generated command text; +- source identifiers require language-specific validation. + +Internal caches used for repeated embedded snippets are bounded performance +details. They neither sanitize content nor guarantee a cache hit. + +## Explicit indentation anchors + +Use `${undent.indent}` as the first interpolation on its own line when a code +generator needs a deliberate baseline instead of inferred common indentation. + +```ts +function emitFunction(name: string): string { + return undent` + ${undent.indent} + export function ${name}() { + return true; + } + `; +} +``` + +Content at the anchor column becomes column zero; deeper content preserves its +relative offset. The anchor is a layout instruction, not emitted text. + +Use it when nesting depth in the implementation must not change generated +output. Avoid it when normal common-indent detection already states the intent. + +## Line-ending ownership + +The default `newline: null` preserves `LF`, `CRLF`, and lone `CR` sequences in +template segments. Set `newline: "\n"` only when the output contract requires +normalization. + +```ts +const portableGeneratedText = undent.with({ newline: "\n" }); +``` + +Newlines inside interpolated values are not rewritten by the template-segment +normalization option. Normalize dynamic snippets separately if the whole output +must use one convention. + +This distinction matters for golden files, generated source, protocol payloads, +and repositories that preserve platform-native endings. + +## Unicode and terminal columns + +The root `columnOffset()` counts raw JavaScript string units after the final +newline. That is deterministic for source layout but not equivalent to display +columns for tabs, combining marks, CJK characters, or emoji. + +The `./unicode` entry point supplies terminal-oriented measurement: + +```ts +import { undent } from "@okikio/undent"; +import { + createUnicodeColumnOffset, +} from "@okikio/undent/unicode"; + +const terminalText = undent.with({ + alignValues: true, + columnOffset: createUnicodeColumnOffset({ + tabWidth: 4, + ambiguous: "narrow", + }), +}); +``` + +The reviewed Unicode options are: + +- `tabWidth: false | positive integer`; `false` treats a tab as one column, + while an integer advances to tab stops; +- `ambiguous: "narrow" | "wide"` for East Asian ambiguous-width code points; +- `widthOf(grapheme, state)` to override individual grapheme widths and return + `undefined` for the default behavior. + +`visualColumnWidth()` walks grapheme clusters and uses best-effort terminal +rules. Renderers can still disagree. The target terminal, font, locale, and +tab policy own the final visual result, so include real display fixtures where +column alignment is user-visible. + +## Configured instances + +`.with()` derives a new immutable instance from the current instance's resolved +settings; `createUndent()` begins from package defaults. + +```ts +const base = undent.with({ newline: "\n" }); +const exactFixture = base.with({ trim: "none" }); + +const classic = createUndent({ + strategy: "first", + trim: "one", +}); +``` + +Use dedicated instances for distinct output contracts rather than scattering +per-call policy choices. Good names describe the destination, such as +`generatedSource`, `terminalHelp`, or `snapshotText`. + +`dedent` is an alias of `undent`. `outdent` is not an alias: in the reviewed +version it is preconfigured with `strategy: "first"` and `trim: "one"`. + +## Integration patterns + +### Generated source + +- use an explicit anchor if implementation nesting must not affect output; +- normalize newlines only when the generated-artifact contract says so; +- use the target language's formatter after generation when canonical syntax + layout is required; +- compile or parse the generated artifact in verification. + +### CLI help and diagnostics + +- use `embed()` for separately authored blocks; +- use Unicode column measurement only for terminal-column alignment; +- keep stable machine output independent of decorative alignment; +- verify narrow widths, redirected output, no-color mode, and Unicode samples. + +### SQL, GraphQL, and configuration snippets + +- use `undent` for readable static structure; +- keep values parameterized; +- treat indentation cleanup and query safety as separate boundaries; +- execute representative syntax through the real parser or database in tests. + +### Snapshots + +- select trim and newline semantics explicitly; +- compare exact strings including invisible edges; +- avoid a project-wide helper whose implicit policy makes fixtures ambiguous. + +## Incorrect patterns + +Do not: + +- use `align()` when the value first needs dedenting; +- claim `embed()` escapes or validates the embedded language; +- assume default column measurement matches terminal width; +- normalize template-segment newlines and claim interpolation newlines changed; +- use `outdent` as if it were a name-only alias; +- depend on undocumented cache sizes or eviction order; +- convert every one-line literal into an `undent` template; +- format generated code visually without parsing, compiling, or executing it. + +## Failure signatures + +| Symptom | Likely cause | Next inspection | +|---|---|---| +| Second interpolation line jumps to column zero | Value was not wrapped and `alignValues` is false | Inspect `align()`, `embed()`, or configured instance ownership | +| Embedded block remains over-indented | Used `align()` for a snippet with baked-in indent | Replace with `embed()` and add exact output test | +| Significant leading blank line disappears | Default trim policy removed it | Set per-side trim explicitly | +| Mixed CRLF/LF output remains | Only template segments were normalized | Inspect interpolated values and whole-artifact policy | +| CJK or emoji causes visual drift | Raw string-unit column measurement | Use the Unicode entry point with target policy | +| Tabs align differently across terminals | Tab stops or renderer differ | Pin `tabWidth` and verify on supported terminals | +| Generated SQL is injectable | Layout helper was mistaken for query construction | Restore driver parameters or a query AST | +| Refactor changes generated indentation | Inferred baseline moved with source nesting | Introduce an explicit anchor where appropriate | ## Verification -Use exact-string tests covering blank edges, nested interpolation, tabs, CRLF, -empty values, strategy and trim modes. For terminal layout include emoji, CJK, -combining marks, ambiguous width, and tab stops under the selected Unicode -column policy. +Use table-driven exact-string tests covering: + +- empty strings and blank-only input; +- common versus first-line strategy; +- all, one, and no trimming on each edge; +- nested `align()` and `embed()` interpolation; +- explicit anchors at multiple source nesting levels; +- `LF`, `CRLF`, and lone `CR` preservation and normalization; +- dynamic interpolations containing their own line endings; +- tabs, emoji, CJK, combining marks, zero-width marks, and ambiguous-width text; +- custom `widthOf()` and invalid width/tab configurations; +- generated output parsed, compiled, or executed by the destination system. + +Verify behavior, not just importability. A strong check asserts the exact output +and then passes that output through the consumer whose contract it must satisfy. + +## Sources and freshness + +- Attachment: `undent.zip/mod.ts`, `unicode.ts`, README, tests, and release scripts from `undent.zip`, inspected 2026-07-17. +- Registry identity: [`@okikio/undent` on JSR](https://jsr.io/@okikio/undent); the detailed API in this reference is grounded in the attached `0.3.3` source, not inferred from the registry landing page. -Do not replace a simple one-line literal with `undent`. Use it where multiline -source readability or generated alignment materially improves. +The exact export map, options, Unicode tables, and cache implementation are version-sensitive. APIs not present in the attached source remain unverified. diff --git a/skills/use-okikio/references/wikitext.md b/skills/use-okikio/references/wikitext.md index 88bb690..82c7977 100644 --- a/skills/use-okikio/references/wikitext.md +++ b/skills/use-okikio/references/wikitext.md @@ -1,44 +1,408 @@ # `@okikio/wikitext` -The reviewed repository is experimental: manifest version `0.0.0`. Classify it -accordingly and inspect exports at the target revision. +## Contents -## Cost ladder +- [When to load this reference](#when-to-load-this-reference) +- [Evidence and maturity boundary](#evidence-and-maturity-boundary) +- [Architectural model](#architectural-model) +- [Choose the cheapest result](#choose-the-cheapest-result) +- [Text and position contracts](#text-and-position-contracts) +- [Token and event layers](#token-and-event-layers) +- [Tree materialization](#tree-materialization) +- [Malformed-input policies](#malformed-input-policies) +- [Findings-first workflows](#findings-first-workflows) +- [Sessions and cache lanes](#sessions-and-cache-lanes) +- [Filtering and diagnostics](#filtering-and-diagnostics) +- [Ecosystem integration](#ecosystem-integration) +- [Unsupported and planned surfaces](#unsupported-and-planned-surfaces) +- [Failure signatures](#failure-signatures) +- [Verification](#verification) +- [Sources and freshness](#sources-and-freshness) -1. `tokens()` for lexical inspection. -2. `outlineEvents()` for block-only structure. -3. `events()` for full streaming syntax. -4. `parse()` for a materialized unist-shaped tree. -5. Analyze once, then materialize with strict, tolerant, or recovery policy. -6. `createSession()`/`BasicSession` for repeated lanes over the same source. +## When to load this reference -Use the cheapest representation that satisfies the operation. Do not allocate a -full AST when a streaming event filter or outline is enough. +Load this reference for wikitext tokenization, structural extraction, linting, +indexing, syntax trees, diagnostics, tolerant parsing, editor tooling, or +integration with unist and unified utilities. -## Architecture +Do not load it for general Markdown, MediaWiki template expansion, HTML +rendering, or serialization unless the task also requires the current parser +surface and its limitations. -Source spans remain authoritative. Tokenization, block parsing, inline parsing, -events, tree building, diagnostics/recovery, filtering, sessions, and -materialization are distinct layers. One findings pipeline can support multiple -materialization policies without reparsing everything. +## Evidence and maturity boundary -Sessions cache lanes separately. Preserve deterministic ranges, token tiling, -properly nested events, optional diagnostics on hot paths, and arbitrary-input -no-throw behavior. +The reviewed repository declares version `0.0.0` and is explicitly +experimental. Confirm the target revision and public export map before using an +exact API. Current source, tests, and exports are authoritative when prose +describes planned work. -## Missing exports +The package is a source parser. It does not expand templates, evaluate parser +functions, resolve wiki pages, render MediaWiki-compatible HTML, or currently +serialize a tree back to wikitext. -The reviewed manifest publish include and README mention `stringify`, but no -`stringify.ts` or public export exists. The README also says `stringify()` and -`parseChunked()` are not implemented. Do not write imports for either until the -actual target export map and tests prove availability. +## Architectural model -This discrepancy is a deliberate anti-hallucination check: documentation intent -does not supersede current source. +The package is event-stream-first and range-first: + +```text +TextSource + -> tokenizer + -> block events + -> inline-enriched events + -> optional tree materialization + -> filters, diagnostics, or downstream tools +``` + +The event stream is the fundamental interchange format. A unist-shaped +`wikist` tree is a consumer-facing convenience when random access outweighs +allocation cost. + +Keep these layers separate: + +| Layer | Owns | Does not own | +|---|---|---| +| `TextSource` | UTF-16 source access and slicing | decoded MediaWiki semantics | +| tokenizer | lexical ranges and token types | nested document meaning | +| block parser | headings, lists, tables, paragraphs, block ranges | inline markup expansion | +| inline parser | inline events inside block text | tree policy | +| tree builder | materialization and recovery policy | source parsing itself | +| filters | selection and traversal | parsing or mutation persistence | +| session | cached lanes for one immutable source | editable-document history | + +## Choose the cheapest result + +Use a cost ladder, not `parse()` by habit: + +1. `tokens(source)` for lexical inspection. +2. `outlineEvents(source)` for block-only structure. +3. `events(source)` for the full streaming syntax model. +4. `parse(source)` for the cheapest materialized default tree. +5. diagnostics or recovery wrappers for malformed-input visibility. +6. `analyze()` plus `materialize()` when one parse must support inspection or + multiple tree policies. +7. `createSession()` when repeated lanes operate on the same immutable source. + +Examples: + +```ts +import { + events, + outlineEvents, + parse, + tokens, +} from "@okikio/wikitext"; + +for (const token of tokens(source)) { + indexToken(token.type, token.start, token.end); +} + +for (const event of outlineEvents(source)) { + collectOutlineEntry(event); +} + +for (const event of events(source)) { + runStreamingRule(event); +} + +const tree = parse(source); +``` + +Do not materialize a full tree merely to count headings or find an event type. +Conversely, do not build ad hoc state machines over events when a task needs +repeated parent/child navigation and a tree is the clearer representation. + +## Text and position contracts + +A string satisfies `TextSource`. The source abstraction exposes character-code +access and slicing so parsing can retain offsets rather than eagerly allocate +every text value. + +Offsets use UTF-16 code units, aligning with JavaScript string indices and LSP +positions. They are not Unicode code-point indices, grapheme indices, or byte +offsets. + +When integrating with a byte-oriented store or a grapheme-oriented editor: + +- retain the source and UTF-16 range as the parser's authority; +- perform explicit coordinate conversion at the boundary; +- never treat `start`/`end` as UTF-8 byte offsets; +- test astral emoji, combining sequences, CRLF, and non-Latin text. + +Range-first output makes exact source slices and diagnostics possible, but a +range is meaningful only with the source revision it came from. + +## Token and event layers + +The reviewed public root re-exports token types, `TokenType`, `isToken()`, and +`tokenize()`, plus event types, constructors, and guards. + +Events include enter, exit, text, token, and error forms. Use the exported type +guards rather than loosely matching arbitrary objects: + +```ts +import { + events, + isEnterEvent, + isErrorEvent, +} from "@okikio/wikitext"; + +for (const event of events(source, { diagnostics: true })) { + if (isEnterEvent(event) && event.node_type === "heading") { + collectHeading(event); + } + + if (isErrorEvent(event)) { + reportParserFinding(event); + } +} +``` + +`outlineEvents()` keeps inline content as text ranges. `events()` enriches the +outline with inline parsing. Diagnostics are an explicit option because error +events add work; they are not silently included in the cheapest lane. + +Low-level composition is available through `tokenize()`, `blockEvents()`, and +`inlineEvents()`. Use it for focused parser integration or testing. Most +consumers should prefer the orchestration wrappers so stage ownership stays +consistent. + +## Tree materialization + +The AST is a unist-shaped `wikist` model. Reviewed node categories include root, +heading, paragraph, preformatted text, lists and list items, definition lists, +tables, formatting, wiki and external links, images, templates and arguments, +parser functions, HTML-like tags, redirects, galleries, references, thematic +breaks, category links, magic words, behavior switches, signatures, breaks, +text, entities, and nowiki regions. + +The existence of a node type does not imply MediaWiki evaluation. For example, +a template node records syntax; it does not fetch or expand the template. + +Programmatic builders and type guards are public. Downstream tools can use +unist utilities, but must verify compatibility with the target revision and +preserve package-specific positions and node fields. + +```ts +import { filterTemplates, parse, visit } from "@okikio/wikitext"; + +const tree = parse(source); +const templates = filterTemplates(tree); + +visit(tree, (node, context) => { + inspectNode(node, context.path); +}); +``` + +## Malformed-input policies + +"Never throws on arbitrary input" does not mean every input is valid or every +consumer should accept the tolerant tree. Choose a result family deliberately: + +| API | Tree policy | Diagnostics | Recovery summary | +|---|---|---|---| +| `parse()` | default HTML-like/tolerant | discarded | no | +| `parseWithDiagnostics()` | same default tree | preserved | no | +| `parseWithRecovery()` | same default tree | preserved | `recovered` | +| `parseStrictWithDiagnostics()` | conservative/source-strict | preserved | no | + +The source-strict lane collapses recovery-heavy wrappers back toward text when +the source did not clearly commit to the recovered structure. It is useful for +linting and editors that must show a finding without presenting a speculative +wrapper as authoritative syntax. + +Known recovery classifications in the reviewed source include: + +- missing close; +- unterminated opener; +- unclosed table; +- mismatched exit; +- orphan exit; +- end-of-file autoclose. + +Treat the recovery taxonomy as versioned API. Do not infer future classes from +diagnostic prose. + +## Findings-first workflows + +`analyze()` separates parser facts from tree policy. It returns collected, +replayable events, diagnostics, and optionally a structured recovery list. +`materialize()` can then produce default or source-strict trees without +rerunning tokenization and parsing. + +```ts +import { + analyze, + materialize, + TreeMaterializationPolicy, +} from "@okikio/wikitext"; + +const findings = analyze(source); + +for (const recovery of findings.recovery ?? []) { + reviewRecovery(recovery.kind, recovery.position, recovery.anchor); +} + +const tolerant = materialize(findings); +const conservative = materialize(findings, { + policy: TreeMaterializationPolicy.SOURCE_STRICT, +}); +``` + +Use this lane when: + +- a tool needs diagnostics before deciding whether to build a tree; +- the same parse needs two materialization policies; +- recovery decisions must be auditable; +- an editor or linter presents parser facts separately from fixes. + +Diagnostic anchors in findings are resolved against the documented default +materialization. A strict materialization returns diagnostics retargeted to its +own tree. Do not reuse a tree path across policy shapes without resolution. + +## Sessions and cache lanes + +`createSession(source)` and `BasicSession` wrap one immutable source with +separate caches for: + +- outlines with and without diagnostics; +- full events with and without diagnostics; +- default tree; +- tree plus diagnostics; +- source-strict tree; +- recovery-aware result; +- findings. + +```ts +import { createSession } from "@okikio/wikitext"; + +const session = createSession(source); +const outline = Array.from(session.outline()); +const diagnostics = session.parseWithDiagnostics(); +const findings = session.analyze(); +``` + +The session is a cache wrapper, not a mutable incremental document. It does not +own edit application, revision rebasing, edit-stable anchors, async chunking, +or cross-document cache invalidation. Create a new session when source changes. + +Do not assume every lane shares one cache. The separation is intentional so a +cheap parse does not pay for diagnostics unless another requested lane has +already produced reusable work. + +## Filtering and diagnostics + +The reviewed filter surface includes generic `visit()` and `filter()`, focused +tree filters for templates, links, images, lists, tables, categories, and +references, plus `filterEvents()` and `collectEvents()` for streaming output. + +Diagnostic helpers include tree-path and anchor resolution. Keep three +authorities explicit: + +1. source range locates exact input; +2. diagnostic code drives machine behavior; +3. human-readable message explains the issue. + +Do not branch business logic on diagnostic messages. Resolve an anchor against +the tree policy and source revision it belongs to. + +For one-pass extraction over large input, prefer event filters. For multiple +queries or parent/child context, materialize once and reuse tree filters. + +## Ecosystem integration + +### unist and unified + +The tree is designed to be unist-compatible, so `unist-util-visit` and related +utilities may apply. Verify node naming, position semantics, and whether a +plugin assumes Markdown/HAST-specific nodes before reusing it. "Unist-shaped" +does not make every unified plugin semantically compatible. + +### Editor and LSP tooling + +- UTF-16 offsets align with common LSP coordinates, but line/column conversion + still needs a tested index; +- diagnostics and recoveries should remain visible rather than silently fixing + source; +- session caches are per immutable source, so edits require revision ownership; +- a quick outline can use block events while full inline diagnostics run later. + +### Indexers and analyzers + +- tokenize or stream events when ordering and ranges are enough; +- materialize only for repeated structural queries; +- persist the parser version with derived indexes if syntax behavior can change; +- keep source slices or content hashes so stored ranges remain auditable. + +### MediaWiki systems + +Parsing source is only one boundary. Template expansion, link resolution, +namespace rules, HTML rendering, sanitization, permissions, and remote fetches +belong to connected systems. Do not attribute their behavior to this package. + +## Unsupported and planned surfaces + +The reviewed README, manifest include list, or design documents mention +`stringify()` and `parseChunked()`, but current source has no `stringify.ts` and +the root module does not export either function. Lazy tree building is also +listed as in progress. + +This is an anti-hallucination boundary: + +```ts +// Do not write this against the reviewed revision. +import { parseChunked, stringify } from "@okikio/wikitext"; +``` + +Documentation intent and publish configuration do not override the actual +export map. If a later version adds these APIs, re-read implementation, tests, +and limitations before teaching their semantics. + +## Failure signatures + +| Symptom | Likely cause | Next inspection | +|---|---|---| +| Large extraction allocates excessively | Full AST built for a streaming query | Reclassify to tokens, outline, or events | +| Emoji makes locations drift | UTF-16 offsets treated as code points or bytes | Audit coordinate conversion | +| Linter silently accepts malformed structure | `parse()` discarded diagnostics | Use diagnostics, strict, recovery, or findings lane | +| Tree differs between tolerant and strict results | Recovery policy legitimately changes shape | Inspect diagnostics and structured recovery entries | +| Diagnostic path resolves to wrong node | Anchor reused across policies or source revisions | Resolve against matching tree and source | +| Repeated operations reparse content | Stateless wrappers used for one immutable source | Consider a session and measure | +| Updated text returns old results | Session reused after source edit | Create a new session; own revisions externally | +| Import for `stringify` fails | Planned surface mistaken for shipped export | Inspect `mod.ts` and target version | +| Template output lacks expanded content | Source parser mistaken for MediaWiki evaluator | Add an explicit expansion/resolution layer | +| Unified plugin corrupts nodes | Plugin assumes mdast/hast semantics | Add an adapter or use compatible utilities only | ## Verification -Test malformed and Unicode-heavy arbitrary input, complete source-range tiling, -event nesting, deterministic repeated sessions, strict/tolerant/recovery -differences, and memory/time at the selected lane. Keep experimental limitations -in public recommendations. +Test invariants at the selected layer: + +- tokens tile the expected source ranges without overlap or gaps where the + contract requires it; +- enter/exit events are properly nested and deterministic; +- outline and full-event results agree on block boundaries; +- source slices match UTF-16 ranges for ASCII, astral, combining, RTL, and + newline-heavy fixtures; +- arbitrary malformed input does not throw; +- diagnostics and recovery kinds are stable for committed fixtures; +- default, source-strict, and recovery-aware results differ only where policy + permits; +- findings can be materialized repeatedly without reparsing; +- session cold and warm lanes return equivalent results; +- a new source revision cannot accidentally reuse an old session; +- filters return the same logical matches as an explicit traversal; +- memory and latency are measured separately for tokens, outline events, full + events, and tree materialization; +- imports are checked against the actual root export map so planned APIs cannot + enter production code. + +For experimental adoption, pin a revision or exact version, keep a corpus of +real and pathological documents, and record the parser version with stored +diagnostics or indexes. Public claims should state the maturity boundary and +should not promise MediaWiki equivalence or round-trip serialization. + +## Sources and freshness + +- Attachment: `wikitext.zip/mod.ts`, `parse.ts`, `session.ts`, parser/tree/filter sources, README, architecture docs, and tests from `wikitext.zip`, inspected 2026-07-17. +- Registry identity: [`@okikio/wikitext` on JSR](https://jsr.io/@okikio/wikitext); the detailed API is grounded in the attached experimental `0.0.0` repository. + +This package is experimental and version-sensitive. `stringify()`, `parseChunked()`, lazy tree building, edit-stable sessions, and any API absent from `mod.ts` are unimplemented or unverified at the reviewed revision. diff --git a/skills/use-okikio/references/workflows.md b/skills/use-okikio/references/workflows.md index e1683ec..c8bcd58 100644 --- a/skills/use-okikio/references/workflows.md +++ b/skills/use-okikio/references/workflows.md @@ -1,33 +1,311 @@ -# Personal workflow platform patterns +# Okikio finance workflow platform: capabilities and limits -The reviewed finance utilities define workflow input, triggers, idempotency, -retry, concurrency, throttle, rate limit, debounce, batch, singleton, -cancellation, and observability, plus a control plane, durable store, PostgreSQL -store, queue, timelines, waits, signals, schedules, and worker loops. +Use this reference only when the consuming repository contains the retained/private `@utils/workflows` source or an explicitly verified successor. It is an ambitious service-owned workflow control plane with real implementation and real gaps. Do not describe it as a complete durable engine without repairing and testing the gaps below. -## Capability status +## Contents -Do not call the platform complete from those types. In the reviewed source: +- Status and authority +- Capability inventory +- Definitions and policy +- Store and PostgreSQL model +- Control plane +- Workers and queues +- Effect runtime adapters +- Waits, signals, schedules, cancellation, and replay +- API integration +- Known durability gaps +- Productionization sequence +- Test and failpoint matrix +- Executable verification +- Deliberate exclusions +- Sources and freshness -- an event dispatcher remains a `return 0` scaffold; -- documentation says current HTTP handlers still use the old control plane; -- execution insert, timeline append, and enqueue are separate operations; -- timeline sequence uses `existing.length + 1`; -- idempotency uses read then insert; -- queue claiming reads then conditionally updates; -- wait timeout resolution and resume enqueue can split across a crash. +## Status and authority -These are evidence of an ambitious design with real incomplete durability and -reachability, not a finished engine. +Observed identity: private/workspace `@utils/workflows` in the uploaded new/old finance repository. It is not established as a public JSR/npm package. Verify source path, export map, lockfile, and revision before using names. -## Reuse rule +The intended authority split is: -Reuse definitions or interfaces only after mapping their authority, transaction, -worker, engine, queue, and API path in the consuming repository. Prefer schemas -as structured-data sources where current code uses parallel interfaces. +```text +service endpoint/event/webhook/schedule + -> WorkflowControlPlane + -> WorkflowStore (PostgreSQL intended durable public facts) + -> ready queue / worker loops + -> WorkflowRuntimeAdapter (Effect workflow execution) +``` -Require atomic uniqueness/sequence/claims, transactions for related database -writes, idempotent cross-system effects, reconciliation, restart tests, worker -deployment, and operator repair before production claims. +PostgreSQL is intended to own product-facing execution/status/timeline/queue/wait/signal/schedule/cancellation facts. Effect workflow runtime owns execution semantics/result. Service modules own public routes, authentication, tenant/family authorization, and response copy. -Compose with `build-workflows` for the full durability evidence ladder. +Current status is mixed: + +- memory store/runtime paths and substantial control-plane tests exist; +- PostgreSQL store and workflow schema exist; +- worker ticks and forever loops exist; +- Effect memory adapter calls real registered workflow operations; +- SQL-backed Effect adapter is an explicit placeholder returning not-implemented errors; +- event dispatch worker returns zero because event fan-out is synchronous in control plane; +- several multi-write transitions have crash gaps; +- cron calendar/timezone expansion remains incomplete. + +## Capability inventory + +Observed definition/control concepts include: + +- object workflow input with schema validation; +- endpoint, event, webhook, schedule, delayed, and invoke triggers; +- idempotency and reuse-existing policy; +- retries/backoff; +- concurrency limits; +- throttle and rate limit; +- debounce and batch policy; +- singleton semantics; +- priority, queue, tags, cancellation capability; +- execution status and timeline; +- ready queue with leases, priority, availability, attempt, payload, error; +- events and event trigger matching; +- waits and signals; +- interval/delayed/cron schedule records; +- cancellation requests and targets; +- replay records/starts; +- memory/PostgreSQL stores; +- ready consumer, active run poller, scheduler, wait router, lease recovery; +- memory and placeholder SQL Effect runtime adapters; +- service-mountable endpoint helpers. + +This is an inventory of authored surfaces. Each feature must be classified independently at the evidence ladder: defined, stored, reachable, worker-executed, restart-safe, operator-repairable. + +## Definitions and policy + +Definitions accept service-owned input schemas and dynamic key/filter callbacks. Observed policy shapes include keys derived from workflow input/context for idempotency, concurrency, throttle, rate-limit, debounce, batch, or singleton behavior. + +Risks and requirements: + +- callbacks are code, so deployed workflow version owns their semantics; +- input must be stable object data suitable for persistence/replay; +- dynamic keys must not contain secrets and need bounded length/canonicalization; +- definition name/version must remain resolvable for in-flight state; +- admission policies need durable state if they must survive process restart; +- process-memory debounce/concurrency maps are not cross-replica durability; +- changes to retry/cancellation/trigger policy need compatibility rules. + +The retained control plane uses some process-local memory for policy bookkeeping. Inspect every policy before claiming multi-process enforcement. + +## Store and PostgreSQL model + +Observed tables cover workflow executions, timeline, ready queue, events, waits, signals, schedules, cancellation, and replay-related facts under a `workflows` schema. + +Store requirements before production: + +- unique idempotency key at the correct service/workflow/scope boundary; +- atomic execution + initial timeline + queue acceptance; +- atomic per-execution timeline sequence allocation; +- atomic/fenced queue claim and acknowledgement; +- conditional terminal/status transitions; +- atomic or repairable signal/wait/enqueue transition; +- atomic or repairable schedule occurrence publication; +- indexes for active runs, ready/expired leases, due waits/schedules, idempotency; +- migration/upgrade and retention policy; +- operator queries/repair transactions; +- explicit DB client lifetime. + +Observed gaps: + +- `createExecution()` checks idempotency, inserts execution, appends timeline records, then enqueues across separate calls; +- `appendTimeline()` selects existing entries and uses `existing.length + 1`; +- ready claim selects rows then conditionally updates each; owner-fenced acknowledgement needs verification; +- signal/wait/resume operations are several writes; +- wait timeout marks timed out before enqueue; +- cancellation request and targets may be separate writes depending on inspected path. + +Comments sometimes describe partial states as operator-repairable, but a repair loop/tool is not automatically present. Implement and test it. + +## Control plane + +Observed operations include starts/invokes, event recording/fan-out, status/timeline reads, cancellation, resume, signaling, replay, schedule/event interaction, and poll-to-store reconciliation. + +Start flow conceptually: + +```text +lookup definition + -> validate object input + -> compute runtime/deterministic execution ID + -> resolve idempotency and admission policy + -> construct execution, first timeline entries, optional queue row + -> store.createExecution(...) + -> return execution/status/cancel URLs +``` + +Production requirements: + +- acceptance is atomic or has an implemented reconciler; +- idempotency is protected by database constraint/atomic write; +- URL builders reflect the actual service-owned route; +- admission state is durable across replicas when promised; +- skipped/delayed states are explicit; +- no external runtime/effect starts before durable intent; +- poll reconciliation handles missing/running/suspended/success/failure/cancelled distinctly; +- public status never implies an external effect committed without domain evidence. + +## Workers and queues + +Observed tick functions: + +- `tickReadyQueueConsumer`; +- `tickActiveRunPoller`; +- `tickScheduler`; +- `tickWaitRouter`; +- `tickLeaseRecovery`; +- `tickEventDispatcher` (currently returns `0`). + +`createWorkflowWorkerLoops()` wraps ticks in forever loops for a supervisor. Prove that a deployed process actually boots/supervises these loops and exposes readiness/heartbeat. + +Queue dispatch updates execution to running, appends timeline, starts/resumes runtime, acknowledges row, and polls. A crash between runtime effect and ack can redeliver. Runtime start/resume must be idempotent by execution ID. A crash after marking running but before start needs reconciliation. + +Add/verify: + +- lease renewal for work exceeding lease duration; +- owner/version fencing on ack/dead/heartbeat; +- bounded attempts and backoff/requeue rather than immediate terminal failure for all errors; +- graceful shutdown; +- deployment/version compatibility; +- dead-letter operator redrive; +- metrics for queue age, claims, expiries, attempts, dead rows, active heartbeat. + +## Effect runtime adapters + +Observed `WorkflowRuntimeAdapter` surface includes: + +- list/register workflows; +- compute execution ID; +- start; +- poll; +- interrupt; +- resume. + +The memory adapter looks up registrations, validates object input, calls `workflow.executionId`, `workflow.execute(..., { discard: true })`, `workflow.poll`, `workflow.interrupt`, and `workflow.resume` through an injected runner. It is useful for tests/local wiring but is not process-loss durability. + +The `effect_sql` adapter currently returns `workflow_runtime_not_implemented` for every operation. Its name and stable seam are not capability evidence. Do not enable public start/status/cancel routes against it as though they work. + +`@effect/workflow` is version-sensitive/alpha in the broader research. Inspect exact installed imports, engine persistence, execution ID semantics, suspension, polling, interrupt/resume, schema evolution, and Layer requirements. Do not fabricate an SQL Layer or adapter API from memory. + +## Waits, signals, schedules, cancellation, and replay + +Wait/signal intended flow: + +```text +runtime suspends -> active wait row with matcher/resume token/timeout +signal endpoint -> record signal -> satisfy wait -> enqueue resume +timeout worker -> time out wait -> enqueue timeout resume +worker -> runtime resume -> poll/store projection +``` + +Require a single-winner rule for signal versus timeout versus cancellation. Conditional state transitions must prevent multiple resume rows, or deterministic occurrence identity must deduplicate them. + +Schedules: + +- interval next-run logic exists; +- delayed one-shot semantics exist; +- cron can be stored but calendar/timezone expansion is incomplete per retained README/code comments; +- two schedulers need atomic due occurrence claiming and deterministic occurrence IDs. + +Cancellation: + +- distinguish requested, cancelling, target applied, and final cancelled; +- pending work may be cancelled without runtime interrupt; +- started work needs actual runtime interrupt and poll reconciliation; +- cancellation can race terminal success/failure; +- external activities need cooperative cancellation/compensation policies. + +Replay should create a related new execution or an explicit replay lineage without rewriting old history. Side effects still require replay-safe activity identities. + +## API integration + +Service modules own routes, authentication, tenant/family authorization, and response shape. Workflow helpers can adapt control-plane operations, but generic workflow utilities must not infer product authorization. + +Conceptual integration: + +```ts +app.post('/organizations/:org/imports', authz, validate, async (c) => { + const accepted = await controlPlane.startWorkflow({ + workflow_name: 'ImportTransactions', + input: { organizationId: c.get('org').id, ...c.req.valid('json') }, + idempotency_key: c.req.header('Idempotency-Key'), + status_url: `/organizations/${org}/imports/${id}/status`, + }) + return c.json(toAccepted(accepted), 202) +}) +``` + +The example is conceptual; inspect actual helper signatures. Apply equivalent authz to status, timeline, signal, cancel, and replay. Avoid leaking existence across tenants. + +## Known durability gaps + +| Gap | Failure window | Required correction/proof | +|---|---|---| +| idempotency read then insert | concurrent duplicate starts | unique constraint/atomic insert | +| execution/timeline/queue separate | accepted orphan or incomplete history | transaction/outbox/reconciler | +| timeline `existing.length + 1` | sequence collision | atomic counter/lock/sequence constraint | +| select then conditional queue claim | concurrency/fairness/fencing questions | multi-worker oracle and owner-fenced updates | +| signal/wait/enqueue separate | signal accepted but no resume | transaction/outbox/reconciler | +| timeout status before enqueue | timed-out wait never resumed | transaction/outbox/reconciler | +| runtime call versus store status | running/no runtime or runtime/no status | idempotent start and reconciliation | +| SQL Effect adapter placeholder | no durable execution backend | implement verified Layer/adapter or disable | +| event dispatcher returns zero | async dispatch not implemented | document synchronous path or implement durable worker | +| cron expansion missing | stored cron never advances | implement versioned timezone parser or reject | +| process-local policy memory | cross-replica divergence | durable atomic policy state or narrow claim | + +## Productionization sequence + +1. Inventory which capabilities product routes actually need. +2. Disable/unregister unsupported paths and make readiness honest. +3. Add database constraints/indexes and transactional acceptance. +4. Implement repair scans/tools for every retained partial state. +5. Choose and implement the real durable runtime adapter at pinned versions. +6. Add owner-fenced leases, renewal, retry/dead/redrive, and worker supervision. +7. Repair waits/signals/schedules/cancellation atomicity. +8. Define versioning for definitions, inputs, checkpoints, and in-flight runs. +9. Run failpoint/restart tests against PostgreSQL and the real runtime. +10. Deploy workers separately/explicitly, expose health/readiness/metrics, and test operator controls. + +## Test and failpoint matrix + +Test: + +- definition/input validation and duplicate registration; +- concurrent idempotent starts across processes; +- crash after every execution/timeline/queue write; +- two-worker claim, lease expiry, old-owner ack, long-run heartbeat; +- runtime start before/after store status failpoints; +- process restart with memory versus durable adapter; +- signal versus timeout versus cancel race; +- crash after each signal/wait/resume write; +- two schedulers and duplicate due occurrence; +- interval/delayed and explicit rejection of unsupported cron; +- cancellation before start, during run, and racing success/failure; +- replay with external side-effect idempotency; +- event fan-out path and honest dispatcher status; +- deployment with queued old workflow version; +- worker shutdown and restart; +- operator repair/redrive audit; +- route authz for start/status/timeline/signal/cancel/replay. + +## Executable verification + +Use a disposable PostgreSQL instance and the actually selected Effect runtime. Run control-plane tests, then deterministic failpoint tests that kill and restart processes. Query workflow tables and call real service endpoints. Require convergence between runtime result and public execution projection, no lost accepted runs, idempotent duplicate delivery, safe wait/signal/schedule races, reachable worker loops, and idempotent operator repair. + +Memory-adapter tests cannot satisfy the durable-backend gate. If no real runtime is configured, record durable runtime verification as blocked and do not claim production readiness. + +## Deliberate exclusions + +- Do not claim `@utils/workflows` is public or installable without evidence. +- Do not call the current SQL Effect adapter implemented. +- Do not call memory behavior durable. +- Do not force this platform when Temporal, Effect Workflow, a simpler job store, or another selected owner is more appropriate. +- Do not infer atomicity from one store interface method when its implementation performs multiple writes. +- Do not expose unsupported cron/cancel/replay behavior as successful. +- Do not assume workflow durability makes external effects idempotent. +- Do not invent APIs beyond inspected exports/source. + +## Sources and freshness + +Grounded in the complete retained finance `utils/workflows` README, definitions, registration, types, control plane, store/memory/PostgreSQL implementations, endpoints, events, worker boot/loops, runtime types/memory/effect-sql adapters, workflow database schema/migration, tests and benchmarks, reviewed 2026-07-17. This is private/workspace source evidence. `@effect/workflow` and Effect runtime APIs are version-sensitive and require installed-source verification.