From 4178a5ce0cde5720f2fb141e5d615d437635fe40 Mon Sep 17 00:00:00 2001 From: Okiki Ojo Date: Thu, 13 Aug 2026 02:55:18 -0400 Subject: [PATCH] refactor: remove outdated media architecture overview and replace with comprehensive design documentation feat: add ecosystem integrations documentation for unstorage, RxDB, db0, and Drizzle docs: create environments documentation detailing execution contexts for OPFS in various runtimes chore: delete obsolete media terms documentation docs: introduce sources documentation outlining research and source register for standards and integrations chore: remove package testing documentation as it is no longer relevant feat: add validation strategy documentation for TypeScript targets and validation commands Signed-off-by: Okiki Ojo --- AGENTS.md | 24 +- docs/adapters.md | 239 +++++++++++++++++++ docs/api.md | 428 ++++++++++++++++++++++++++++++++++ docs/architecture/overview.md | 24 -- docs/design.md | 339 +++++++++++++++++++++++++++ docs/ecosystems.md | 274 ++++++++++++++++++++++ docs/environments.md | 177 ++++++++++++++ docs/media/terms.md | 12 - docs/sources.md | 165 +++++++++++++ docs/testing/packages.md | 8 - docs/validation.md | 156 +++++++++++++ 11 files changed, 1791 insertions(+), 55 deletions(-) create mode 100644 docs/adapters.md create mode 100644 docs/api.md delete mode 100644 docs/architecture/overview.md create mode 100644 docs/design.md create mode 100644 docs/ecosystems.md create mode 100644 docs/environments.md delete mode 100644 docs/media/terms.md create mode 100644 docs/sources.md delete mode 100644 docs/testing/packages.md create mode 100644 docs/validation.md diff --git a/AGENTS.md b/AGENTS.md index 7e579fd..3adf43d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,17 +1,19 @@ # Repository implementation rules -- Preserve the copied `utils/` programming model unless a deliberate refactor is requested. -- Put concrete media capabilities in `packages/media/`. -- Prefer one-word names. Use two or three words only when needed for clarity. +- Treat `@okikio/opfs` as a library programming model, not as an application runtime. +- Keep the root entrypoint import-safe in Window, Worker, Deno, Bun, and Node contexts. +- Put concrete storage integrations under `src/adapter/` and expose them through explicit public subpaths. +- Put reverse ecosystem interfaces under `src/driver/`. +- Prefer one-word file and folder names. Use more words only when the precise concept requires them. - All Zod schema constants end in `Schema`. - Project-owned data types normally end in `Type`. - Prefer direct schema and type exports. Use namespace imports only when they improve short operation call sites. -- Prefer `get`, `create`, `open`, `save`, `inspect`, `plan`, `convert`, `download`, `select`, `write`, `close`, `pause`, `resume`, and `cancel` over vague verbs. +- Prefer `get`, `create`, `open`, `save`, `inspect`, `plan`, `convert`, `read`, `write`, `close`, `remove`, `copy`, and `move` over vague verbs. - Avoid `generate`, `execute`, `handle`, `process`, `manager`, `helper`, `common`, `shared`, and `misc` unless an external protocol requires the word. -- Use `node:test` with `describe` and `it`; use `@std/expect` for expectations. -- Keep Deno and Node on the same TypeScript source. Do not create runtime-specific source forks. -- Prefer Web APIs and `@std/*`; use the existing utilities when their stronger programming model is required. -- Use LogTape categories in reusable packages. Applications own LogTape configuration. -- Use `@okikio/observables` for observation, not as authority for cancellation or terminal results. -- TSDoc and comments use plain Simplified Technical English and teach options, examples, impact, reasoning, ownership, limits, and necessary background. -- Do not add a root `scripts/` directory. Put repository tasks under `.mise/tasks/`. +- Keep Deno, Bun, Node, browsers, and Workers on the same core TypeScript source. Runtime-specific adapters can use runtime-specific APIs behind explicit subpaths. +- Prefer Web APIs and existing standard-library capabilities before custom infrastructure. +- Adapters never configure logging, read environment variables, or acquire unrelated global resources at import time. +- The caller owns injected database, collection, storage, and filesystem resources unless an adapter option explicitly transfers ownership. +- TSDoc and comments use plain technical English. They teach options, examples, impact, reasoning, ownership, limits, failure behavior, and necessary background. +- Document public schemas, types, properties, functions, classes, and adapter contracts. Document internal symbols when their invariant, lifecycle, or failure behavior is not obvious. +- Comments explain why a rule exists or what must remain true. Do not restate obvious syntax. diff --git a/docs/adapters.md b/docs/adapters.md new file mode 100644 index 0000000..22ebf70 --- /dev/null +++ b/docs/adapters.md @@ -0,0 +1,239 @@ +Adapter guide +============= + +An adapter translates canonical virtual filesystem operations into one backend. This document describes the included adapters and the contract for new ones. + +Use `createFileSystem()` for every adapter: + +```ts +import { createFileSystem } from "@okikio/opfs"; + +const fileSystem = createFileSystem(adapter, { + coordination: "auto", + maxBufferedWriteBytes: 64 * 1024 * 1024, +}); +``` + +Included adapters +----------------- + +| Public subpath | Backend | Main use | +| --- | --- | --- | +| `adapter/opfs` | browser OPFS root | native browser persistence | +| `adapter/deno` | `Deno.*` file APIs | Deno services and CLIs | +| `adapter/bun` | Bun + Bun's Node-compatible fs APIs | Bun services and CLIs | +| `adapter/node` | `node:fs` | Node services, Electron main process | +| `adapter/memory` | in-memory record map | tests, examples, temporary state | +| `adapter/record` | generic `RecordStoreType` | build a new value/document/SQL adapter | +| `adapter/unstorage` | unstorage `Storage` | use any compatible unstorage mount as filesystem persistence | +| `adapter/rxdb` | RxDB `RxCollection` | use RxDB and its selected RxStorage | +| `adapter/db0` | db0 `Database` | use db0 connector/dialect infrastructure | +| `adapter/drizzle` | Drizzle database + table | use an existing Drizzle schema/driver | + +### OPFS + +```ts +import { openFileSystem } from "@okikio/opfs"; +``` + +or explicitly: + +```ts +import { createFileSystem } from "@okikio/opfs"; +import { createOpfsAdapter } from "@okikio/opfs/adapter/opfs"; + +const root = await navigator.storage.getDirectory(); +const fileSystem = createFileSystem(createOpfsAdapter(root)); +``` + +The explicit adapter retains `nativeRoot` for advanced browser interop. Synchronous access is exposed only when the current native file handle actually provides `createSyncAccessHandle()`. + +The adapter does not attempt browser or incognito detection. + +### Deno + +```ts +import { createFileSystem } from "@okikio/opfs"; +import { createDenoAdapter } from "@okikio/opfs/adapter/deno"; + +const fileSystem = createFileSystem( + createDenoAdapter({ root: "./data" }), + { coordination: "local" }, +); +``` + +`root` is the host directory represented by virtual `/`. `createRoot` defaults to true. + +The adapter uses Deno filesystem APIs for data operations, rename, sync access, and flush. It uses Node's path compatibility module only to normalize the configured host root and to verify that a virtual path stays below it. + +### Bun + +```ts +import { createBunAdapter } from "@okikio/opfs/adapter/bun"; + +const adapter = createBunAdapter({ root: "./data" }); +``` + +The replace/read fast path uses Bun file APIs. Directory operations, update-mode writes, native rename, and synchronous random access use Bun's Node-compatible filesystem APIs. + +The module resolves `Bun` lazily during adapter creation. Importing the module does not require the Bun global to exist. + +### Node + +```ts +import { createNodeAdapter } from "@okikio/opfs/adapter/node"; + +const adapter = createNodeAdapter({ root: "./data" }); +``` + +Node supports native streaming reads/writes, byte ranges, rename, and synchronous random access. + +The host root is created by default. The virtual path mapper rejects any resolved host path that would leave that root. + +### Memory + +```ts +import { createMemoryAdapter } from "@okikio/opfs/adapter/memory"; +``` + +The memory adapter uses the same record-store layer as database adapters. It is intentionally deterministic and dependency-free. It is suitable for tests and temporary state, not durable storage. + +The companion `createMemoryRecordStore()` is useful when testing another record-store wrapper directly. + +RecordStoreType +--------------- + +Use `RecordStoreType` when the backend is fundamentally value-based instead of filesystem-based. + +```ts +import { + createRecordAdapter, + type RecordStoreType, +} from "@okikio/opfs/adapter/record"; + +const store: RecordStoreType = { + async get(path) { /* ... */ }, + async set(record) { /* ... */ }, + async delete(path) { /* ... */ }, + async *list(parent) { /* direct children only */ }, +}; + +const adapter = createRecordAdapter(store, { + name: "my-store", +}); +``` + +Important contract rules: + +- `get()` returns one validated logical path. +- `set()` replaces one logical record as atomically as the provider permits. +- `delete()` removes one record only. Recursive behavior belongs to the filesystem facade. +- `list(parent)` yields direct children only. +- The store receives canonical paths. +- The store can expose `dispose()` for resources it explicitly owns. +- Use `readOnly: true` when the storage can read but cannot mutate. + +Record adapters do not claim native streaming. Their stream inputs are materialized under `maxBufferedWriteBytes`. + +Custom AdapterType +------------------ + +Use `AdapterType` directly when the backend can expose real file-like primitives. + +```ts +import { + defineAdapter, + type AdapterType, +} from "@okikio/opfs/adapter"; + +export const adapter = defineAdapter({ + name: "provider", + capabilities: { + read: true, + write: true, + streamRead: false, + streamWrite: false, + rangeRead: true, + nativeMove: false, + syncAccess: false, + }, + + async stat(path, options) { /* ... */ }, + async readFile(path, options) { /* ... */ }, + async writeFile(path, bytes, options) { /* ... */ }, + async *readDir(path, options) { /* direct children */ }, + async createDir(path, options) { /* parent already exists */ }, + async remove(path, options) { /* file or empty directory */ }, +}); +``` + +`defineAdapter()` validates the adapter name and capability object at runtime. It does not register the adapter globally. + +Required adapter semantics +-------------------------- + +`stat` +: Return `null` only for not-found. A file stat includes size, last-modified milliseconds, and a media type string. + +`readFile` +: Respect optional byte offset and maximum length. Return the bytes actually read. + +`writeFile` +: Preserve `replace`, `append`, and `update` semantics. Respect `truncate` at the final write cursor. + +`readDir` +: Yield direct child names and kinds lazily. Do not recursively traverse here. + +`createDir` +: Create exactly one directory. The parent already exists when the facade calls this primitive. + +`remove` +: Remove one file or one empty directory. Recursive behavior belongs to the facade. + +Optional native operations +-------------------------- + +Only advertise a capability when the adapter implements the corresponding native method. + +```text +streamRead -> openReadStream +streamWrite -> writeStream +nativeMove -> move +syncAccess -> openSyncFile +``` + +`rangeRead` describes whether the adapter can avoid materializing the complete file for a range. The facade still exposes ranged `readFile()` to all adapters. + +Cancellation +------------ + +Every async operation that accepts an `AbortSignal` should check it before expensive work and between long-running chunks. When a stream write fails or aborts, cancel the source producer when possible so upstream work does not continue after the file operation is terminal. + +Errors +------ + +Adapters can throw native errors. The facade maps known browser and server error shapes through `toFileSystemError()`. + +If an adapter itself must create a package error, use `FileSystemError` with a precise operation and canonical path. Do not return a boolean for exceptional filesystem states. + +Ownership +--------- + +Adapters borrow injected resources unless their options explicitly transfer ownership. + +Good: + +```ts +createUnstorageAdapter(storage, { disposeStorage: true }); +createDb0Adapter(database, { disposeDatabase: true }); +createFileSystem(adapter, { disposeAdapter: true }); +``` + +Avoid an adapter that always disposes a resource supplied by the caller. + +Import safety +------------- + +A concrete adapter subpath can depend on its runtime, but importing unrelated entrypoints must not pull that runtime into the graph. + +Do not export server adapters from the root module. Do not probe environment variables, configure logs, connect to providers, or start workers at module evaluation time. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..0913603 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,428 @@ +Public API guide +================ + +This guide is organized by developer task. Exact low-level schemas and types are also available through the explicit package subpaths. + +Open or create a filesystem +--------------------------- + +### `openFileSystem(options?)` + +Opens the browser's native Origin Private File System and returns `FileSystemType`. + +```ts +import { openFileSystem } from "@okikio/opfs"; + +const fileSystem = await openFileSystem(); +``` + +Use this only when native browser OPFS is the chosen backend. Server runtimes should create a runtime adapter and pass it to `createFileSystem()`. + +### `createFileSystem(adapter, options?)` + +Creates the adapter-independent facade. + +```ts +const fileSystem = createFileSystem(adapter, { + coordination: "auto", + lockPrefix: "my-app:filesystem", + maxBufferedWriteBytes: 64 * 1024 * 1024, + disposeAdapter: false, +}); +``` + +`FileSystemOptionsType`: + +- `coordination`: `auto`, `web-locks`, `local`, or `none`. +- `lockPrefix`: stable lock namespace used for cooperating filesystem facades. +- `maxBufferedWriteBytes`: maximum stream size materialized for non-streaming adapters. +- `disposeAdapter`: transfers adapter disposal ownership to the facade when true. + +`coordination` is runtime-validated by `CoordinationModeSchema`. + +Path API +-------- + +### `getDirectoryHandle(path, options?)` + +Returns a package `DirectoryHandleType` for one directory. + +Options: + +- `create`: create exactly that directory when absent. +- `recursive`: create missing ancestors as well. +- `signal`: abort before commit. + +The virtual root `/` always exists. + +### `getFileHandle(path, options?)` + +Returns a package `FileHandleType`. + +Options: + +- `create`: create the file when absent. +- `parents`: create missing parent directories. +- `signal`: abort before commit. + +A read-only lookup never creates a file or directory. + +### `getFile(path, options?)` + +Returns a `File` snapshot. Changes written later are not reflected in the already-returned File object. + +### `stat(path, options?)` + +Returns: + +```ts +type StatType = FileStatType | DirectoryStatType; +``` + +File stat includes canonical path, name, size, last-modified milliseconds, and media type. Directory stat includes canonical path, name, and last-modified when the adapter provides it. + +### `exists(path, options?)` + +Returns an advisory boolean. `kind` can restrict the answer to `file` or `directory`. + +Do not use `exists()` as a substitute for operation error handling. Another context can mutate the backend after the check. + +### `mkdir(path, options?)` + +Creates one directory. `recursive: true` creates missing ancestors. + +### `ensureDir(path, options?)` + +Ensures a directory and its parents exist. A file at the same path produces `type-mismatch`. + +### `ensureFile(path, options?)` + +Ensures an empty file exists and creates its parent directories. + +Read APIs +--------- + +### `readFile(path, options?)` + +Returns `Uint8Array`. + +Options: + +- `at`: zero-based byte offset. +- `length`: maximum bytes after `at`. +- `signal`: cancellation. + +### `readText(path, options?)` + +Reads bytes and decodes them. `encoding` defaults to UTF-8. + +### `openReadStream(path, options?)` + +Returns `ReadableStream`. + +If the adapter provides native stream reads, the facade forwards them. Otherwise it creates a stream from the adapter's materialized read result. Cancellation remains connected after the stream opens. + +Write API +--------- + +### `writeFile(path, data, options?)` + +Accepted `WriteDataType` values: + +```text +string +Blob +ArrayBuffer +ArrayBufferView +ReadableStream +AsyncIterable +``` + +Options: + +- `mode`: `replace` (default), `append`, or `update`. +- `at`: starting byte offset for update mode. +- `truncate`: truncate at the final cursor. +- `parents`: create missing parents. +- `mediaType`: metadata for record/native adapters that can preserve it. +- `signal`: cancellation. + +The mode is runtime-validated by `WriteModeSchema`. + +A non-streaming adapter buffers stream input up to `maxBufferedWriteBytes`. Crossing the limit cancels the producer and throws `too-large`. + +Directory iteration +------------------- + +### `readDir(path, options?)` + +Lazy direct-child iterator. + +```ts +for await (const entry of fileSystem.readDir("/projects")) { + console.log(entry.kind, entry.name, entry.path); +} +``` + +### `walk(path, options?)` + +Lazy recursive iterator. + +Options: + +- `maxDepth`: maximum depth below the requested path. +- `includeRoot`: include the requested path itself before descendants. +- `includeFiles`: yield files. Defaults to true. +- `includeDirectories`: yield directories. Defaults to true. +- `signal`: cancel traversal between yielded entries. + +The iterator does not eagerly collect the entire tree. + +Structural operations +--------------------- + +### `copy(source, destination, options?)` + +Copies one file or tree. Directory file bodies use bounded `concurrency`, default 4. + +`overwrite: true` replaces the destination tree instead of merging stale entries into it. + +Source and destination cannot be the same path or ancestors of each other. + +### `move(source, destination, options?)` + +Uses adapter-native move when `nativeMove` is true. Otherwise calls copy then remove. The fallback is not atomic. + +### `remove(path, options?)` + +Removes one file or empty directory. `recursive: true` removes descendants first. + +The virtual root cannot be removed. + +### `emptyDir(path?, options?)` + +Removes children while retaining the directory. `path` defaults to `/`. Child removals use bounded concurrency. + +Synchronous file API +-------------------- + +### `openSyncFile(path, options?)` + +Returns `SyncFileType` only when `adapter.capabilities.syncAccess` is true. + +Options: + +- `create` +- `parents` +- `signal` + +The resource owns its native file and the facade path lock for the complete lifetime. + +```ts +const file = await fileSystem.openSyncFile("/db.sqlite", { + create: true, + parents: true, +}); + +try { + file.writeAll(bytes, { at: 0 }); + file.flush(); +} finally { + file.close(); +} +``` + +`SyncFileType` operations: + +```text +read +write +writeAll +getSize +truncate +flush +close +``` + +`writeAll()` loops over partial native writes. + +OPFS-shaped handle API +---------------------- + +Every `FileSystemType` has `root: DirectoryHandleType`. + +### Directory handle + +```text +kind +name +path +getDirectoryHandle() +getFileHandle() +removeEntry() +resolve() +entries() +keys() +values() +isSameEntry() +[Symbol.asyncIterator]() +``` + +### File handle + +```text +kind +name +path +getFile() +createWritable() +createSyncAccessHandle() +isSameEntry() +``` + +These are package facades, not native browser handle instances. Their `path` property is package-specific. + +### `createWritable()` + +Returns `WritableFileStreamType`. The staged image commits on close and discards on abort. + +Supported write commands: + +```ts +await writable.write(data); +await writable.write({ type: "write", position: 10, data }); +await writable.write({ type: "seek", position: 20 }); +await writable.write({ type: "truncate", size: 100 }); +``` + +Blob also has a `type` property, so the implementation identifies a command only when `type` is exactly `write`, `seek`, or `truncate`. + +Adapter API +----------- + +`@okikio/opfs/adapter` exports the complete backend contract: + +- `AdapterSignalOptionsType` +- `AdapterReadOptionsType` +- `AdapterWriteOptionsType` +- `AdapterMoveOptionsType` +- `AdapterDirectoryEntryType` +- `AdapterFileStatType` +- `AdapterDirectoryStatType` +- `AdapterStatType` +- `AdapterSyncFileType` +- `AdapterType` +- `FileSystemOptionsType` +- `defineAdapter()` + +`defineAdapter()` validates `AdapterNameSchema` and `AdapterCapabilitiesSchema` without adding a registry or global mutation. Adapter methods always receive canonical virtual paths. See [adapters.md](./adapters.md) for every primitive and first-party adapter. + +Record API +---------- + +`@okikio/opfs/adapter/record` exports: + +- `RecordStoreType` +- `RecordAdapterOptionsType` +- `createRecordAdapter()` + +`@okikio/opfs/schema` exports the validated persistence schemas: + +- `PathSchema` / `PathType` +- `AdapterNameSchema` / `AdapterNameType` +- `EntryKindSchema` / `EntryKindType` +- `OpfsContextSchema` / `OpfsContextType` +- `CoordinationModeSchema` / `CoordinationModeType` +- `WriteModeSchema` / `WriteModeType` +- `AdapterCapabilitiesSchema` / `AdapterCapabilitiesType` +- `ErrorCodeSchema` / `ErrorCodeType` +- `RecordVersionSchema` / `RecordVersionType` +- `DirectoryRecordSchema` / `DirectoryRecordType` +- `FileRecordSchema` / `FileRecordType` +- `RecordSchema` / `RecordType` +- `Db0DialectSchema` / `Db0DialectType` +- `SqlIdentifierSchema` / `SqlIdentifierType` + +Path utility API +---------------- + +`@okikio/opfs/path` exposes the canonical virtual-path model used by adapters: + +- `ROOT_PATH`: the canonical `/` root. +- `normalizePath(path)`: resolves `.`, `..`, duplicate separators, and relative input while rejecting root escape, backslashes, and NUL. +- `splitPath(path)`: returns canonical path segments without `/`. +- `joinPath(...parts)`: joins inputs and returns a canonical `PathType`. +- `dirname(path)`: returns the canonical parent path. +- `basename(path)`: returns the final name. Root returns an empty string. +- `isAncestorPath(ancestor, path)`: tests strict ancestry after normalization. +- `validateName(name)`: validates one direct File System API child name. +- `PathType`: validated canonical virtual path type. + +Use the high-level filesystem methods for normal application work. These helpers are primarily for adapters, drivers, and code that persists canonical paths. + +Error API +--------- + +The root module exports `FileSystemError`, `getErrorName()`, `getErrorMessage()`, and `toFileSystemError()`. + +`FileSystemError` carries: + +```text +code stable ErrorCodeType +operation filesystem operation that failed +path canonical path when one exists +cause original runtime/provider failure when retained +``` + +`toFileSystemError()` maps known DOMException names and server error codes such as `ENOENT`, `EEXIST`, and quota/permission failures into the stable package categories. Unknown provider failures remain `unknown` and retain the original cause. + +`getErrorName()` and `getErrorMessage()` are safe extraction helpers for diagnostics where the caught value is `unknown`. + +Browser capability APIs +----------------------- + +### `probeOpfs()` + +Returns a non-throwing `OpfsCapabilitiesType` report with: + +- execution context; +- root availability and normalized root error; +- embedded/same-origin-top facts when observable; +- Web Locks availability; +- sync access exposure; +- storage estimate when available; +- persistence status when available. + +It does not report `isIncognito` or `isPrivate`. + +### `getOpfsContext()` + +Classifies the current browser execution context as window, dedicated worker, shared worker, service worker, generic worker, or unknown. + +### iframe subpath + +`@okikio/opfs/iframe` exports: + +- `supportsUnpartitionedOpfsRequest()` +- `requestUnpartitionedFileSystem()` + +The request is explicit because browser permission/user-activation requirements must remain under application control. + +Lifecycle +--------- + +`FileSystemType` implements `AsyncDisposable`. + +```ts +await fileSystem.close(); +``` + +or with supported explicit resource management syntax: + +```ts +await using fileSystem = createFileSystem(adapter, { + disposeAdapter: true, +}); +``` + +Closing is idempotent. The adapter is closed only when ownership was explicitly transferred. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md deleted file mode 100644 index f791ca5..0000000 --- a/docs/architecture/overview.md +++ /dev/null @@ -1,24 +0,0 @@ -Media architecture -================== - -The media domain is decomposed into focused packages. A package owns one -capability and exposes an intentional API. Generic programming models remain in -`utils/`. - -```text -@media/convert - |-- @media/source - |-- @media/inspect - |-- @media/plan - |-- @media/output - `-- @media/task - -@media/download - |-- @media/source - |-- @media/output - `-- @media/task -``` - -The initial package files establish contracts and independently testable -operations. The application layer will be added after the library behavior and -performance characteristics are stable. diff --git a/docs/design.md b/docs/design.md new file mode 100644 index 0000000..0c4bff2 --- /dev/null +++ b/docs/design.md @@ -0,0 +1,339 @@ +Architecture and invariants +=========================== + +This document explains why `@okikio/opfs` has two frontend styles, one adapter contract, and a second record-store contract. + +The core rule is simple: + +> Filesystem semantics belong to the filesystem facade. Persistence mechanics belong to the adapter. + +That rule keeps OPFS-style application code reusable without flattening meaningful differences between a browser filesystem, a host filesystem, a key-value store, a document collection, and a SQL database. + +The complete data path +---------------------- + +```text + application + | + +------------+------------+ + | | + v v + path methods OPFS-shaped handles + readFile/writeFile FileHandle/DirectoryHandle + | | + +------------+------------+ + | + v + FileSystemType + | + path normalization / errors / cancellation + recursive copy / move / walk / remove + lock ownership / sync-file lifetime + stream fallback / buffer limit + | + v + AdapterType + | + +---------------+---------------+ + | | + v v + native adapters RecordStoreType + OPFS/Deno/Bun/Node | + | + +------------------+------------------+ + | | | + v v v + unstorage RxDB SQL rows + | + +------+------+ + | | + v v + db0 Drizzle +``` + +Why `AdapterType` is small +-------------------------- + +The required primitive set is deliberately smaller than the public filesystem API: + +```text +stat +readFile +writeFile +readDir +createDir +remove +``` + +Optional native capabilities add faster or stronger paths: + +```text +openReadStream +writeStream +move +openSyncFile +``` + +The facade builds higher-level behavior from these primitives. A custom backend therefore does not need to implement recursive copy, recursive remove, parent creation, OPFS-shaped handles, or lock orchestration independently. + +This avoids a common adapter anti-pattern where every backend reimplements the full public API and gradually develops different semantics. + +Capability flags describe native behavior +----------------------------------------- + +`AdapterCapabilitiesSchema` contains: + +```text +read +write +streamRead +streamWrite +rangeRead +nativeMove +syncAccess +``` + +These values describe what the adapter itself can do. They do not describe everything the facade can emulate. + +For example, a record-store adapter reports `streamWrite: false`. The facade can still accept a `ReadableStream`, but it must buffer the stream before storing the record. The capability remains false because pretending that buffering is native streaming would hide an important memory and latency difference. + +The record-store layer +---------------------- + +Value stores, document stores, and SQL databases do not naturally expose files and directories. `RecordStoreType` is the reusable translation point for those systems. + +```ts +interface RecordStoreType { + get(path): Promise; + set(record): Promise; + delete(path): Promise; + list(parent): AsyncIterableIterator; + dispose?(): void | Promise; +} +``` + +The shared record is versioned and validated by Zod. + +Directory: + +```json +{ + "version": 1, + "path": "/projects", + "parent": "/", + "name": "projects", + "kind": "directory", + "lastModified": 1786550000000 +} +``` + +File: + +```json +{ + "version": 1, + "path": "/projects/state.bin", + "parent": "/projects", + "name": "state.bin", + "kind": "file", + "data": "AAECAwQ=", + "size": 5, + "lastModified": 1786550000000, + "mediaType": "application/octet-stream" +} +``` + +`path` is the durable logical identity. `parent` is stored independently because directory listing should not require parsing every stored path. Backends are free to index `parent` in the way that best fits the provider. + +File bytes are base64. The choice is not an assertion that base64 is the most storage-efficient format. It is the common representation that survives JSON, RxDB documents, unstorage values, and SQL text columns without backend-specific binary contracts. Native filesystem adapters do not pay this cost. + +Streaming policy +---------------- + +Native adapters stream when the backend gives a real streaming primitive. + +Record-backed adapters materialize one file record. A streamed input therefore passes through this sequence: + +```text +ReadableStream / AsyncIterable + | + v + bounded byte collector + | + +-------+-------+ + | | + under limit over limit + | | + v v + RecordStore.set cancel source + throw too-large +``` + +`maxBufferedWriteBytes` defaults to 64 MiB. The limit is part of `FileSystemOptionsType`, not a hidden constant inside each database adapter, so the application can choose a memory policy once. + +Path invariant +-------------- + +Every adapter receives canonical virtual paths. + +Valid: + +```text +/ +/a +/a/b.txt +``` + +Rejected at the canonical adapter seam: + +```text +a/b +/a/ +/a//b +/a/./b +/a/../b +/a\b +``` + +Public path APIs may accept relative or non-canonical input. `normalizePath()` resolves it before the adapter sees it. + +The virtual path namespace is not a host path namespace. `createLocalPath()` maps virtual paths below one configured host root and verifies that the result does not escape that root. + +Handle invariant +---------------- + +`FileHandle` and `DirectoryHandle` are facades. They are not native `FileSystemHandle` objects and they do not claim to be. + +The facades preserve the useful OPFS programming shape: + +```text +root.getFileHandle() +root.getDirectoryHandle() +file.getFile() +file.createWritable() +file.createSyncAccessHandle() +directory.removeEntry() +directory.resolve() +entries()/keys()/values() +``` + +They also expose a package-specific canonical `path` property because the adapter architecture needs a stable logical address. + +`createWritable()` stages an in-memory file image and commits only on close. Abort discards the staged image. This mirrors the commit-on-close behavior an application expects from the File System API, but it is intentionally not the recommended large-file path. Large sequential writes should use `FileSystemType.writeFile()` so a streaming-capable adapter can bypass the staged image. + +Coordination invariant +---------------------- + +There are two classes of mutation. + +File mutation: + +```text +shared tree lock + | +exclusive /path/to/file lock + | +write or sync file lifetime +``` + +Structural mutation: + +```text +exclusive tree lock + | +copy / move / recursive remove / emptyDir +``` + +This lets independent files make progress at the same time while ensuring that a recursive tree mutation cannot race an active library file mutation. + +`local` coordination shares lock state by lock name inside one JavaScript realm. New readers queue behind an already-waiting exclusive request so writers do not starve. + +`web-locks` uses the browser Web Locks API. `auto` selects Web Locks when present and local FIFO locks otherwise. `none` retains cancellation checks but does not coordinate mutations. + +The adapter still owns any stronger backend-level locking. Library locks are application-level coordination for callers that use this library. + +Synchronous file lifecycle +-------------------------- + +A synchronous file has two resources with one lifetime: + +```text +facade path lock <------ same lifetime ------> adapter sync file + | | + +---------------- close() ----------------+ +``` + +`ManagedSyncFile` keeps the path lock until the native resource closes. This prevents an async write through the same facade from entering while synchronous random access is active. + +`writeAll()` must handle partial writes. It repeats the write until the complete input is committed or the backend reports no progress. + +Move semantics +-------------- + +Adapters with a native rename/move set `nativeMove: true` and provide `move()`. + +```text +Deno/Bun/Node +source -------- native rename --------> destination +``` + +Adapters without that primitive use: + +```text +source ---- copy ----> destination + | + +---- remove source after successful copy +``` + +The second sequence is not atomic. A failure between copy and remove can leave both entries. The API and documentation state this rather than presenting every backend as a POSIX filesystem. + +Before either form, source and destination are checked for ancestor overlap. An overwrite never removes an ancestor or descendant containing the source. + +Resource ownership +------------------ + +Injected resources are borrowed by default. + +```text +caller creates resource + | + +----> adapter borrows resource + | | + | +---- filesystem closes + | +---- resource stays open + | + +---- caller still owns resource +``` + +Ownership changes only through an explicit option: + +```text +disposeAdapter +disposeStorage +disposeDatabase +disposeFileSystem +``` + +This rule matters for connection pools, shared RxDB collections, process-wide unstorage instances, and server databases. A library adapter must not quietly dispose infrastructure that another subsystem still owns. + +Error invariant +--------------- + +Backends fail differently. Browsers use DOMException names. Node commonly reports `error.code`. Database bridges can throw provider errors. + +`toFileSystemError()` normalizes known failures to stable categories while retaining the original `cause`. The package does not erase unexpected backend failures into one generic string. + +Adapter import invariant +------------------------ + +The root package is import-safe for browsers. Runtime-specific code remains behind explicit subpaths. + +```text +@okikio/opfs browser-safe core + native OPFS +@okikio/opfs/adapter/node node:fs imports +@okikio/opfs/adapter/deno Deno globals +@okikio/opfs/adapter/bun Bun globals + Node compatibility APIs +@okikio/opfs/adapter/drizzle optional drizzle-orm peer +``` + +No adapter configures logging, reads environment variables, connects to a database, or mutates global application state merely because the module was imported. diff --git a/docs/ecosystems.md b/docs/ecosystems.md new file mode 100644 index 0000000..aa6192a --- /dev/null +++ b/docs/ecosystems.md @@ -0,0 +1,274 @@ +Ecosystem integrations +====================== + +The package integrates at the highest stable storage abstraction each ecosystem already provides. This is intentional. Reimplementing every upstream driver inside `@okikio/opfs` would duplicate provider code and create a second compatibility matrix that would immediately drift. + +```text +unstorage: Storage -> RecordStoreType -> AdapterType +RxDB: RxCollection -> RecordStoreType -> AdapterType +db0: Database -> RecordStoreType -> AdapterType +Drizzle: Database+Table -> RecordStoreType -> AdapterType +``` + +The filesystem semantics above those bridges are identical. + +unstorage +--------- + +`createUnstorageAdapter(storage)` accepts the high-level unstorage `Storage` contract. It uses: + +```text +getItem +setItem +removeItem +getKeys +optional dispose +``` + +This means the bridge is independent of the mounted driver. + +As reviewed on 2026-08-12, unstorage's generated built-in driver catalog includes these families: + +- Azure App Configuration, Cosmos, Key Vault, Storage Blob, and Storage Table +- Capacitor Preferences +- Cloudflare Cache, KV binding/HTTP, and R2 +- db0 +- Deno KV and Deno KV Node +- fs and fs-lite +- GitHub +- HTTP +- IndexedDB +- localStorage and sessionStorage +- LRU cache and memory +- MongoDB +- Netlify Blobs +- null and overlay +- PlanetScale +- Redis +- S3 +- UploadThing +- Upstash +- Vercel Blob and Vercel Runtime Cache + +The list is upstream inventory, not a claim that every provider has filesystem-quality write semantics. A driver can be read-only, eventually consistent, size-limited, or expensive to enumerate. Configure `{ readOnly: true }` when the selected Storage cannot safely mutate. + +The adapter stores records below a reserved key prefix, `opfs` by default. Virtual path segments are encoded reversibly before they become unstorage key segments. + +```ts +const adapter = createUnstorageAdapter(storage, { + prefix: "my-app-fs", + readOnly: false, + disposeStorage: false, +}); +``` + +### Reverse unstorage direction + +`createUnstorageDriver(fileSystem)` lets unstorage consume any `FileSystemType`. + +```text +unstorage Storage + | + v +@okikio/opfs unstorage Driver + | + v +FileSystemType + | + +--- OPFS + +--- Node/Deno/Bun + +--- RxDB + +--- db0 + +--- Drizzle + +--- custom adapter +``` + +The driver maps `:` hierarchy segments to private filesystem directories with reversible percent-based encoding. Each key stores its payload in a dedicated `value` leaf file. This indirection is required because unstorage can hold both `foo` and `foo:bar`, while a normal filesystem cannot make `/foo` both a file and a directory. Literal `%` and literal `~` remain distinct. + +`disposeFileSystem` defaults to false because the injected filesystem is borrowed. + +RxDB +---- + +RxDB explicitly defines `RxStorage` as the storage-engine abstraction. The upstream storage interface creates `RxStorageInstance` objects that own bulk writes, queries, attachment access, change streams, cleanup, close, and remove semantics. + +`@okikio/opfs` does not implement `RxStorage`. Instead it uses a normal RxDB collection whose underlying storage can be any RxStorage chosen by the application. + +The package exports `RxDbRecordJsonSchema`: + +```ts +await database.addCollections({ + files: { schema: RxDbRecordJsonSchema }, +}); + +const fileSystem = createFileSystem( + createRxDbAdapter(database.files), +); +``` + +The bridge uses collection operations that preserve RxDB's document concurrency semantics: + +```text +findOne(path).exec() +find({ selector: { parent } }).exec() +incrementalUpsert(record) +incrementalRemove() +``` + +The `path` field is the primary key. `parent` is indexed for direct directory listing. The exported schema sets `maxLength: 4096` on both indexed path fields because RxDB requires a maximum length for indexed strings; the adapter rejects longer paths before querying or writing the collection. + +### RxStorage coverage + +As reviewed from RxDB's current storage guide on 2026-08-12, upstream documents these storage implementations and wrappers: + +Native/storage implementations: + +- Memory +- LocalStorage +- premium IndexedDB +- premium OPFS +- premium Filesystem Node + +Storage wrappers/infrastructure: + +- premium Worker +- premium SharedWorker +- Remote +- premium Sharding +- premium Memory Mapped +- premium Localstorage Meta Optimizer +- Electron IPC renderer/main integration + +Third-party or premium-backed storage families documented by RxDB: + +- premium Expo Filesystem +- premium SQLite +- Dexie.js +- MongoDB +- DenoKV +- FoundationDB + +Because this package sits above the collection, the adapter does not need a separate implementation for each item in that list. The selected RxStorage still owns its own requirements, licensing, runtime constraints, multi-instance behavior, replication behavior, and performance characteristics. + +db0 +--- + +The db0 bridge targets the high-level `Database` interface: + +```text +dialect +prepare(sql) +statement.get/all/run +optional dispose +``` + +The current db0 type contract reports four SQL dialects: + +```text +sqlite +libsql +postgresql +mysql +``` + +`createDb0Adapter()` has explicit SQL generation for all four. The behavioral test suite exercises all four dialect branches. + +As reviewed from db0's generated connector catalog on 2026-08-12, upstream connector names include: + +- better-sqlite3 +- bun-sqlite and bun alias +- Cloudflare D1 +- Cloudflare Hyperdrive MySQL +- Cloudflare Hyperdrive PostgreSQL +- libSQL core, HTTP, Node, web, and alias +- mysql2 +- node-sqlite and sqlite alias +- PGlite +- PlanetScale +- PostgreSQL +- sqlite3 + +The bridge depends on the `Database` contract and dialect, not the connector name. A connector therefore does not need bespoke OPFS code when it presents a compatible db0 Database. + +Prepared statements use db0's portable `?` placeholders. PostgreSQL-family db0 connectors own translation to native `$1`, `$2`, and later parameters, so the filesystem bridge does not duplicate connector-specific parameter rewriting. + +### db0 table + +Default table: `opfs_entries`. + +The adapter can initialize it: + +```ts +const adapter = await createDb0Adapter(database, { + initialize: true, + table: "opfs_entries", +}); +``` + +The path primary key is a SHA-256 hex digest. The original path is stored separately. This is important for MySQL because arbitrary `TEXT` is not a portable primary-key choice. + +`parent_path` is used for directory listing. Large installations should add a provider-appropriate index through their normal migration system if directory listing becomes a hot query. + +`disposeDatabase` defaults to false. + +Drizzle +------- + +Drizzle is not one SQL dialect. It exposes dialect-specific schema builders and many driver entrypoints. The adapter therefore does not create or migrate a universal table. + +The caller provides: + +1. a connected Drizzle database; +2. a table built for that database dialect; +3. the required logical columns. + +Required table properties: + +```text +path +parent +name +kind +data +size +lastModified +mediaType +``` + +`path` must be unique or a primary key. `size` and `lastModified` must round-trip JavaScript safe integers. + +The bridge uses Drizzle's common CRUD shape: + +```text +select().from(table).where(eq(...)) +insert(table).values(...) +delete(table).where(eq(...)) +``` + +That choice keeps the integration usable across Drizzle database objects that expose this common surface. It also means record replacement is delete-then-insert rather than dialect-specific upsert SQL. + +### Concurrency consequence + +Inside one `FileSystemType` with normal coordination, same-path mutations are serialized. Across multiple processes, hosts, or independently configured applications, delete-then-insert is not an atomic database transaction. + +If cross-process atomic replacement is required, provide a database-level transaction/serialization strategy appropriate for the actual Drizzle dialect and driver. The package does not hide that requirement behind a false portability claim. + +### Drizzle driver breadth + +The current Drizzle source tree contains dedicated runtime/dialect integrations such as AWS Data API, better-sqlite3, Bun SQL, Bun SQLite, Cloudflare D1, Durable SQLite, Expo SQLite, and many additional PostgreSQL/MySQL/SQLite-family drivers. The package's compatibility condition is the database object's CRUD surface and the caller's correct table schema, not a hard-coded driver-name allowlist. + +Choosing between the ecosystem bridges +-------------------------------------- + +Use the bridge for the abstraction your application already owns. + +| Existing application resource | Use | +| --- | --- | +| unstorage `Storage` | `createUnstorageAdapter()` | +| RxDB collection | `createRxDbAdapter()` | +| db0 `Database` | `createDb0Adapter()` | +| Drizzle database + table | `createDrizzleAdapter()` | +| custom document/KV layer | `createRecordAdapter()` | +| host directory | Deno/Bun/Node adapter | + +Do not wrap a db0 Database in unstorage merely to reach this package if the application already owns db0 directly. Each extra storage layer adds semantics and performance behavior that has to be understood. diff --git a/docs/environments.md b/docs/environments.md new file mode 100644 index 0000000..c2494dd --- /dev/null +++ b/docs/environments.md @@ -0,0 +1,177 @@ +Execution environments +====================== + +`@okikio/opfs` separates the frontend filesystem model from backend availability. This lets the same core APIs compile in Window, WebWorker, Deno, Bun, and Node TypeScript targets while concrete adapters stay on explicit runtime subpaths. + +Browser OPFS +------------ + +The native OPFS adapter requires `navigator.storage.getDirectory()`. + +The package does not select behavior from a browser brand table. It probes the APIs the current realm exposes and preserves native failures when the browser denies storage. + +### Window + +Use the full async facade. + +```ts +const fileSystem = await openFileSystem(); +await fileSystem.writeFile("/state.json", "{}", { parents: true }); +``` + +Do not assume `createSyncAccessHandle()` is available in Window. The sync API is capability-gated. + +### DedicatedWorker + +The async facade works when OPFS is exposed. A DedicatedWorker is also the intended browser context for synchronous access handles in the File System standard. + +```ts +const capabilities = await probeOpfs(); +if (capabilities.syncAccessExposed) { + const file = await fileSystem.openSyncFile("/database.sqlite", { + create: true, + parents: true, + }); + try { + // synchronous random access + } finally { + file.close(); + } +} +``` + +### SharedWorker + +Use the async facade. Do not infer synchronous access only from the fact that code is running in a worker. The package checks the actual handle method. + +### ServiceWorker + +Use async filesystem methods and keep event lifetime explicit. + +```ts +self.addEventListener("message", (event) => { + const operation = (async () => { + const fileSystem = await openFileSystem(); + await fileSystem.writeFile("/events/latest.json", "{}", { + parents: true, + }); + })(); + + event.waitUntil(operation); +}); +``` + +The filesystem cannot extend a service-worker event lifetime on its own. + +Iframes +------- + +### Same-origin iframe + +Normal `openFileSystem()` opens the storage associated with the iframe's current storage key. + +### Third-party iframe + +Browser storage partitioning can give a third-party iframe storage isolated by the embedding site. Normal `openFileSystem()` deliberately does not attempt to escape that policy. + +For browsers that support the Storage Access API extension for OPFS, the separate iframe module can request unpartitioned access: + +```ts +import { + requestUnpartitionedFileSystem, + supportsUnpartitionedOpfsRequest, +} from "@okikio/opfs/iframe"; +``` + +The application must make the request from the appropriate user-activation/permission flow. The package never does it automatically. + +### Opaque sandbox + +A sandboxed iframe without a usable origin can reject storage access. Treat `probeOpfs().rootAvailable` and the returned root error as the source of truth for that context. + +Private browsing +---------------- + +Private/incognito modes can change storage quota, persistence, availability, or lifetime. The package does not fingerprint the browsing mode. + +The decision flow is: + +```text +probe actual capability + | + +-- root available -> use selected OPFS strategy + | + +-- root unavailable -> inspect normalized error + choose application fallback +``` + +This is more reliable than inferring behavior from a browser/private-mode label. + +`file:` documents +----------------- + +Current browser behavior for OPFS in `file:` documents is not fully interoperable. The WHATWG File System issue tracker contains an active request to specify this case more clearly. + +Do not promise OPFS availability for a packaged `file:` application. Probe the actual context. + +Web Locks +--------- + +When `coordination: "auto"`, the facade uses Web Locks if `navigator.locks` is available. This can coordinate cooperating tabs and workers that share the same origin and lock names. + +If Web Locks are unavailable, `auto` falls back to one-realm FIFO locks. That fallback cannot coordinate another tab, worker realm, or OS process. + +Deno +---- + +Use `@okikio/opfs/adapter/deno`. + +```ts +const fileSystem = createFileSystem( + createDenoAdapter({ root: "./data" }), + { coordination: "local" }, +); +``` + +The runtime needs filesystem permissions appropriate for the configured root. The adapter does not request broader permissions or inspect environment variables itself. + +Native move and sync random access are available through Deno filesystem APIs. + +Bun +--- + +Use `@okikio/opfs/adapter/bun`. + +The adapter uses Bun's file primitives where they provide a clear benefit and Bun's Node-compatible filesystem surface for directory/update/sync operations. + +The root entrypoint never imports the Bun adapter, so browser code does not evaluate Bun-specific code accidentally. + +Node +---- + +Use `@okikio/opfs/adapter/node`. + +The adapter uses `node:fs` and `node:fs/promises`. It supports streaming, range reads, rename, synchronous random access, and flush. + +The configured host root is the only directory intentionally exposed through the virtual namespace. + +Electron +-------- + +The Node adapter is suitable for a trusted Electron main-process filesystem layer. Do not expose arbitrary host roots directly to untrusted renderer content merely because the frontend API resembles OPFS. + +A renderer can instead communicate with a controlled main-process service or use browser OPFS where that matches the application model. + +Record/database backends +------------------------ + +RxDB, unstorage, db0, and Drizzle integrations work in any runtime where the injected upstream resource works and where the package's core Web APIs (`ReadableStream`, `Blob`, `File`, `TextEncoder`, AbortSignal) are available. + +Their filesystem file bodies are record-backed, so stream writes are bounded-buffer operations rather than native streaming. + +Server coordination +------------------- + +`coordination: "local"` coordinates only within one JavaScript realm. It does not serialize writes across separate Node/Deno/Bun processes or separate hosts. + +Database-backed applications that need cross-process same-path atomicity must use backend-level transactions, leases, advisory locks, or another coordination mechanism suitable for that provider. diff --git a/docs/media/terms.md b/docs/media/terms.md deleted file mode 100644 index 45762ae..0000000 --- a/docs/media/terms.md +++ /dev/null @@ -1,12 +0,0 @@ -Media terms -=========== - -- **source**: Media bytes that the library can inspect, download, or convert. -- **track**: One synchronized sequence of video, audio, or text media. -- **packet**: Encoded media data. -- **sample**: Decoded media data. -- **remux**: Copy encoded media into another container without decoding it. -- **transcode**: Decode media and encode it again. -- **download**: Copy remote resource bytes to an output without media conversion. -- **writer**: Resource that writes bytes to an output position. -- **task**: One long-running media operation with authoritative state and a terminal result. diff --git a/docs/sources.md b/docs/sources.md new file mode 100644 index 0000000..52d61a6 --- /dev/null +++ b/docs/sources.md @@ -0,0 +1,165 @@ +Research and source register +============================ + +Research date: 2026-08-12. + +Source priority +--------------- + +When sources disagree, use this order: + +1. current standards and current upstream source contracts; +2. current `okikio/mediad` repository rules and current Kaiju Platform/Crawl architecture guides; +3. current package implementation and tests; +4. older experiments and secondary articles. + +The old `okikio/testing-opfs` experiment was reviewed for intent only. It is not an implementation base. + +Browser File System / OPFS +-------------------------- + +Primary standards and interoperability sources: + +- WHATWG File System Standard: +- WHATWG File System issues: +- Web Platform Tests File System suite: +- WPT interoperability issue supplied for review: +- MDN Origin Private File System overview: +- web.dev OPFS article: + +Important design facts traced into code/tests: + +- normal file/directory handle operations are asynchronous; +- synchronous access handle exposure is context/capability-specific and can hold native file locks; +- writable streams and sync files have explicit close/abort lifecycle; +- current portable OPFS does not provide the same universal native rename contract as a host filesystem; +- error names, locks, storage policy, private browsing, iframe partitioning, and `file:` documents contain interoperability details that must not be hidden by browser-name guessing. + +Additional OPFS material supplied by the user and reviewed for behavior/performance context: + +- +- +- +- + +These secondary/performance sources informed test cases and tradeoffs. They do not override the standard or current upstream contracts. + +Deno standard filesystem +------------------------ + +- Deno standard library repository: +- `@std/fs`: +- current `fs/mod.ts`, `fs/walk.ts`, `fs/copy.ts`, and `fs/move.ts` source were reviewed. + +Useful patterns retained: + +- lazy tree walking; +- explicit overwrite behavior; +- source/destination overlap checks; +- bounded, understandable helper APIs. + +Native-host assumptions deliberately not copied into OPFS/record adapters: + +- symbolic links; +- host permission bits; +- OS path identity; +- portable timestamp mutation; +- universal native rename. + +RxDB +---- + +- RxStorage guide: +- RxStorage interface: +- RxCollection implementation: +- RxDocument type contract: + +The integration point is `RxCollection`, while RxDB retains responsibility for the chosen RxStorage implementation, wrappers, replication, multi-instance behavior, conflicts, and licensing. + +unstorage +--------- + +- repository: +- `src/types.ts` for `Storage` and `Driver` contracts +- generated `src/_drivers.ts` for current built-in driver inventory + +The forward bridge targets `Storage`. The reverse bridge implements the stable Driver subset used by unstorage. + +db0 +--- + +- site: +- repository: +- `src/types.ts` for Database/Statement/dialect contracts +- generated `src/_connectors.ts` for current connector inventory + +The adapter targets `Database` and its reported SQL dialect, not a connector name. + +Drizzle +------- + +- repository: +- current package metadata and `drizzle-orm/src` driver/dialect tree +- SQLite core database/query builder source for the common CRUD shape + +Drizzle schema and DDL remain caller-owned because they are dialect-specific. The integration uses a caller-supplied table and common select/insert/delete builders. + +Mediad conventions +------------------ + +Current private repository reviewed through the connected GitHub source: + +- `okikio/mediad/AGENTS.md` +- root workspace/package/TypeScript configuration +- `docs/` organization +- `packages/media/*` organization +- `packages/media/storage` source + +Rules applied here include: + +- one-word capability-oriented folders where practical; +- precise verbs; +- `Schema` suffix for Zod schema constants; +- `Type` suffix for project-owned data types; +- same core TypeScript source across runtimes; +- explicit runtime subpaths; +- caller-owned injected resources by default; +- TSDoc that explains examples, impact, ownership, limits, failure behavior, and necessary background. + +Kaiju Platform and Crawl conventions +------------------------------------ + +The connected Library sources reviewed include: + +- Kaiju Platform Programming Model +- library-first architecture guidebook +- Kaiju naming and folder structure guide +- Kaiju code formatting guide +- Kaiju readable Markdown / technical writing handbook +- Kaiju Platform package/service architecture handoff +- Kaiju Crawl architecture and capability alignment handoff + +The project guidance used here includes: + +- library-first composition; +- explicit resource ownership and disposal; +- import-safe capability packages; +- exact runtime-resource names instead of vague terms; +- focused public subpaths; +- lazy iterators and bounded active memory; +- `.agents/` for temporary Node validation when Deno/JSR are not available; +- authored documentation must explain how exports compose into real developer workflows. + +Uploaded skill pack +------------------- + +`skills(20260806-212711).zip` was reviewed before this refactor. Relevant software delivery references included: + +- documentation requirements; +- comment/TSDoc requirements; +- TypeScript requirements; +- library architecture and packaging; +- Deno software packaging; +- storage/database design and Drizzle guidance. + +The skill pack is a process/reference input. It is not copied into the package artifact. diff --git a/docs/testing/packages.md b/docs/testing/packages.md deleted file mode 100644 index 78c37df..0000000 --- a/docs/testing/packages.md +++ /dev/null @@ -1,8 +0,0 @@ -Package testing -=============== - -Each `@media/*` package owns `mod_test.ts`. Tests use `node:test` for the runner -and BDD suite shape, and `@std/expect` for assertions. The same test file is also -valid under Deno. - -Run all package tests with `mise run test`. diff --git a/docs/validation.md b/docs/validation.md new file mode 100644 index 0000000..e97cc52 --- /dev/null +++ b/docs/validation.md @@ -0,0 +1,156 @@ +Validation strategy +=================== + +The repository keeps validation support under `.agents/` because the ChatGPT execution host does not provide every production runtime or registry dependency. + +Production code remains Deno/browser/server-native. Validation shims do not enter the package exports or publish list. + +What is validated here +---------------------- + +The validation matrix has separate TypeScript targets so one environment cannot accidentally provide globals for another. + +```text +Window target + core + browser OPFS + ecosystem structural adapters + +WebWorker target + core + browser OPFS worker declarations + +Server target + Node + Deno + Bun concrete adapters + +Deno test source target + repository tests with validation-only Deno.test declaration + +Emit target + ESM + declarations for public output inspection and behavior tests +``` + +Commands +-------- + +```sh +tsc -p .agents/tsconfig.window.json +tsc -p .agents/tsconfig.worker.json +tsc -p .agents/tsconfig.server.json +tsc -p .agents/tsconfig.tests.json + +tsc -p .agents/tsconfig.emit.json +node .agents/scripts/prepare-node-runtime.mjs +node --test .agents/tests/adapters.test.mjs +``` + +Representative consumers are also type-checked: + +```sh +tsc -p .agents/tsconfig.consumer.window.json +tsc -p .agents/tsconfig.consumer.worker.json +``` + +The npm payload is inspected without publishing: + +```sh +npm pack --dry-run --json +``` + +`package.json#files` excludes `.agents/`, tests, and generated validation output from the npm package. + +The browser matrix has its own build configuration: + +```sh +tsc -p .agents/tsconfig.browser.emit.json +node .agents/scripts/prepare-browser-runtime.mjs +node .agents/browser/server.mjs +``` + +The browser server is validation-only. It is not package runtime infrastructure. + +Behavioral coverage +------------------- + +The Node-hosted adapter contract suite covers: + +- runtime adapter capability/schema rejection; +- Web Locks request shape; +- path normalization and virtual-root escape rejection; +- replace, append, update, byte range, and stat semantics; +- OPFS-shaped file/directory handles over a non-OPFS backend; +- Blob versus File System write-command discrimination; +- staged writable close/abort semantics; +- bounded record-adapter stream buffering and producer cancellation; +- overwrite copy replacing stale trees; +- fallback move removing source only after successful copy; +- source/destination overlap protection; +- aborted queued write recovery; +- independent file write concurrency; +- structural operations waiting for active file mutation; +- post-open stream cancellation; +- unstorage high-level Storage bridge; +- reverse unstorage driver, reversible key encoding, and `foo` plus `foo:bar` prefix-collision handling; +- RxDB collection bridge; +- db0 SQLite, libSQL, PostgreSQL, and MySQL SQL branches; +- db0 SQLite DDL/upsert/select/delete execution against Node's real SQLite engine; +- Drizzle common CRUD bridge; +- real Node filesystem streaming, rename, sync random access, flush, and sync-lock lifetime; +- explicit adapter disposal ownership; +- non-throwing OPFS probe outside a browser OPFS context. + +The repository also contains Deno-native tests for path handling and the memory adapter frontend contract. Their source is type-checked here even when the Deno executable is unavailable. + +Dependency validation in this host +---------------------------------- + +Network package installation is unavailable in the current execution host. The validation configs map `zod` and the small `drizzle-orm` `eq()` dependency to `.agents/stubs/` only for local type/behavior execution. + +These stubs are not production dependencies and are not published. + +The purpose of the Zod stub is to exercise the package's schema calls and failure branches. The purpose of the Drizzle stub is to exercise the adapter's common CRUD builder translation with a fake connected database. + +A release environment with registry access must run the same checks against the real dependency versions before publish. + +Deno runtime status +------------------- + +The current host does not have a Deno executable. Therefore the following production-native commands cannot be truthfully marked passed here: + +```text +deno task check +deno task test +deno task fmt:check +deno task lint +deno publish --dry-run +``` + +The repository keeps these tasks in `deno.json` so a Deno-capable release environment can run them directly. + +Bun runtime status +------------------ + +The current host does not provide Bun. The Bun adapter is strict-type-checked against its declared runtime shape and shares host-path/Node-compatible primitives with tested code, but a real Bun filesystem execution remains a release-environment check. + +Browser runtime status +---------------------- + +Chromium is installed in this host, but its administrator policy rejects the local trustworthy origin used by the browser matrix with `net::ERR_BLOCKED_BY_ADMINISTRATOR` before the page can run. A synthetic intercepted HTTPS origin was also blocked before interception. + +Therefore live Window/DedicatedWorker/SharedWorker/ServiceWorker/iframe OPFS execution is recorded as environment-blocked, not passed. + +The harness remains in `.agents/browser/` so it can run in a normal Chromium/Firefox/Safari test environment. + +Artifact verification +--------------------- + +Before a ZIP is delivered: + +1. run every available type and behavior check against the working tree; +2. inspect generated ESM/declaration output; +3. inspect package exports and stale symbol references; +4. remove generated `.agents/build`, browser build, and validation `node_modules`; +5. create the ZIP; +6. extract that exact ZIP into a clean directory; +7. recreate only validation-side host shims; +8. rerun the same available checks against the extracted artifact; +9. compare source/extracted file lists and compute SHA-256. + +A result is not called complete if the extracted deliverable fails a check that the source working tree passed. -- 2.51.2