diff --git a/README.md b/README.md index a6aa042..972949a 100644 --- a/README.md +++ b/README.md @@ -48,20 +48,16 @@ This package separates the two responsibilities: The separation is deliberate. It keeps backend-specific behavior out of application code without pretending that every backend has the same performance or durability characteristics. -Install -------- +Pre-release use +--------------- -Deno / JSR: +This source tree is being prepared for JSR and npm publication. Until the package is published, consume it through the workspace or another explicit local source reference instead of assuming the registry entry exists. After release, the intended imports are: ```ts import { createFileSystem, openFileSystem } from "jsr:@okikio/opfs"; ``` -npm-compatible runtimes: - -```sh -npm install @okikio/opfs -``` +and the intended npm-compatible install is `npm install @okikio/opfs`. Release validation must prove both registry artifacts before these forms are treated as available. Drizzle integration also needs the optional peer dependency: @@ -102,6 +98,30 @@ const root = await navigator.storage.getDirectory(); const fileSystem = createFileSystem(createOpfsAdapter(root)); ``` +Use one long-lived positional file +----------------------------------- + +Media muxers and database engines can rewrite earlier byte ranges while output is still open. Use `openWritableFile()` for that access pattern instead of issuing one `writeFile(update)` operation per chunk. + +```ts +const file = await fileSystem.openWritableFile("/output.mp4", { + create: true, + parents: true, +}); + +try { + await file.write(header, { at: 0 }); + await file.write(mediaChunk, { at: offset }); + await file.flush(); + await file.close(); +} catch (error) { + await file.abort(error); + throw error; +} +``` + +The OPFS, Node, Deno, and Bun adapters advertise this capability. Record-backed adapters such as memory, unstorage, RxDB, db0, and Drizzle do not. They remain appropriate for small records and ordinary bounded writes, but the facade will not disguise repeated record replacement as a native positional-file resource. + Use OPFS-shaped handles over Node --------------------------------- diff --git a/docs/adapters.md b/docs/adapters.md index 22ebf70..5f5b6c2 100644 --- a/docs/adapters.md +++ b/docs/adapters.md @@ -46,7 +46,7 @@ const root = await navigator.storage.getDirectory(); const fileSystem = createFileSystem(createOpfsAdapter(root)); ``` -The explicit adapter retains `nativeRoot` for advanced browser interop. Synchronous access is exposed only when the current native file handle actually provides `createSyncAccessHandle()`. +The explicit adapter retains `nativeRoot` for advanced browser interop. It exposes long-lived positional writes through one `FileSystemWritableFileStream`. Synchronous access is exposed only when the current native file handle actually provides `createSyncAccessHandle()`. The adapter does not attempt browser or incognito detection. @@ -64,7 +64,7 @@ const fileSystem = createFileSystem( `root` is the host directory represented by virtual `/`. `createRoot` defaults to true. -The adapter uses Deno filesystem APIs for data operations, rename, sync access, and flush. It uses Node's path compatibility module only to normalize the configured host root and to verify that a virtual path stays below it. +The adapter uses Deno filesystem APIs for data operations, rename, long-lived positional access, sync access, and flush. It uses Node's path compatibility module only to normalize the configured host root and to verify that a virtual path stays below it. ### Bun @@ -86,7 +86,7 @@ import { createNodeAdapter } from "@okikio/opfs/adapter/node"; const adapter = createNodeAdapter({ root: "./data" }); ``` -Node supports native streaming reads/writes, byte ranges, rename, and synchronous random access. +Node supports native streaming reads/writes, byte ranges, rename, long-lived asynchronous positional writes, and synchronous random access. The host root is created by default. The virtual path mapper rejects any resolved host path that would leave that root. @@ -155,6 +155,7 @@ export const adapter = defineAdapter({ streamWrite: false, rangeRead: true, nativeMove: false, + positionalWrite: false, syncAccess: false, }, @@ -196,12 +197,15 @@ Optional native operations Only advertise a capability when the adapter implements the corresponding native method. ```text -streamRead -> openReadStream -streamWrite -> writeStream -nativeMove -> move -syncAccess -> openSyncFile +streamRead -> openReadStream +streamWrite -> writeStream +nativeMove -> move +positionalWrite -> openWritableFile +syncAccess -> openSyncFile ``` +`positionalWrite` means the adapter can keep one asynchronous file resource open while callers write explicit byte positions. It is not inferred from ordinary `writeFile(update)` support. This distinction matters for media muxers and database files because repeated whole-record replacement can turn many chunk writes into nonlinear work. + `rangeRead` describes whether the adapter can avoid materializing the complete file for a range. The facade still exposes ranged `readFile()` to all adapters. Cancellation diff --git a/docs/api.md b/docs/api.md index 0913603..80c35a1 100644 --- a/docs/api.md +++ b/docs/api.md @@ -151,6 +151,42 @@ The mode is runtime-validated by `WriteModeSchema`. A non-streaming adapter buffers stream input up to `maxBufferedWriteBytes`. Crossing the limit cancels the producer and throws `too-large`. +Long-lived positional output +---------------------------- + +### `openWritableFile(path, options?)` + +Returns `WritableFileType` only when `adapter.capabilities.positionalWrite` is true. The facade does not emulate this operation with repeated `writeFile(..., { mode: "update" })` calls because a record-backed adapter can otherwise rematerialize the complete file for every chunk. + +Options: + +- `create`: create an empty file when it is absent. +- `parents`: create missing parent directories when `create` is true. +- `signal`: cancel ordinary work before commit. Cleanup through `close()` or `abort()` still releases the owned resource after cancellation. + +The returned resource owns the file mutation lock for its complete lifetime. The create/check/open sequence occurs under that same lock, so another mutation cannot enter between file creation and adapter open. + +```ts +const file = await fileSystem.openWritableFile("/media/output.mp4", { + create: true, + parents: true, +}); + +try { + await file.write(header, { at: 0 }); + await file.write(chunk, { at: chunkOffset }); + await file.flush(); + await file.close(); +} catch (error) { + await file.abort(error); + throw error; +} +``` + +`write()` is positional, `truncate()` changes byte length, and `flush()` requests backend durability without closing. `close()` and `abort()` are idempotent terminal operations. A browser OPFS writable can discard its staged image on abort. Host filesystems generally cannot roll back bytes already written, so an application that needs publish-on-success semantics should write a staging path and move it after close. + +Record/database adapters report `positionalWrite: false`. They remain valid for ordinary materialized writes and bounded stream buffering, but they are not presented as a large-file positional output path. + Directory iteration ------------------- diff --git a/docs/design.md b/docs/design.md index 0c4bff2..65fe72e 100644 --- a/docs/design.md +++ b/docs/design.md @@ -90,6 +90,7 @@ streamRead streamWrite rangeRead nativeMove +positionalWrite syncAccess ``` @@ -97,6 +98,8 @@ These values describe what the adapter itself can do. They do not describe every For example, a record-store adapter reports `streamWrite: false`. The facade can still accept a `ReadableStream`, but it must buffer the stream before storing the record. The capability remains false because pretending that buffering is native streaming would hide an important memory and latency difference. +The same rule applies to `positionalWrite`. A record adapter can implement one `writeFile(update)` operation by reading and replacing a record, but that does not mean it can keep a writable file open across thousands of positional chunks. Record adapters therefore report `positionalWrite: false`. Native OPFS, Node, Deno, and Bun adapters expose `openWritableFile()` when they can keep one underlying writable resource open. + The record-store layer ---------------------- @@ -252,6 +255,21 @@ This lets independent files make progress at the same time while ensuring that a The adapter still owns any stronger backend-level locking. Library locks are application-level coordination for callers that use this library. +Asynchronous positional file lifecycle +-------------------------------------- + +A long-lived asynchronous writable owns the same file mutation lock from its create/check/open sequence until terminal cleanup. + +```text +facade file lock <------ same lifetime ------> adapter writable file + | | + +-------------- close/abort --------------+ +``` + +The facade deliberately does not build this contract from repeated `writeFile(update)` calls. Backends with native or staged file resources can preserve positional-write throughput, while record backends remain explicit about the fact that they materialize whole values. + +Cancellation stops ordinary `write()`, `truncate()`, and `flush()` calls. It does not prevent `close()` or `abort()` from releasing the backend resource and the facade lock. If a backend cannot roll back writes, callers that need all-or-nothing publication should use a staging path. + Synchronous file lifecycle -------------------------- diff --git a/docs/ecosystems.md b/docs/ecosystems.md index aa6192a..6bc4725 100644 --- a/docs/ecosystems.md +++ b/docs/ecosystems.md @@ -27,7 +27,7 @@ optional dispose This means the bridge is independent of the mounted driver. -As reviewed on 2026-08-12, unstorage's generated built-in driver catalog includes these families: +unstorage's generated built-in driver catalog includes these families: - Azure App Configuration, Cosmos, Key Vault, Storage Blob, and Storage Table - Capacitor Preferences diff --git a/docs/validation.md b/docs/validation.md deleted file mode 100644 index e97cc52..0000000 --- a/docs/validation.md +++ /dev/null @@ -1,156 +0,0 @@ -Validation strategy -=================== - -The repository keeps validation support under `.agents/` because the ChatGPT execution host does not provide every production runtime or registry dependency. - -Production code remains Deno/browser/server-native. Validation shims do not enter the package exports or publish list. - -What is validated here ----------------------- - -The validation matrix has separate TypeScript targets so one environment cannot accidentally provide globals for another. - -```text -Window target - core + browser OPFS + ecosystem structural adapters - -WebWorker target - core + browser OPFS worker declarations - -Server target - Node + Deno + Bun concrete adapters - -Deno test source target - repository tests with validation-only Deno.test declaration - -Emit target - ESM + declarations for public output inspection and behavior tests -``` - -Commands --------- - -```sh -tsc -p .agents/tsconfig.window.json -tsc -p .agents/tsconfig.worker.json -tsc -p .agents/tsconfig.server.json -tsc -p .agents/tsconfig.tests.json - -tsc -p .agents/tsconfig.emit.json -node .agents/scripts/prepare-node-runtime.mjs -node --test .agents/tests/adapters.test.mjs -``` - -Representative consumers are also type-checked: - -```sh -tsc -p .agents/tsconfig.consumer.window.json -tsc -p .agents/tsconfig.consumer.worker.json -``` - -The npm payload is inspected without publishing: - -```sh -npm pack --dry-run --json -``` - -`package.json#files` excludes `.agents/`, tests, and generated validation output from the npm package. - -The browser matrix has its own build configuration: - -```sh -tsc -p .agents/tsconfig.browser.emit.json -node .agents/scripts/prepare-browser-runtime.mjs -node .agents/browser/server.mjs -``` - -The browser server is validation-only. It is not package runtime infrastructure. - -Behavioral coverage -------------------- - -The Node-hosted adapter contract suite covers: - -- runtime adapter capability/schema rejection; -- Web Locks request shape; -- path normalization and virtual-root escape rejection; -- replace, append, update, byte range, and stat semantics; -- OPFS-shaped file/directory handles over a non-OPFS backend; -- Blob versus File System write-command discrimination; -- staged writable close/abort semantics; -- bounded record-adapter stream buffering and producer cancellation; -- overwrite copy replacing stale trees; -- fallback move removing source only after successful copy; -- source/destination overlap protection; -- aborted queued write recovery; -- independent file write concurrency; -- structural operations waiting for active file mutation; -- post-open stream cancellation; -- unstorage high-level Storage bridge; -- reverse unstorage driver, reversible key encoding, and `foo` plus `foo:bar` prefix-collision handling; -- RxDB collection bridge; -- db0 SQLite, libSQL, PostgreSQL, and MySQL SQL branches; -- db0 SQLite DDL/upsert/select/delete execution against Node's real SQLite engine; -- Drizzle common CRUD bridge; -- real Node filesystem streaming, rename, sync random access, flush, and sync-lock lifetime; -- explicit adapter disposal ownership; -- non-throwing OPFS probe outside a browser OPFS context. - -The repository also contains Deno-native tests for path handling and the memory adapter frontend contract. Their source is type-checked here even when the Deno executable is unavailable. - -Dependency validation in this host ----------------------------------- - -Network package installation is unavailable in the current execution host. The validation configs map `zod` and the small `drizzle-orm` `eq()` dependency to `.agents/stubs/` only for local type/behavior execution. - -These stubs are not production dependencies and are not published. - -The purpose of the Zod stub is to exercise the package's schema calls and failure branches. The purpose of the Drizzle stub is to exercise the adapter's common CRUD builder translation with a fake connected database. - -A release environment with registry access must run the same checks against the real dependency versions before publish. - -Deno runtime status -------------------- - -The current host does not have a Deno executable. Therefore the following production-native commands cannot be truthfully marked passed here: - -```text -deno task check -deno task test -deno task fmt:check -deno task lint -deno publish --dry-run -``` - -The repository keeps these tasks in `deno.json` so a Deno-capable release environment can run them directly. - -Bun runtime status ------------------- - -The current host does not provide Bun. The Bun adapter is strict-type-checked against its declared runtime shape and shares host-path/Node-compatible primitives with tested code, but a real Bun filesystem execution remains a release-environment check. - -Browser runtime status ----------------------- - -Chromium is installed in this host, but its administrator policy rejects the local trustworthy origin used by the browser matrix with `net::ERR_BLOCKED_BY_ADMINISTRATOR` before the page can run. A synthetic intercepted HTTPS origin was also blocked before interception. - -Therefore live Window/DedicatedWorker/SharedWorker/ServiceWorker/iframe OPFS execution is recorded as environment-blocked, not passed. - -The harness remains in `.agents/browser/` so it can run in a normal Chromium/Firefox/Safari test environment. - -Artifact verification ---------------------- - -Before a ZIP is delivered: - -1. run every available type and behavior check against the working tree; -2. inspect generated ESM/declaration output; -3. inspect package exports and stale symbol references; -4. remove generated `.agents/build`, browser build, and validation `node_modules`; -5. create the ZIP; -6. extract that exact ZIP into a clean directory; -7. recreate only validation-side host shims; -8. rerun the same available checks against the extracted artifact; -9. compare source/extracted file lists and compute SHA-256. - -A result is not called complete if the extracted deliverable fails a check that the source working tree passed. diff --git a/tsconfig.json b/tsconfig.json index 27809f0..f68b280 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -1,16 +1,17 @@ { - "compilerOptions": { - "target": "ES2024", - "module": "NodeNext", - "moduleResolution": "NodeNext", - "lib": ["ES2024", "DOM", "DOM.Iterable"], - "strict": true, - "noUncheckedIndexedAccess": true, - "verbatimModuleSyntax": true, - "allowImportingTsExtensions": true, - "noEmit": true, - "skipLibCheck": true, - "types": ["node"] - }, - "include": ["packages/media/**/*.ts"] + "compilerOptions": { + "target": "esnext", + "module": "esnext", + "moduleResolution": "bundler", + "lib": ["DOM", "DOM.Iterable"], + "strict": true, + "noUncheckedIndexedAccess": true, + "exactOptionalPropertyTypes": true, + "useUnknownInCatchVariables": true, + "verbatimModuleSyntax": true, + "allowImportingTsExtensions": true, + "noEmit": true, + "skipLibCheck": true, + "types": ["node", "deno"] + } }