diff --git a/crates/cli/tests/build.rs b/crates/cli/tests/build.rs index 8a2c805..7e513fc 100644 --- a/crates/cli/tests/build.rs +++ b/crates/cli/tests/build.rs @@ -8,7 +8,7 @@ fn builds_working_examples() { for (example, target) in [ ("examples/scalar_project", None), ("examples/multi_module_project", None), - ("examples/browser_scalar", Some("browser")), + ("examples/scalar_project", Some("browser")), ] { let out_dir = unique_temp_dir("regulus_cli_example_build"); fs::create_dir_all(&out_dir).expect("create output dir"); diff --git a/docs/book/src/chapter_7/running_wasmtime_browser.md b/docs/book/src/chapter_7/running_wasmtime_browser.md index 3861111..ee8dbf1 100644 --- a/docs/book/src/chapter_7/running_wasmtime_browser.md +++ b/docs/book/src/chapter_7/running_wasmtime_browser.md @@ -62,23 +62,19 @@ A browser host usually fetches bytes and instantiates them with an import object: ```js -import { createRegulusBrowserImports } from "./host.js"; +import { initBrowserPage } from "./module.mjs"; -let instance; -const imports = createRegulusBrowserImports(() => instance); +const exports = await initBrowserPage(fetch("module.wasm")); -({ instance } = await WebAssembly.instantiateStreaming(fetch("module.wasm"), imports)); - -console.log(instance.exports.id(42n)); +console.log(exports.id(42n)); ``` `instantiateStreaming` is a browser Web API convenience for compiling and instantiating from a `Response`. Hosts that cannot stream can fetch an `ArrayBuffer` and call `WebAssembly.instantiate` instead. -The browser import module is named `browser`. The example glue in -`examples/browser/host.js` provides `print`, `println`, `debug_i64`, -`debug_f64`, `debug_bool`, and `debug_value`. The debug imports print and +The browser import module is named `browser`. Generated JS adapters provide the +checked instantiation path for browser-target modules. Browser imports print and return control to compiled code; the compiler preserves the debugged value on its own stack. diff --git a/docs/internal/README.md b/docs/internal/README.md index 1e5be6a..5cc379a 100644 --- a/docs/internal/README.md +++ b/docs/internal/README.md @@ -13,15 +13,13 @@ Wasm artifact. Current work is focused on making that Gleam-project-to-Wasm path more usable: -1. Finish the JavaScript host ABI pieces needed by Wasm projects. -2. Harden runtime memory and host pointer behavior. -3. Improve CLI artifacts, diagnostics, and host metadata. -4. Fill stdlib and dependency gaps that block realistic projects. -5. Add larger examples as acceptance fixtures after the core gaps are planned. +1. Harden runtime memory and host pointer behavior. +2. Improve CLI artifacts, diagnostics, and host metadata. +3. Fill stdlib and dependency gaps that block realistic projects. +4. Add larger examples as acceptance fixtures after the core gaps are planned. ## Specs -- [JS host ABI](specs/15_js_host_abi.md) - [Stdlib and host interop](specs/16_stdlib_and_host_interop.md) - [CLI and build outputs](specs/17_cli_and_build_outputs.md) - [Example projects](specs/18_example_projects.md) @@ -30,7 +28,6 @@ Current work is focused on making that Gleam-project-to-Wasm path more usable: ## Task Trackers -- [JS host ABI](tasks/15_js_host_abi.md) - [Stdlib and host interop](tasks/16_stdlib_and_host_interop.md) - [CLI and build outputs](tasks/17_cli_and_build_outputs.md) - [Runtime memory hardening](tasks/20_runtime_memory_hardening.md) diff --git a/docs/internal/specs/15_js_host_abi.md b/docs/internal/specs/15_js_host_abi.md deleted file mode 100644 index 15776fb..0000000 --- a/docs/internal/specs/15_js_host_abi.md +++ /dev/null @@ -1,215 +0,0 @@ -# JS host ABI - -Regulus targets browser-capable Wasm. Raw Wasm only passes scalar values, so -JavaScript hosts need a stable ABI for strings, managed values, host APIs, and -generated or handwritten JS glue. - -This spec defines the shared JavaScript host boundary. Browser, bundler, and -Node.js profiles should build on this boundary instead of adding -product-specific compiler behavior. - -The JS host ABI is the next usability milestone for Regulus. Broad stdlib -coverage should not block this work. A small, documented JS boundary is more -important than reimplementing library behavior in the compiler. - -## Priority - -Regulus should first prove that a Gleam project can compile to Wasm plus JS -glue, call a JS external, pass a JS string into Gleam, return a Gleam string to -JS, and run through a bundler-oriented smoke test. Browser and Node.js profiles -should reuse the same value ABI after that path works. - -Stdlib and dependency support should use this interop substrate wherever -possible. Runtime intrinsics remain appropriate only for compiler-owned value -representations, allocation, closure dispatch, structural equality, failure -payloads, and narrow primitives that cannot be expressed in Gleam source. - -## Scope - -The JS host ABI covers values and calls crossing between compiled Gleam Wasm and -a JavaScript host. It does not define application logic, browser networking -policy, routing, response construction semantics, or library behavior that can -compile from Gleam source. - -The ABI should cover: - -- writing JS strings into guest memory -- reading Gleam strings from managed pointers -- reading common managed values from guest memory -- passing scalar values directly through Wasm parameters and results -- passing lists, tuples, records, custom types, `Result`, and `Option` -- passing opaque JS handles when copying values is not appropriate -- naming imports and exports for browser, bundler, and Node.js profiles -- generated or documented JS glue for common host calls - -## Relationship to the core host ABI - -The core host ABI defines the low-level Wasm shapes: scalars are raw Wasm -values, and managed Gleam values are borrowed pointers into guest memory. The JS -host ABI is a higher-level profile over that contract for JavaScript hosts. - -Hosts may inspect guest-managed values through exported runtime helpers. Hosts -must not mutate guest object memory or retain pointers across instance reset or -any future arena reset. - -## JavaScript glue - -Browser-capable Wasm needs JS glue comparable in role to Rust's -`wasm-bindgen`, even if Regulus starts with a smaller handwritten adapter. The -glue should hide pointer arithmetic from example code and provide explicit -helpers for common shapes. - -The first glue layer can be small and stable: - -- load a bundler-oriented ES module wrapper for a `.wasm` artifact -- instantiate a module with target-specific imports -- write JS strings and receive managed string pointers -- read managed strings from exported function results -- read tagged managed values for simple records, tuples, lists, and custom - types -- call exported Gleam functions with checked argument conversion - -Later glue can add generated bindings from compiler metadata. - -## Structured values - -Structured data crossing the JS boundary should use one of two mechanisms: - -1. Copy values into or out of the guest runtime representation. -2. Pass opaque JS handles managed by the host profile. - -Copied values are appropriate for strings, lists of strings, small records, -tuples, `Result`, `Option`, and simple custom types. Opaque handles are better -for host objects such as `Request`, `Response`, streams, DOM nodes, timers, -promises, file handles, and sockets. - -The ABI must define how the host reads tags, fields, arity, string data, list -contents, and record/custom-type metadata. Unsupported shapes should fail before -byte emission with source-spanned diagnostics. - -## Opaque JS handles - -Opaque JS handles use normal managed runtime objects with tag `8`. The object -header size word is `0`. Payload word `0` is a stable `i32` type tag owned by -the host profile or package adapter. Payload word `1` is an `i32` adapter-table -handle id. Guest code may pass opaque pointers around and compare identity, but -must not read the payload directly. - -The JS adapter owns the handle table. Wrapping a JS value allocates one table id -and one tag-8 guest object. Releasing a handle removes the JS table entry; any -existing guest pointer for that id is then invalid for JS lookup. Reinitializing -a Wasm instance clears the table because all managed pointers from the previous -instance are invalid. - -Opaque pointers are borrowed across calls. The adapter must keep a wrapped JS -value alive until `releaseHandle`, `clearHandles`, or instance reinitialization. -Guest memory never owns, copies, or frees the JavaScript object itself. - -## JS host profiles - -The shared ABI is independent of the JavaScript engine. Profiles describe -module loading, generated glue shape, available host APIs, and import module -names. - -The initial profiles should mirror the common `wasm-bindgen` deployment modes: - -- `browser`: direct use from a browser page without a bundler -- `bundler`: ES module glue intended for Vite, Rollup, Webpack, or other - toolchains that package JavaScript and Wasm together -- `nodejs`: Node.js loading and host APIs - -Additional JavaScript environments should start from one of these profiles -unless they require a distinct module format, loading model, or host API set. -Those differences should not change the core value ABI. - -## Browser profile - -The browser profile reserves the `browser` import module for browser APIs used -by examples. The first stable names are: - -| Import module | Import name | Gleam ABI shape | JavaScript behavior | -| ------------- | ------------------------- | --------------------------- | ------------------- | -| `browser` | `fetch` | `String -> opaque handle` | Calls `fetch(url)` and stores the returned promise or response in the adapter handle table. | -| `browser` | `localStorage.getItem` | `String -> String` | Reads a key and maps missing values to the empty string. | -| `browser` | `localStorage.setItem` | `String, String -> Nil` | Writes a string value for a key. | -| `browser` | `localStorage.removeItem` | `String -> Nil` | Removes a key. | -| `browser` | `time.now` | `Nil -> Int` | Returns Unix time in milliseconds. | -| `browser` | `online.isOnline` | `Nil -> Bool` | Reads `navigator.onLine`, defaulting to `true` when unavailable. | - -These imports are ordinary Gleam external functions with target validation and -ABI checks. - -The compiler should not implement browser API semantics. The JS host profile -owns the real browser calls and adapts values through the JS host ABI. -Browser glue exposes `initBrowserPage(wasm, imports, options)`, which -instantiates the Wasm module with checked browser imports from -`createBrowserImports(options)` and any additional application imports. - -## Bundler profile - -The bundler profile should emit or document ES module glue suitable for modern -JS toolchains. It should avoid assumptions about global script loading and -should expose imports and exports in a shape that bundlers can tree-shake. - -Server-side JS adapters should start here when they use bundled ES modules. -Request and response objects may cross the boundary as copied data or opaque -handles as the ABI matures, but route logic and response shaping should remain -compiled Gleam code where possible. - -## Node.js profile - -The Node.js profile should define loading, filesystem access for `.wasm` files, -and Node-specific host imports. It should use the same value ABI as browser and -bundler profiles. - -Node support is useful for CLI smoke tests, local examples, and host-side tests -that should not require a browser. - -Node glue is emitted as an ES module next to the generated `.wasm` file. The -default Wasm location is the sibling `.wasm` URL resolved from -`import.meta.url`. The adapter exposes `initNode(wasm, imports, options)`, -which accepts the default sibling artifact, a relative or absolute filesystem -path, a `file:` URL, bytes, or a precompiled `WebAssembly.Module`. -File-backed loads use `node:fs/promises` and then instantiate with the same -checked import wrapping and export helpers as the browser and bundler profiles. - -The first stable Node-specific imports are: - -| Import module | Import name | Gleam ABI shape | JavaScript behavior | -| ------------- | ----------- | --------------- | ------------------- | -| `nodejs` | `env.get` | `String -> String` | Reads an environment key and maps missing values to the empty string. | -| `nodejs` | `time.now` | `Nil -> Int` | Returns Unix time in milliseconds. | - -Node glue exposes `createNodeImports(options)` for those APIs. `initNode` -merges those standard imports with any additional application imports before -calling the shared checked JS ABI instantiation path. - -## Diagnostics - -Unsupported JS host imports, exports, value shapes, opaque handle uses, target -profiles, and glue-generation requests should produce source-spanned diagnostics -before Wasm emission. - -## Bodyless externals - -Source declarations of the form -`@external(javascript, "module", "name") pub fn f(...) -> ...` are represented -as external functions with target, module, and function metadata. Project and -dependency module interfaces preserve that metadata so a later lowering phase -can select the external that matches the compile target. - -Selected JavaScript externals lower through the same host-import ABI as -handwritten `external fn ... = "module" "name"` declarations. Imported -bodyless externals from project modules or dependency interfaces become -synthetic host-import functions in IR, using generated backend names for linked -modules while preserving the declared JavaScript module and function names. - -If dependency source cannot be compiled and a referenced dependency member is -not represented by selected external metadata, lowering reports a source-spanned -missing-member diagnostic instead of emitting unresolved calls. Unsupported -parameter or return ABI shapes are validated before Wasm emission for both -local external declarations and imported bodyless external metadata. - -## Active tasks - -See [JS host ABI tasks](../tasks/15_js_host_abi.md). diff --git a/docs/internal/specs/16_stdlib_and_host_interop.md b/docs/internal/specs/16_stdlib_and_host_interop.md index 9799af3..7bc042f 100644 --- a/docs/internal/specs/16_stdlib_and_host_interop.md +++ b/docs/internal/specs/16_stdlib_and_host_interop.md @@ -1,109 +1,171 @@ # Standard library and host interop The backend has a raw scalar and managed-pointer ABI for current tests. This -spec tracks the work needed for useful standard library modules and concrete -host interfaces for Wasmtime, WASI, and the browser. +spec tracks the remaining work needed to compile useful upstream +`gleam_stdlib` source, keep host interop explicit, and remove bespoke stdlib +behavior over time. -This is now a foundation and follow-on track, not the next product milestone. -The JS host ABI should come first. Stdlib support should increasingly ride on -normal source compilation, bodyless externals, and JS host interop rather than -bespoke compiler implementations. +The stdlib package is a Hex dependency, but it is not just ordinary Gleam source +for this compiler yet. Upstream stdlib also relies on target-specific +declarations, bodyless runtime types, JavaScript and Erlang shims, native +collection representations, dynamic values, and bit-string/text/binary +primitives. Those shapes must be represented directly before Regulus can treat +the package as another dependency. -## Strategies +The registry remains useful only as a transitional interface and strategy +table. It should bootstrap common programs, describe compiler-owned primitives, +and produce clear diagnostics while upstream source support matures. The end +state is no stdlib registry: stdlib interfaces come from dependency metadata or +compiled package source, and native behavior is represented by ordinary +externals, package assets, or named runtime primitives. -Possible standard library strategies include: +## Strategy -- compile Gleam standard library modules to WASM -- provide selected modules as host imports -- implement selected modules as runtime intrinsics -- combine compiled modules with host shims +Every stdlib member exposed by the compiler should declare one strategy: -The strategy can vary by module, but it should be explicit and tested. +- compiled upstream Gleam source +- temporary interface only +- compiler or runtime intrinsic +- validated stdlib host shim +- adapter around a target host import -Regulus should keep explicit stdlib support narrow and transitional. It exists -to bootstrap common programs, expose compiler-owned primitives, and provide -clear diagnostics while ordinary source and external support matures. +The preferred migration path is to compile upstream source first, then delete +temporary registry behavior module by module. Intrinsics and shims are reserved +for language/runtime primitives or target facilities that normal source cannot +express. Once those primitives are represented directly, the registry should be +removed rather than kept as a compatibility layer. -The current stdlib surface is tracked in two groups. +Unsupported stdlib code should identify the blocker category: -### Group 1: initial useful stdlib +- source language feature +- target selection +- dependency package asset +- runtime primitive +- host or JS ABI shape -Group 1 is the small stdlib surface needed for examples and tests. These -modules should be visible to resolver and type checker through the same module -interface path as project modules. Members that are not implemented yet should -still fail with source-spanned unsupported diagnostics, not during WASM -assembly. +## Upstream compatibility slices -The group contains: +The following slices cover the known upstream stdlib shapes that are not fully +handled yet. -- `gleam/io` -- `gleam/int` -- `gleam/string` -- `gleam/list` -- `gleam/result` -- `gleam/option` -- `gleam/order` +### Source compile readiness -Initial implementations should cover: +Regulus should be able to compile selected modules from the published +`gleam_stdlib` package as fixtures. The first fixtures should use modules with +few native dependencies, such as `gleam/pair`, then expand through pure portions +of `gleam/order`, `gleam/result`, `gleam/option`, `gleam/list`, `gleam/int`, +and `gleam/float`. -| Module | Initial support | -| -------------- | ------------------------------------------- | -| `gleam/io` | `println`, `print`, maybe `debug` | -| `gleam/int` | `to_string`, later `parse` | -| `gleam/string` | `append`, `concat`, `length`, `is_empty` | -| `gleam/list` | `length`, `reverse`, later `map` and `fold` | -| `gleam/result` | `Result`, `Ok`, `Error`, maybe `map` | -| `gleam/option` | `Option`, `Some`, `None`, maybe `map` | -| `gleam/order` | `Order`, `Lt`, `Eq`, `Gt` | +Each fixture should report why compilation stops instead of failing later in +Wasm assembly. The report should group blockers by module and category so the +registry can be deleted deliberately. -Each implemented member should declare one lowering strategy: intrinsic, host -import, compiled Gleam source, or adapter around a host import. +### Target attributes -### Group 2: remaining stdlib +Upstream stdlib uses standalone `@target(erlang)` and `@target(javascript)` +attributes on declarations. Regulus already has target groups for externals, +but upstream stdlib also needs declaration filtering for functions, constants, +types, and externals that share names across targets. -Group 2 is the rest of the Gleam stdlib. It should be expanded after the JS -host ABI and target adapters are useful enough to host real examples. +For JS profiles, upstream `javascript` should select browser, bundler, and +Node.js builds unless a narrower Regulus target rule is added later. -The remaining modules are: +### Bodyless runtime types -- `gleam/bit_array` -- `gleam/bool` -- `gleam/bytes_tree` -- `gleam/dict` -- `gleam/dynamic` -- `gleam/dynamic/decode` -- `gleam/float` -- `gleam/function` -- `gleam/pair` -- `gleam/set` -- `gleam/string_tree` -- `gleam/uri` +Bodyless runtime types such as `Dynamic`, `Dict`, and `StringTree` are ordinary +interfaces for values implemented by the runtime or host shims. Regulus should +preserve them as external type interfaces and require a declared strategy for +operations that create, inspect, or transform those values. -Group 2 should prefer compiling stdlib Gleam source where possible. Runtime -intrinsics and host adapters should be reserved for functions that require -special ABI support, target capabilities, or efficient runtime primitives. +### The `anything` type -The compiler should not reimplement library behavior when the same behavior can -be compiled from Gleam source or expressed through bodyless externals and the JS -host ABI. Unsupported library code should identify the missing language, -dependency, runtime, or ABI feature that blocks ordinary compilation. +Upstream stdlib uses `anything` for native dynamic and inspection boundaries. +Regulus currently treats that shape like a generic type, which makes lowering +fail without monomorphization. The compiler needs an explicit representation +for `anything` as an opaque dynamic boundary type. + +Initial support should be narrow. It may appear in stdlib-native externals such +as dynamic casts, dynamic indexes, and `string.inspect`. User-facing exports or +general host ABI positions using `anything` should remain unsupported until the +ABI semantics are defined. + +### Native stdlib shims + +Upstream JavaScript externals refer to stdlib package assets such as +`../gleam_stdlib.mjs` and `../dict.mjs`. Regulus should either package those +validated stdlib-relative shims or map them to equivalent compiler/runtime +helpers. These paths are dependency assets, not arbitrary application imports. + +The compiler should preserve source module names in diagnostics and metadata +even when it maps an upstream shim to a Regulus helper. + +### Dict and native collections + +`gleam/dict` depends on a native dictionary representation and JavaScript HAMT +and transient dictionary helpers. Regulus can keep a registry-backed dict +surface as a temporary strategy, but upstream source support requires a real +runtime representation or a validated `dict.mjs` shim. + +The dict strategy must define equality and hashing behavior, callback ABI for +fold/map-style operations, transient mutation boundaries, and how dict values +cross the JS host ABI. + +### Dynamic and decoding + +`gleam/dynamic` and `gleam/dynamic/decode` should compile as much Gleam source +as possible. The compiler/runtime owns only the dynamic representation, bridge +from host JSON or JSON text, primitive classification, lookup, traversal, +construction, and any primitive `DecodeError` support required by source. + +Decoder combinators such as `field`, `map`, `then`, `one_of`, and `recursive` +should use normal compiled closure dispatch. Runtime-specific decoder behavior +should be treated as a temporary blocker, not a permanent stdlib strategy. + +The bridge must map JSON values consistently: + +| JSON shape | Dynamic shape | +| ---------- | ------------------------------------------------------------- | +| `null` | dynamic nil/null | +| boolean | dynamic bool | +| number | dynamic int or float, preserving integer values when possible | +| string | dynamic string | +| array | dynamic indexed sequence of dynamic values | +| object | dynamic properties with string keys | + +### Text, binary, and URI primitives + +`gleam/string`, `gleam/string_tree`, `gleam/bytes_tree`, +`gleam/bit_array`, and `gleam/uri` require native primitives that are larger +than the current group-1 string/list helpers. The missing surface includes +Unicode codepoint and grapheme operations, string slicing, byte slicing, +base16/base64 encoding, percent encode/decode, URI parsing, iodata-style byte +and string trees, bit-array concat/slice, and bit-string segment semantics. + +Where upstream source delegates to native helpers, Regulus should choose either +validated stdlib shims or small runtime primitives with the same behavior. + +### Bit-string semantics + +The current bit-array support is useful for tests but is not complete Gleam +bit-string semantics. Upstream `gleam/bit_array` needs sized segment matching, +binary construction rules, byte alignment checks, and diagnostics that name the +unsupported segment form. ## Host ABI -The host ABI should define how values cross the WASM boundary: +The host ABI defines how values cross the Wasm boundary: - scalar values -- strings -- lists and tuples -- custom types +- strings and bit arrays +- lists, tuples, records, and custom values - functions and closures +- dynamic values and opaque native values - errors and panics - memory ownership -- arbitrary managed-value import/export wrappers -Concrete host ABI for Group 1 uses raw WASM values plus runtime adapters: +Concrete low-level host ABI uses raw Wasm values plus runtime adapters: -| Gleam shape | WASM shape | Host rule | +| Gleam shape | Wasm shape | Host rule | | -------------- | ---------- | ---------------------------------- | | `Int` | `i64` | signed 64-bit integer | | `Float` | `f64` | IEEE-754 double | @@ -112,10 +174,10 @@ Concrete host ABI for Group 1 uses raw WASM values plus runtime adapters: | managed values | `i32` | borrowed pointer into guest memory | Managed values include strings, bit arrays, lists, tuples, records, custom -values, functions, errors, and panics. The guest runtime owns these values. -Hosts may read them during and after a call while the instance is alive, but -must not mutate object memory or retain pointers across instance reset or any -future arena reset. +values, functions, errors, panics, dynamic values, and opaque native handles. +The guest runtime owns these values. Hosts may read them while the instance is +alive, but must not mutate object memory or retain pointers across instance +reset or any future arena reset. The runtime exports adapter helpers for low-level hosts: @@ -130,8 +192,8 @@ The runtime exports adapter helpers for low-level hosts: | `__regulus_value_field_i32(ptr, index)` | raw field slot narrowed to `i32` | Compiler code receives host-provided managed values as borrowed `i32` pointers. -The host must only pass pointers to values allocated in the same guest memory -or adapter functions that explicitly document another ownership rule. +The host must only pass pointers allocated in the same guest memory or adapter +values with an explicit ownership rule. Current stdlib host imports are: @@ -146,21 +208,27 @@ produce source-spanned diagnostics before WAT assembly. ## External functions -General Gleam external functions should lower to Wasm imports, not only current -stdlib shims. The compiler should preserve the declared external module and -function name, validate that the selected target accepts that module, and reject -unsupported ABI shapes before byte emission. +General Gleam external functions lower to Wasm imports. The compiler preserves +the declared external module and function name, validates that the selected +target accepts that module, and rejects unsupported ABI shapes before byte +emission. + +Stdlib package shims are a constrained extension of this rule. A dependency may +reference known package assets, but those imports must be validated and either +packaged into JS output or mapped to compiler/runtime helpers. User modules +should not gain arbitrary relative JS imports through this path. Browser examples need imports for fetch, local storage, and browser state. JS-hosted server examples need imports or exported wrappers for request routing -and response construction. These should be ordinary target-specific external -functions with documented adapters, not special cases in user code. +and response construction. These are ordinary target-specific external +functions with documented adapters. ## JS host ABI JavaScript hosts need a stable boundary over the low-level managed-pointer ABI. The shared value rules, browser, bundler, and Node.js profiles, opaque handles, -and JS glue are defined in [JS host ABI](./15_js_host_abi.md). +and JS glue are defined in +[JavaScript host ABI](../../website/development/js_abi.md). ## Higher-order intrinsics and runtime callbacks @@ -172,56 +240,35 @@ callback-taking stdlib members are defined in Unsupported callback shapes should be rejected before WAT assembly with a source-spanned diagnostic naming the intrinsic, closure type, and ABI shape. -## Dynamic values and structured data - -The weather example needs a small JSON decoding path for NWS responses. The -Wisp reference API needs structured output. These needs should be covered by -ordinary language, dependency, runtime, and host ABI support, not by -example-specific compiler features. - -JSON decoding should happen in Gleam by compiling `gleam/dynamic` and -`gleam/dynamic/decode` where possible. The compiler/runtime only owns the -language semantics, dynamic value representation, primitive dynamic operations, -and any target-specific bridge that turns JSON text or host JSON values into -`dynamic.Dynamic`. - -The bridge must map JSON values to dynamic values consistently: - -| JSON shape | Dynamic shape | -| ---------- | ------------------------------------------------------------- | -| `null` | `dynamic.nil()` | -| boolean | dynamic bool | -| number | dynamic int or float, preserving integer values when possible | -| string | dynamic string | -| array | dynamic array/list of dynamic values | -| object | dynamic properties with string keys | - -Runtime helpers should stay primitive: dynamic classification, property lookup, -list traversal, object traversal, value construction, and decode error value -construction where the stdlib needs it. Decoder combinators such as `field`, -`map`, `then`, `one_of`, and `recursive` should run as compiled Gleam code using -normal closure dispatch. - -Unsupported dynamic operations, dependency modules, bridge shapes, and -structured response shapes should fail with source-spanned diagnostics. - ## Runtime scope The runtime is part of the compiler distribution when it supports compiled Gleam semantics or the host ABI. It may own allocation, managed value layout, strings, lists, records, custom values, closures, equality, debug formatting, -panic values, dynamic primitives, and adapter helpers. +panic values, dynamic primitives, opaque native handles, and adapter helpers. The runtime should not own application or library behavior that can be compiled from Gleam source. Networking policy, routing, response construction, JSON -decoder combinator semantics, and product-specific data shaping should stay in -user or dependency modules unless a narrow primitive is required by the ABI. +decoder combinator semantics, URI business rules, and product-specific data +shaping should stay in user or dependency modules unless a narrow primitive is +required by the ABI. ## Diagnostics -Unsupported stdlib modules, dependency modules, host calls, ABI shapes, or -target combinations should produce clear diagnostics rather than failing during -WASM assembly. +Unsupported stdlib modules, dependency modules, package assets, host calls, ABI +shapes, or target combinations should produce clear diagnostics rather than +failing during Wasm assembly. Diagnostics should say whether the missing piece +is source syntax, target filtering, dependency packaging, a runtime primitive, +or a host ABI rule. + +## End state + +The completed design has no stdlib-specific registry. `gleam_stdlib` is loaded +like a dependency package, selected source declarations are compiled normally, +bodyless native types are dependency interfaces, and target shims are validated +package assets or explicit runtime primitives. Any remaining stdlib-specific +table is a bug unless it only names compiler-owned primitives that are also +available through the normal external/runtime mechanism. ## Active tasks diff --git a/docs/internal/specs/18_example_projects.md b/docs/internal/specs/18_example_projects.md index fc30a4d..db943d3 100644 --- a/docs/internal/specs/18_example_projects.md +++ b/docs/internal/specs/18_example_projects.md @@ -27,16 +27,19 @@ The compiler can load selected Hex and path dependency source modules, but it does not compile arbitrary dependency code without subset limits. General external functions lower to Wasm imports when the selected target and -ABI shape are supported. Bodyless `@external` declarations from sources are -still missing. +ABI shape are supported. Bodyless `@external` declarations from project and +dependency source use the same selected target ABI when metadata is available. The low-level host ABI supports scalar values and borrowed managed pointers. -The bundler JavaScript adapter is the current checked host path for strings, -supported structured export results, import/export metadata, and adapter-owned -opaque handle tables. - -Browser-specific host APIs, Node.js loading, structured imports, and -opaque-handle externals are still incomplete. +The generated JavaScript adapter is the checked host path for browser, bundler, +and Node.js targets. It supports strings, selected structured export results, +import/export metadata, standard browser and Node.js import helpers, and +adapter-owned opaque handle tables. + +Structured JavaScript values passed into Gleam imports or exported function +parameters are still deferred. Broader dependency source compilation, full +dynamic decoding, and runtime memory hardening remain the main gaps for larger +examples. `gleam/dynamic`, `gleam/dynamic/decode`, `gleam/uri`, `gleam/pair`, `gleam/set`, `gleam/string_tree`, and `gleam/bytes_tree` are still unsupported diff --git a/docs/internal/tasks/15_js_host_abi.md b/docs/internal/tasks/15_js_host_abi.md deleted file mode 100644 index d27f922..0000000 --- a/docs/internal/tasks/15_js_host_abi.md +++ /dev/null @@ -1,125 +0,0 @@ -# JS host ABI tasks - -## Goal - -Define and implement the first usable JavaScript host ABI for Regulus. This is -the primary usability milestone before broad stdlib completion. - -The public ABI contract lives in -[JavaScript host ABI contract](../../website/development/js_abi.md). - -## Milestone slice - -- [x] Compile a Gleam project with one selected JavaScript external. -- [x] Emit or package bundler-oriented ES module glue for the Wasm artifact. -- [x] Pass a JavaScript string into Gleam through the glue. -- [x] Return a Gleam string to JavaScript through the glue. -- [x] Add one smoke test that runs the whole path without handwritten pointer - arithmetic in application code. - -## Tasks - -### ABI shape - -- [x] Define the JS host ABI document for scalar and managed values. -- [x] Define import module names for shared JS, browser, bundler, and Node.js - profiles. -- [x] Add a profile-selection check that accepts browser, bundler, and Node.js - and rejects unknown JS host profiles. -- [x] Define supported exported function parameter and return shapes for JS - hosts. -- [x] Add source-spanned diagnostics for unsupported JS import and export - shapes. - -### String helpers - -- [x] Export a stable helper for allocating or writing a JS string into guest - memory. -- [x] Export stable helpers for reading string length and bytes from a managed - string pointer. -- [x] Add JS tests for passing strings from JS to Gleam imports and exports. -- [x] Add JS tests for reading strings returned from exported Gleam functions. - -### Managed value readers - -- [x] Export stable helpers for reading managed value tags, arity, and fields. -- [x] Define the JS reader contract for tuples, records, and custom types. -- [x] Define the JS reader contract for lists and lists of strings. -- [x] Define the JS reader contract for `Result` and `Option` values. -- [x] Implement JS adapter readers for tuples, records, and custom types. -- [x] Implement JS adapter readers for lists and lists of strings. -- [x] Implement JS adapter readers for `Result` and `Option` values. -- [x] Add export metadata for structured values consumed by JS readers. -- [x] Allow supported structured return shapes for JS exports once typed readers - are available. -- [x] Add JS tests that read records, lists, `Result`, and `Option` values from - exported Gleam functions. - -### Opaque JS handles - -- [x] Define the runtime representation for opaque host handles. -- [x] Implement the runtime representation for opaque host handles. -- [x] Define ownership and lifetime rules for JS handles passed to Gleam. -- [x] Implement JS handle table ownership and release behavior. -- [x] Add ABI validation for externals that accept or return opaque handles. -- [x] Add JS adapter conversion for opaque handle imports and exports. -- [x] Add e2e tests for passing a handle through Gleam without exposing JS - object internals to guest memory. - -### Bodyless externals - -- [x] Parse bodyless `@external` declarations from project and dependency - source. -- [x] Represent bodyless externals in AST, resolver interfaces, and typed - module interfaces. -- [x] Select the target-specific external before lowering. -- [x] Lower selected JavaScript externals through the same ABI as handwritten - project externals. -- [x] Add diagnostics for missing selected externals and unsupported external - ABI shapes. - -### Browser profile - -- [x] Define browser import module names for fetch, local storage, time, and - online state. -- [x] Implement browser adapter imports for the defined browser APIs. -- [x] Add browser-profile validation for allowed external modules and names. -- [x] Add browser-page glue that instantiates Wasm with checked imports. -- [x] Add a browser smoke test for one string import and one string export. - -### Bundler profile - -- [x] Define ES module glue shape for bundler-based hosts. -- [x] Add bundler-profile validation for allowed external modules and names. -- [x] Add bundler glue that exposes checked imports and typed exported calls. -- [x] Add a bundler smoke test for one string import and one string export. -- [x] Add a bundler fixture for a string request input and string response - output. -- [x] Add a bundler fixture that returns tagged response data across the JS host - ABI. -- [x] Wire bundler glue to structured value readers for tagged response data. - -### Node.js profile - -- [x] Define Node.js loading for generated or packaged `.wasm` files. -- [x] Implement Node.js loading for generated or packaged `.wasm` files. -- [x] Add Node.js-profile validation for allowed external modules and names. -- [x] Add Node.js glue that instantiates Wasm with checked imports. -- [x] Add a Node.js smoke test for one string import and one string export. - -### Generated or packaged JS glue - -- [x] Decide whether the CLI emits JS glue, copies a checked adapter template, - or documents a stable handwritten adapter for the first milestone. -- [ ] Emit or package deterministic JS host adapter files for browser and - Node.js profiles. -- [x] Include import and export metadata needed by generated bundler glue. -- [ ] Include import and export metadata in explicit debug output where useful. -- [x] Generate or expose typed JS wrappers from import and export metadata. -- [x] Add CLI tests for stable bundler JS adapter artifact paths. - -## Done when - -Browser, bundler, and Node.js hosts can call compiled Gleam Wasm through a -documented JS host ABI without handwritten pointer arithmetic in example -application code. diff --git a/docs/internal/tasks/16_stdlib_and_host_interop.md b/docs/internal/tasks/16_stdlib_and_host_interop.md index 8ebf9ec..cb608db 100644 --- a/docs/internal/tasks/16_stdlib_and_host_interop.md +++ b/docs/internal/tasks/16_stdlib_and_host_interop.md @@ -2,16 +2,17 @@ ## Goal -Support useful standard library modules and host calls without making broad -stdlib completion block the JS host ABI milestone. +Compile upstream `gleam_stdlib` from source and remove the stdlib registry. ## Direction - [ ] Keep explicit stdlib intrinsics limited to compiler/runtime primitives. -- [ ] Prefer compiled Gleam source or bodyless externals for library behavior. +- [ ] Prefer compiled upstream Gleam source for library behavior. +- [ ] Use bodyless externals and validated stdlib shims for native behavior. - [ ] Use the JS host ABI for JavaScript-backed stdlib and dependency calls. -- [ ] Treat remaining stdlib modules as follow-on work driven by examples and - dependency gaps, not as a prerequisite for usable JS output. +- [ ] Replace registry behavior module by module as source fixtures compile. +- [ ] Finish by deleting the stdlib registry instead of preserving it as an + interface cache or compatibility table. ## Tasks @@ -31,6 +32,72 @@ stdlib completion block the JS host ABI milestone. module interface schemes. - [x] Add diagnostics for unsupported stdlib modules, functions, types, or target combinations. +- [ ] Mark each registry entry as compiled source, temporary interface, + intrinsic, stdlib shim, or target host adapter. +- [ ] Record the intended replacement path for each temporary registry member. +- [ ] Move compiler-owned primitives out of the stdlib registry and into the + normal runtime or external primitive tables. +- [ ] Delete registry entries once their interfaces come from package metadata + or compiled source. + +### Upstream stdlib audit and migration + +- [ ] Add a fixture that compiles the published upstream `gleam_stdlib` source + as a dependency package. +- [ ] Snapshot the first compile blocker for each upstream stdlib module. +- [ ] Group blocker reports by source language feature, target selection, + dependency package asset, runtime primitive, and ABI shape. +- [ ] Compile `gleam/pair` from upstream source as the first registry-removal + proof. +- [ ] Compile pure portions of `gleam/order`, `gleam/result`, `gleam/option`, + `gleam/list`, `gleam/int`, and `gleam/float` from upstream source where + the current runtime is sufficient. +- [ ] Keep registry-backed behavior only where the blocker report shows a real + native, runtime, target, or ABI dependency. +- [ ] Remove the table-driven stdlib registry once every remaining member is + represented by package source, package metadata, validated shims, or + runtime primitives. + +### Target attributes + +- [x] Filter target-group declarations before typing and lowering so + target-specific externals can reuse local names safely. +- [ ] Preserve standalone `@target(erlang)` and `@target(javascript)` + attributes on parsed declarations. +- [ ] Apply target filtering to upstream functions, constants, types, and + externals, not only grouped externals. +- [ ] Treat upstream `javascript` declarations as available to browser, + bundler, and Node.js profiles. +- [ ] Add duplicate-name fixtures where target selection prevents conflicts, + including the upstream `gleam/set` token shape. +- [ ] Add diagnostics for declarations eliminated by target filtering when they + are referenced from selected code. + +### Bodyless types and `anything` + +- [x] Preserve bodyless runtime types as external type interfaces. +- [ ] Define the internal type representation for `anything`. +- [ ] Allow `anything` in stdlib-native externals such as dynamic casts, + dynamic indexes, and `string.inspect`. +- [ ] Reject unsupported user exports, imports, and general ABI positions that + use `anything`. +- [ ] Add diagnostics that distinguish `anything` from ordinary generic type + variables when lowering cannot support it. +- [ ] Add fixtures for upstream `dynamic.cast`, `dynamic/decode.bare_index`, + and `string.inspect`. + +### Stdlib native shims + +- [ ] Decide whether JS stdlib shims are packaged as source assets, mapped to + runtime helpers, or replaced by compiler intrinsics. +- [ ] Validate stdlib-relative JS external modules such as + `../gleam_stdlib.mjs` and `../dict.mjs` for the stdlib package only. +- [ ] Preserve upstream external module and function names in diagnostics and + JS metadata even when a Regulus helper is used internally. +- [ ] Reject arbitrary user relative JS imports unless a separate package asset + policy defines them. +- [ ] Add fixtures for selected upstream JS externals that exercise stdlib + package asset resolution. ### Group 1: initial useful stdlib @@ -46,6 +113,8 @@ stdlib completion block the JS host ABI milestone. lowering. - [x] Support `gleam/order.Order`, `Lt`, `Eq`, and `Gt` in interfaces and lowering. +- [ ] Replace eligible group-1 registry behavior with compiled upstream source + after the source fixture and native blockers are in place. ### Host ABI @@ -57,7 +126,9 @@ stdlib completion block the JS host ABI milestone. - [x] Define how host code reads compiler memory and how compiler code receives host-provided managed values. - [x] Add rich managed-value wrappers or adapter functions for ABI shapes that - do not map directly to raw WASM parameters and results. + do not map directly to raw Wasm parameters and results. +- [ ] Define how dynamic values and opaque native stdlib values cross the JS + host ABI. ### Intrinsics and host calls @@ -74,13 +145,13 @@ stdlib completion block the JS host ABI milestone. - [x] Lower non-stdlib external functions to Wasm imports. - [x] Preserve external module and function names in import metadata. - [x] Validate external import modules against the selected target. -- [x] Filter target-group declarations before typing and lowering so - target-specific externals can reuse local names safely. - [x] Add a centralized external ABI validator with source-spanned diagnostics for unsupported parameter and return shapes. - [x] Reject unsupported external function ABI shapes before byte emission. - [x] Add table-driven tests for selected target groups, unsupported ABI shapes, supported managed shapes, and JS host imports. +- [ ] Split ordinary user JS import validation from validated dependency + package asset imports. ### Higher-order intrinsics and runtime callbacks @@ -98,22 +169,39 @@ stdlib completion block the JS host ABI milestone. - [x] Add tests for closures passed to intrinsics, captured closures, nested callbacks, generic callbacks, and callback failures. +### Dict and collection runtime + +- [x] Support the current registry-backed `gleam/dict` surface. +- [ ] Define whether upstream dict uses `dict.mjs`, a Regulus runtime helper, + or compiler intrinsics. +- [ ] Implement or shim the native `Dict` and `TransientDict` + representations used by upstream source. +- [ ] Define equality and hashing semantics for dict keys. +- [ ] Support callback ABI shapes needed by dict fold, map, filter, and merge + operations. +- [ ] Define how dict values cross the JS host ABI and structured output + helpers. +- [ ] Add an upstream `gleam/dict` compile fixture and blocker report. +- [ ] Add behavior fixtures for insert, delete, get, fold, merge, equality, + and transient update paths. + ### Dynamic values and structured data - [ ] Define the JSON bridge from host JSON or JSON text to `Dynamic`. - [ ] Map JSON null, bool, number, string, array, and object values to documented dynamic runtime shapes. - [ ] Support full `gleam/dynamic` value construction and classification. +- [ ] Support `anything` in dynamic cast and dynamic index boundaries. - [ ] Implement primitive dynamic runtime operations for classification, - property lookup, list traversal, object traversal, and value construction. -- [ ] Add a compile fixture for the upstream `gleam/dynamic/decode` module. -- [ ] Add a blocker report for that fixture that lists each unsupported syntax, - dependency interface, runtime primitive, and ABI shape encountered. + property lookup, index lookup, null checks, list traversal, object + traversal, and value construction. +- [ ] Add compile fixtures for upstream `gleam/dynamic` and + `gleam/dynamic/decode`. - [ ] Reuse normal closure dispatch for decoder continuations used by `field`, `map`, `then`, `recursive`, and generated record decoders. -- [ ] Implement any missing dynamic primitives required by compiled decoder - source for field lookup, path traversal, collection traversal, error - aggregation, recursion, and primitive custom decoders. +- [ ] Implement missing primitives required by compiled decoder source for path + traversal, collection traversal, error aggregation, recursion, and custom + primitive decoders. - [ ] Construct real `DecodeError(expected, found, path)` values through compiled stdlib code or a documented primitive constructor. - [ ] Add diagnostics for unsupported dynamic operations, dependency modules, @@ -122,14 +210,38 @@ stdlib completion block the JS host ABI milestone. - [ ] Add fixtures for nested objects, optional/null fields, lists, dicts, records, enum variants, `one_of`, and decode error paths. +### Text, binary, and URI primitives + +- [x] Support the current registry-backed `gleam/bit_array` surface. +- [ ] Define native helper strategy for upstream `gleam/string`. +- [ ] Implement or shim Unicode codepoint, grapheme, slicing, replace, split, + casing, trimming, inspect, and parse helpers required by upstream string + source. +- [ ] Implement or shim `gleam/string_tree` and its iodata-style conversion + helpers. +- [ ] Implement or shim `gleam/bytes_tree` and byte-tree flattening helpers. +- [ ] Implement or shim base16, base64, byte slicing, and byte classification + helpers used by upstream stdlib. +- [ ] Implement or shim URI parsing, percent encode/decode, query handling, and + reconstruction helpers needed by `gleam/uri`. +- [ ] Expand bit-string construction and deconstruction to sized segments used + by upstream `gleam/bit_array`. +- [ ] Add diagnostics for unsupported segment options, byte alignment, and + binary pattern forms. +- [ ] Add upstream compile fixtures for `gleam/string`, + `gleam/string_tree`, `gleam/bytes_tree`, `gleam/bit_array`, and + `gleam/uri`. + ### Runtime scope - [ ] Add a runtime helper inventory grouped by allocation, managed values, - closures, equality, debug, dynamic values, and host adapters. + closures, equality, debug, dynamic values, native shims, opaque handles, + and host adapters. - [ ] Add tests that prove dynamic decoder combinators call normal compiled closures rather than runtime-specific callback paths. - [ ] Replace any runtime helper that implements library-level decoder, - routing, or response behavior with a compile fixture for the library code. + routing, URI, or response behavior with a compile fixture for the library + code. - [ ] Add unsupported-feature diagnostics for any runtime primitive requested by compiled library code but not implemented. @@ -140,13 +252,14 @@ stdlib completion block the JS host ABI milestone. - [x] Support `gleam/float`. - [x] Support `gleam/function`. - [x] Support `gleam/bit_array`. -- [ ] Support `gleam/bytes_tree`. -- [ ] Support `gleam/string_tree`. -- [ ] Support full `gleam/dynamic`. -- [ ] Support full `gleam/dynamic/decode`. -- [ ] Support `gleam/pair`. -- [ ] Support `gleam/set`. -- [ ] Support `gleam/uri`. +- [ ] Compile or support `gleam/pair`. +- [ ] Compile or support `gleam/set` after target attributes and dict runtime + support are in place. +- [ ] Compile or support `gleam/bytes_tree`. +- [ ] Compile or support `gleam/string_tree`. +- [ ] Compile or support full `gleam/dynamic`. +- [ ] Compile or support full `gleam/dynamic/decode`. +- [ ] Compile or support `gleam/uri`. - [ ] Prefer compiling stdlib Gleam source or using bodyless externals for Group 2 where possible. - [ ] Add target-specific intrinsics or host adapters only when source @@ -155,5 +268,8 @@ stdlib completion block the JS host ABI milestone. ## Done when Small programs using selected Gleam stdlib functionality compile and execute -against a documented host interface, and unsupported stdlib usage fails with a -specific source-spanned diagnostic. +against a documented host interface, unsupported stdlib usage fails with a +specific source-spanned diagnostic, and the stdlib registry has been removed. +The compiler may still have runtime primitive tables, external import +validation, and dependency package metadata, but none of those tables should be +a bespoke stdlib interface registry. diff --git a/docs/internal/tasks/17_cli_and_build_outputs.md b/docs/internal/tasks/17_cli_and_build_outputs.md index 620877e..34ade8a 100644 --- a/docs/internal/tasks/17_cli_and_build_outputs.md +++ b/docs/internal/tasks/17_cli_and_build_outputs.md @@ -25,12 +25,12 @@ Make compiler commands predictable and generated artifacts useful. - [x] Add optional AST, resolved AST, typed output, and IR debug dumps. - [x] Emit Wasm bytes from a structured module without going through WAT. - [x] Add deterministic WAT snapshots generated from structured Wasm. -- [x] Emit deterministic bundler `.mjs` adapter files when Wasm output is - requested for the `bundler` target. -- [x] Expose bundler import and export metadata needed by checked host calls. +- [x] Emit deterministic `.mjs` adapter files when Wasm output is requested for + browser, bundler, and Node.js targets. +- [x] Expose JS host import and export metadata needed by checked host calls. - [ ] Add optional runtime layout and ABI debug output where helpful. -- [ ] Include import and export metadata in explicit debug output where useful. -- [ ] Emit or package deterministic browser and Node.js host adapter files when +- [x] Include import and export metadata in explicit debug output where useful. +- [x] Emit or package deterministic browser and Node.js host adapter files when requested. ### Backend cleanup @@ -59,7 +59,7 @@ Make compiler commands predictable and generated artifacts useful. - [x] Add CLI integration tests for diagnostics across multiple files. - [x] Add tests for output path handling and optional WAT/debug artifacts. - [ ] Add tests for target selection and unsupported target combinations. -- [ ] Add tests for browser and Node.js JS host profile output. +- [x] Add tests for browser and Node.js JS host profile output. - [x] Add tests for bundler host adapter artifact paths. ## Done when diff --git a/docs/website/changelog.md b/docs/website/changelog.md index 88272e2..3148617 100644 --- a/docs/website/changelog.md +++ b/docs/website/changelog.md @@ -4,6 +4,16 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ## [Unreleased] +### 2026-06-19 + +#### Changed + +- Promoted the JavaScript host ABI contract to the website development docs. + +#### Removed + +- Removed the completed internal JavaScript host ABI spec and task tracker. + ### 2026-06-05 #### Added diff --git a/docs/website/development/js_abi.md b/docs/website/development/js_abi.md index 01da5be..c438efa 100644 --- a/docs/website/development/js_abi.md +++ b/docs/website/development/js_abi.md @@ -8,6 +8,18 @@ This page defines the first JS host ABI. Browser, bundler, and Node.js profiles share these value rules. Profiles only change loading behavior, available host APIs, and accepted import module names. +## Scope + +The ABI covers values and calls crossing between compiled Gleam Wasm and a +JavaScript host. It does not define application logic, browser networking +policy, routing, response construction, or library behavior that can compile +from Gleam source. + +The milestone path is intentionally small: compile a Gleam project to Wasm plus +JS glue, call a selected JavaScript external, pass a JS string into Gleam, +return a Gleam string to JS, and run the path through browser, bundler, and +Node.js profiles without handwritten pointer arithmetic in application code. + ## Ownership model Scalar values cross the boundary as raw Wasm values. Managed Gleam values cross @@ -123,6 +135,28 @@ Scalar `Int` fields are the signed `i64` value. `Bool` fields use `0n` and slot. Managed fields store a borrowed pointer in the low 32 bits. Glue should convert those pointer fields before recursively reading them. +## Generated adapters + +For browser, bundler, and Node.js targets, `compile` and `build` write a +deterministic sibling `.mjs` adapter whenever Wasm output is requested. The +adapter embeds compiler-generated ABI metadata for supported imports and +exports, exposes it as `abi`, and uses the same metadata to wrap host imports +and checked exported calls. + +The adapter is shared across JS profiles. Profile helpers only provide loading +defaults and standard imports: + +- `init(wasm, imports)` for the shared checked instantiation path +- `initBrowserPage(wasm, imports, options)` and `createBrowserImports(options)` +- `initNode(wasm, imports, options)` and `createNodeImports(options)` +- `call(name, ...args)` for metadata-driven export calls +- `callExport(name, params, result, ...args)` for explicit checked calls +- `exportFunction(name, params, result)` for reusable checked wrappers + +Hosts may pass a URL, bytes, an `ArrayBufferView`, or a precompiled +`WebAssembly.Module`. Node.js hosts may also pass a relative or absolute +filesystem path or `file:` URL. + ## Structured readers Generated or packaged JS glue exposes reader helpers over borrowed managed @@ -195,10 +229,10 @@ readValue(ptr, { kind: "List", item: "String" }); readValue(ptr, { kind: "Result", ok: "String", error: "Int" }); ``` -Bundler adapters embed export ABI metadata for supported structured returns. -The metadata uses the same shape objects consumed by `readValue`, so `call` and -`exportFunction` can decode structured return pointers without handwritten -shape arrays. +Browser, bundler, and Node.js adapters embed ABI metadata for supported +imports and exports. The metadata uses the same shape objects consumed by +`readValue`, so `call` and `exportFunction` can decode structured return +pointers without handwritten shape arrays. ## Import modules @@ -255,6 +289,26 @@ conversion as the browser and bundler profiles. Non-JS targets use different modules and are outside this contract. Wasmtime uses `env`, and WASI uses `wasi_snapshot_preview1`. +## Bodyless externals + +Source declarations of the form +`@external(javascript, "module", "name") pub fn f(...) -> ...` are represented +as external functions with target, module, and function metadata. Project and +dependency module interfaces preserve that metadata so lowering can select the +external that matches the compile target. + +Selected JavaScript externals lower through the same host-import ABI as +handwritten `external fn ... = "module" "name"` declarations. Imported +bodyless externals from project modules or dependency interfaces become +synthetic host-import functions in IR, using generated backend names for linked +modules while preserving the declared JavaScript module and function names. + +If dependency source cannot be compiled and a referenced dependency member is +not represented by selected external metadata, lowering reports a +source-spanned missing-member diagnostic instead of emitting unresolved calls. +Unsupported parameter or return ABI shapes are validated before Wasm emission +for both local external declarations and imported bodyless metadata. + ## Imported functions A JavaScript host import is a Gleam `external fn` whose module is accepted by @@ -279,9 +333,8 @@ Supported imported return shapes are: Structured managed values may lower as borrowed pointers internally. Stable JS conversion for imported structured parameters and returns is deferred until -structured writers and generated import metadata are complete. Generic values -and function values are unsupported across JS host imports until their ABI -contracts are defined. +structured writers are complete. Generic values and function values are +unsupported across JS host imports until their ABI contracts are defined. ## Exported functions @@ -311,7 +364,7 @@ Supported exported return shapes for checked call wrappers are: - `Result(a, e)` when `a` and `e` are supported reader shapes - `Option(a)` when `a` is a supported reader shape -Generated bundler metadata maps public export names to parameter and return +Generated JS adapter metadata maps public export names to parameter and return shapes. `call("name", ...args)` uses that metadata to convert scalar parameters, invoke the Wasm export, and decode the return value. @@ -341,7 +394,6 @@ This contract intentionally does not define: - writing structured JavaScript values into Gleam - structured import parameters or returns -- generated binding metadata -Those pieces build on the scalar, string, module-name, and validation contract -above. +Those pieces build on the scalar, string, module-name, metadata, and validation +contract above. diff --git a/docs/website/guide/examples.md b/docs/website/guide/examples.md index bf59f9d..004f661 100644 --- a/docs/website/guide/examples.md +++ b/docs/website/guide/examples.md @@ -8,7 +8,7 @@ compiler's current surface area. ```sh reggie build examples/scalar_project reggie build examples/multi_module_project -reggie build examples/browser_scalar --target browser +reggie build examples/scalar_project --target browser ``` ## Example projects @@ -17,10 +17,12 @@ reggie build examples/browser_scalar --target browser | --- | --- | | `examples/scalar_project` | Smallest normal project build. | | `examples/multi_module_project` | Same-project imports and linked output. | -| `examples/browser_scalar` | Browser-target Wasm with host glue. | | `examples/diagnostics/duplicate_modules` | Intentional project diagnostic. | Diagnostic examples are expected to fail before Wasm emission. +Use `--target browser`, `--target bundler`, or `--target nodejs` with +`examples/scalar_project` to check JS adapter emission without keeping +duplicate scalar examples. ## Roadmap examples diff --git a/docs/website/guide/usage/cli.md b/docs/website/guide/usage/cli.md index f3c67cb..8bc5ba2 100644 --- a/docs/website/guide/usage/cli.md +++ b/docs/website/guide/usage/cli.md @@ -76,7 +76,7 @@ into guest memory at the Wasm boundary. Programs can still print strings through Both `build` and `compile` accept `--target`: ```sh -reggie build examples/browser_scalar --target browser +reggie build examples/scalar_project --target browser reggie build examples/scalar_project --target bundler reggie compile path/to/module.gleam --target wasmtime ``` diff --git a/docs/website/reference/cli-and-build-outputs.md b/docs/website/reference/cli-and-build-outputs.md index 6205b2e..54dc590 100644 --- a/docs/website/reference/cli-and-build-outputs.md +++ b/docs/website/reference/cli-and-build-outputs.md @@ -38,10 +38,12 @@ The `examples/` directory follows this command behavior: - `examples/scalar_project` builds the smallest project Wasm artifact. - `examples/multi_module_project` builds linked same-project modules. -- `examples/browser_scalar` builds a browser-target artifact with handwritten - JS instantiation glue. - `examples/diagnostics/duplicate_modules` documents a failing project shape. +Use `examples/scalar_project --target browser`, `--target bundler`, or +`--target nodejs` to check JS adapter emission without duplicating the scalar +project example. + See `docs/website/guide/usage/cli.md` for a usage guide/manual. ## Targets @@ -51,11 +53,11 @@ See `docs/website/guide/usage/cli.md` for a usage guide/manual. default for `compile`. When `build` omits `--target`, it uses the project target from `gleam.toml`. -The `bundler` target writes a `.mjs` adapter next to the `.wasm` artifact when -Wasm output is requested. The adapter is the current checked JavaScript host -path. It loads the Wasm module, exposes ABI metadata, checks imports, converts -scalar and string calls, reads supported structured export results, and keeps -opaque JavaScript handles in an adapter-owned table. +The `browser`, `bundler`, and `nodejs` targets write a `.mjs` adapter next to +the `.wasm` artifact when Wasm output is requested. The adapter is the current +checked JavaScript host path. It loads the Wasm module, exposes ABI metadata, +checks imports, converts scalar and string calls, reads supported structured +export results, and keeps opaque JavaScript handles in an adapter-owned table. Target selection filters target-group declarations before later compiler phases and checks that host imports are valid for the selected target. diff --git a/docs/website/reference/project-model-and-modules.md b/docs/website/reference/project-model-and-modules.md index d6d607d..b72146e 100644 --- a/docs/website/reference/project-model-and-modules.md +++ b/docs/website/reference/project-model-and-modules.md @@ -64,10 +64,12 @@ The checked examples cover usable project shapes: - `examples/scalar_project` is the smallest working project build. - `examples/multi_module_project` exercises same-project imports. -- `examples/browser_scalar` exercises browser-target scalar Wasm and a small - handwritten host. - `examples/diagnostics/duplicate_modules` shows a project-shape diagnostic. +JS target adapter emission is covered by building `examples/scalar_project` +with a JS target such as `--target browser`, avoiding a duplicate scalar +project. + A larger long-term fixture should exercise real Gleam project structure, dependencies, UI code, records, custom types, and browser-facing Wasm. diff --git a/docs/website/reference/supported-subset.md b/docs/website/reference/supported-subset.md index 1fbd3e7..0d47c21 100644 --- a/docs/website/reference/supported-subset.md +++ b/docs/website/reference/supported-subset.md @@ -17,7 +17,7 @@ Gleam source -> parse -> AST -> resolve -> type check -> IR -> Wasm | Type checking | Broad subset | | Managed runtime values | Partial | | Standard library | Selected modules and intrinsics | -| Browser and JS ABI | Partial | +| JavaScript host ABI | Browser, bundler, and Node.js | | WASI ABI | Incomplete | | Memory management | Bump allocation only | @@ -31,6 +31,8 @@ Gleam source -> parse -> AST -> resolve -> type check -> IR -> Wasm - Filter target-group declarations for Wasmtime, browser, bundler, Node.js, and WASI targets. - Emit optional debug dumps: AST, resolved AST, typed AST, IR, and WAT. +- Emit generated `.mjs` JS host adapters for browser, bundler, and Node.js + targets when Wasm output is requested. ### Language surface @@ -125,6 +127,7 @@ The backend emits deterministic Wasm and optional WAT for: - selected runtime helpers - target-aware host imports - selected stdlib intrinsics +- JS host ABI helpers for strings, structured readers, and opaque handles ABI mapping: @@ -141,6 +144,9 @@ String exports with no parameters also receive: - `__data` - `__len` +JS host builds export stable runtime helpers for writing strings, reading +strings and structured managed values, and wrapping opaque host handles. + ## Partially supported ### Project model @@ -173,15 +179,20 @@ The stdlib registry models selected interfaces and lowering strategies for: - `gleam/order` - `gleam/bool` - `gleam/dict` +- `gleam/dynamic` +- `gleam/dynamic/decode` - `gleam/float` - `gleam/function` - `gleam/bit_array` -This is not full stdlib source compilation. +Some remaining modules are interface-only or unsupported beyond dependency +metadata. This is not full stdlib source compilation. ### Externals and targets General non-stdlib externals lower to Wasm imports when their ABI is supported. +Bodyless `@external` declarations from project and dependency source lower +through the selected target ABI when metadata is available. The compiler: @@ -189,12 +200,15 @@ The compiler: - filters target-specific declarations - validates selected target modules - rejects unsupported ABI shapes before byte emission -- emits bundler-oriented JavaScript adapter glue for the `bundler` target when - Wasm output is requested +- emits generated JavaScript adapter glue for browser, bundler, and Node.js + targets when Wasm output is requested +- embeds import and export metadata for checked JS host calls +- exposes standard browser and Node.js import helpers -The bundler adapter is the most complete JavaScript host path today. Browser -and Node.js targets are accepted and validated, but complete profile-specific -host glue and API adapters are still in progress. +The JavaScript adapter supports scalar and string imports, scalar and string +export parameters, supported structured export returns, and opaque host-handle +conversion. Structured JavaScript values passed into Gleam imports or exported +function parameters are still deferred. ## Not yet supported @@ -202,15 +216,14 @@ host glue and API adapters are still in progress. - Compiling every valid Gleam project shape. - Broad dependency source compilation without subset limits. -- Bodyless `@external` declarations from project and dependency source. - Full stdlib source compilation. ### Host interop -- Full browser, bundler, and Node.js JS ABI. - Complete WASI adapters. -- Rich managed-value wrappers for arbitrary imports and exports. -- Opaque JS handle validation and conversion for external imports and exports. +- Writing arbitrary structured JavaScript values into Gleam. +- Structured JS import parameters and returns. +- Arbitrary managed-value wrappers for every import and export shape. ### Runtime and memory management diff --git a/examples/README.md b/examples/README.md index b142b46..bf69df2 100644 --- a/examples/README.md +++ b/examples/README.md @@ -6,7 +6,6 @@ Examples are small project directories used by users and contributors. - `scalar_project`: smallest project build with scalar exported functions. - `multi_module_project`: same-project import and linked project build. -- `browser_scalar`: browser-target build with minimal JS instantiation glue. ## Diagnostic examples @@ -17,5 +16,7 @@ Examples are small project directories used by users and contributors. No roadmap examples are checked in yet. Add future examples here only when they document planned behavior rather than supported behavior. -Working examples should build with `gleam-wasm build`. Diagnostic examples are -expected to fail with clear diagnostics and should not reach Wasm emission. +Working examples should build with `reggie build`. Use `--target browser`, +`--target bundler`, or `--target nodejs` with `scalar_project` when checking JS +adapter emission. Diagnostic examples are expected to fail with clear +diagnostics and should not reach Wasm emission. diff --git a/examples/browser/host.js b/examples/browser/host.js deleted file mode 100644 index fc01bd6..0000000 --- a/examples/browser/host.js +++ /dev/null @@ -1,124 +0,0 @@ -const textDecoder = new TextDecoder(); - -const TAG = { - STRING: 1, - LIST_CONS: 2, - TUPLE: 3, - RECORD: 4, - CUSTOM: 5, - CLOSURE: 6, - BIT_ARRAY: 7, - OPAQUE: 8, - ERROR: 9, - PANIC: 10, -}; - -export async function instantiateRegulus(source, options = {}) { - let instance; - const imports = createRegulusBrowserImports(() => instance, options); - const result = await instantiate(source, imports); - instance = result.instance; - return result; -} - -export function createRegulusBrowserImports(getInstance, options = {}) { - const write = options.write ?? ((text) => console.log(text)); - const debug = options.debug ?? ((value) => console.debug(value)); - - const memory = () => { - const instance = typeof getInstance === "function" ? getInstance() : getInstance; - if (!instance?.exports?.memory) { - throw new Error("Regulus instance memory is not available"); - } - return instance.exports.memory; - }; - - const readString = (ptr) => readRegulusString(memory(), ptr); - const readDebugValue = (ptr) => inspectRegulusValue(memory(), ptr); - - return { - browser: { - print(ptr) { - write(readString(ptr)); - }, - println(ptr) { - write(`${readString(ptr)}\n`); - }, - debug_i64(value) { - debug(value.toString()); - }, - debug_f64(value) { - debug(value); - }, - debug_bool(value) { - debug(value !== 0); - }, - debug_value(ptr) { - debug(readDebugValue(ptr)); - }, - }, - }; -} - -export function readRegulusString(memory, ptr) { - const view = new DataView(memory.buffer); - const tag = view.getUint32(ptr, true); - if (tag !== TAG.STRING) { - throw new Error(`expected string at ${ptr}, found tag ${tag}`); - } - const len = view.getUint32(ptr + 4, true); - const bytes = new Uint8Array(memory.buffer, ptr + 8, len); - return textDecoder.decode(bytes); -} - -export function inspectRegulusValue(memory, ptr) { - if (ptr === 0) { - return "Nil"; - } - - const view = new DataView(memory.buffer); - const tag = view.getUint32(ptr, true); - const size = view.getUint32(ptr + 4, true); - - switch (tag) { - case TAG.STRING: - return JSON.stringify(readRegulusString(memory, ptr)); - case TAG.LIST_CONS: - return `ListCons(size: ${size}, ptr: ${ptr})`; - case TAG.TUPLE: - return `Tuple(size: ${size}, ptr: ${ptr})`; - case TAG.RECORD: - return `Record(size: ${size}, ptr: ${ptr})`; - case TAG.CUSTOM: { - const constructor = view.getUint32(ptr + 8, true); - return `Custom#${constructor}(size: ${size}, ptr: ${ptr})`; - } - case TAG.CLOSURE: { - const functionId = view.getUint32(ptr + 8, true); - return `Closure#${functionId}(captures: ${size}, ptr: ${ptr})`; - } - case TAG.BIT_ARRAY: - return `BitArray(bits: ${size}, ptr: ${ptr})`; - case TAG.OPAQUE: - return `Opaque(ptr: ${ptr})`; - case TAG.ERROR: - return `Error(size: ${size}, ptr: ${ptr})`; - case TAG.PANIC: - return `Panic(size: ${size}, ptr: ${ptr})`; - default: - return `Unknown(tag: ${tag}, ptr: ${ptr})`; - } -} - -async function instantiate(source, imports) { - if (source instanceof Response) { - return WebAssembly.instantiateStreaming(source, imports); - } - if (typeof source === "string" || source instanceof URL) { - return WebAssembly.instantiateStreaming(fetch(source), imports); - } - if (source instanceof ArrayBuffer || ArrayBuffer.isView(source)) { - return WebAssembly.instantiate(source, imports); - } - throw new TypeError("source must be a URL, Response, ArrayBuffer, or view"); -} diff --git a/examples/browser_scalar/README.md b/examples/browser_scalar/README.md deleted file mode 100644 index e6f4ba8..0000000 --- a/examples/browser_scalar/README.md +++ /dev/null @@ -1,10 +0,0 @@ -# Browser scalar - -This working example builds a browser-target Wasm module without host imports. -The checked-in `host.js` shows the minimal browser instantiation boundary. - -```sh -gleam-wasm build examples/browser_scalar --out-dir build/examples -``` - -The command writes `build/examples/browser_scalar.wasm`. diff --git a/examples/browser_scalar/gleam.toml b/examples/browser_scalar/gleam.toml deleted file mode 100644 index 2cd46ce..0000000 --- a/examples/browser_scalar/gleam.toml +++ /dev/null @@ -1,8 +0,0 @@ -name = "browser_scalar" -version = "1.0.0" -description = "Small browser-target project for Regulus project builds." -licences = ["Apache-2.0"] -target = "javascript" - -[dependencies] -gleam_stdlib = ">= 0.44.0 and < 2.0.0" diff --git a/examples/browser_scalar/host.js b/examples/browser_scalar/host.js deleted file mode 100644 index c8c857a..0000000 --- a/examples/browser_scalar/host.js +++ /dev/null @@ -1,7 +0,0 @@ -async function loadCounter(wasmUrl) { - const bytes = await fetch(wasmUrl).then((response) => response.arrayBuffer()); - const module = await WebAssembly.instantiate(bytes, {}); - return module.instance.exports; -} - -export { loadCounter }; diff --git a/examples/browser_scalar/manifest.toml b/examples/browser_scalar/manifest.toml deleted file mode 100644 index 6e75206..0000000 --- a/examples/browser_scalar/manifest.toml +++ /dev/null @@ -1,14 +0,0 @@ -# Do not manually edit this file, it is managed by Gleam. -# -# This file locks the dependency versions used, to make your build -# deterministic and to prevent unexpected versions from being included -# in your application. -# -# You should check this file into your source control repository. - -packages = [ - { name = "gleam_stdlib", version = "1.0.3", build_tools = ["gleam"], requirements = [], otp_app = "gleam_stdlib", source = "hex", outer_checksum = "1F543AFBA5D33DA493E6087F4E4C4F20D899411343512686C98A8ABB2963CF22" }, -] - -[requirements] -gleam_stdlib = { version = ">= 0.44.0 and < 2.0.0" } diff --git a/examples/browser_scalar/src/main.gleam b/examples/browser_scalar/src/main.gleam deleted file mode 100644 index 22231ba..0000000 --- a/examples/browser_scalar/src/main.gleam +++ /dev/null @@ -1,7 +0,0 @@ -pub fn initial_count() -> Int { - 0 -} - -pub fn increment(count: Int) -> Int { - count + 1 -} diff --git a/examples/diagnostics/duplicate_modules/README.md b/examples/diagnostics/duplicate_modules/README.md index 4aeb6d2..4f10313 100644 --- a/examples/diagnostics/duplicate_modules/README.md +++ b/examples/diagnostics/duplicate_modules/README.md @@ -5,7 +5,7 @@ and `test/app.gleam`. Project loading should reject the project before later compiler phases run. ```sh -gleam-wasm build examples/diagnostics/duplicate_modules +reggie build examples/diagnostics/duplicate_modules ``` Expected result: a duplicate module diagnostic that points at both files. diff --git a/examples/multi_module_project/README.md b/examples/multi_module_project/README.md index 3704cca..1d98b8c 100644 --- a/examples/multi_module_project/README.md +++ b/examples/multi_module_project/README.md @@ -4,7 +4,7 @@ This working example proves that a project build can resolve a same-project import and link the lowered modules into one Wasm artifact. ```sh -gleam-wasm build examples/multi_module_project --out-dir build/examples +reggie build examples/multi_module_project --out-dir build/examples ``` The command writes `build/examples/multi_module_project.wasm`. diff --git a/examples/scalar_project/README.md b/examples/scalar_project/README.md index 5ebf94b..70fd843 100644 --- a/examples/scalar_project/README.md +++ b/examples/scalar_project/README.md @@ -4,7 +4,7 @@ This is the smallest working project build example. It exercises project loading, type checking, lowering, and artifact emission for scalar functions. ```sh -gleam-wasm build examples/scalar_project --out-dir build/examples +reggie build examples/scalar_project --out-dir build/examples ``` The command writes `build/examples/scalar_project.wasm`.