diff --git a/PLAN.md b/PLAN.md index 8e10ad9..6815119 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,7 +1,7 @@ # AArch64 ELF-to-WebAssembly Browser Runtime ## Engineering Build Plan and Agent Handoff -**Status:** Phase 7 complete; Phase 8 next +**Status:** Phase 8 in progress **Primary implementation language:** Rust **Initial browser target:** Google Chrome **Guest architecture:** AArch64, little-endian, Linux userspace @@ -1899,6 +1899,8 @@ Phase 6 is complete for its initial static C scope. `pipe2(O_CLOEXEC)` now suppo Phase 7 is complete for the guest-musl scope. Strict `PT_INTERP` parsing feeds a two-image process loader with correct main-program `AT_ENTRY`, interpreter `AT_BASE`, and initial interpreter PC; the runtime resolves the interpreter from the guest filesystem for initial execution and `execve`. File-backed page-aligned `mmap`, fixed mappings, protection changes, and descriptor-offset preservation let the unmodified musl 1.2.6 dynamic linker map and relocate shared objects itself. A checked-in PIE fixture covers startup TLS, `libanswer.so`, late-loaded plugin TLS, and guest `dlopen`/`dlsym` in native and Chromium-facing sessions. Executable-byte identities keep translated and unsupported-block caches coherent when dynamic mappings replace code. The packaged system image provides a deterministic shared-library cache, and missing interpreters or DSOs retain precise loader/musl diagnostics. Finally, a reproducible dynamic CPython 3.12.13 checkpoint loads shared `libpython3.12.so` and imports `_struct` through CPython's ordinary `dlopen` path, printing `0000002a`. This satisfies all Phase 7 acceptance criteria; Phase 8 begins with a packaged AArch64 Rust host toolchain and Cargo. +The first Phase 8 host-tool checkpoint is complete. `toolchains/rust-musl/build.sh` checksum-pins the official Rust 1.93.0 `aarch64-unknown-linux-musl` rustc, Cargo, and standard-library components, strips host debug data, retains the distributed licenses, and emits an ignored approximately 580 MiB system image. A `libgcc_s.so.1` compatibility DSO is linked from the component's own PIC LLVM libunwind archive. The native verifier runs the dynamically linked rustc and its driver/proc-macro DSOs through the guest musl loader and checks its exact version after 34.7 million guest instructions. Rustc's detached Ctrl-C waiter introduced one tightly bounded musl pthread clone shape: the runtime records a dormant helper TID without scheduling it and still rejects a second or general worker thread. The next checkpoint is a no-dependency source-to-static-ELF compile using one codegen unit and the packaged Clang/musl linker. + Do not begin the full web IDE before item 30 passes. --- diff --git a/README.md b/README.md index ac62e4c..d3b3e91 100644 --- a/README.md +++ b/README.md @@ -106,6 +106,14 @@ shared `libpython3.12.so` and `_struct`; its bounded native verifier imports the extension through CPython's normal `dlopen` path. Both reproducible builds keep their sources, tools, caches, and generated artifacts under `.tmp`. +Phase 8 has started with a checksum-pinned official Rust 1.93.0 AArch64 musl +host distribution. `toolchains/rust-musl/build.sh` packages rustc, Cargo, the +musl standard library, and the required unwind compatibility DSO entirely +under `.tmp`; `toolchains/rust-musl/verify.sh` runs the real dynamically linked +host compiler through the guest loader. Source compilation, Cargo registry +caching, build scripts, and general worker-thread scheduling remain the next +checkpoints. + The native verifier now invokes the Clang driver once rather than manually staging `-cc1` and LLD. Its musl `posix_spawn` path uses an inherited blocking-mode close-on-exec pipe, runs both child executables, and emits a diff --git a/crates/linux-runtime/src/lib.rs b/crates/linux-runtime/src/lib.rs index f627714..4a11469 100644 --- a/crates/linux-runtime/src/lib.rs +++ b/crates/linux-runtime/src/lib.rs @@ -107,6 +107,7 @@ const CLONE_VM: u64 = 0x100; const CLONE_VFORK: u64 = 0x4000; const SIGCHLD: u64 = 17; const SUPPORTED_CLONE_FLAGS: u64 = CLONE_VM | CLONE_VFORK | SIGCHLD; +const DORMANT_THREAD_CLONE_FLAGS: u64 = 0x7d_0f00; const RUSAGE_SIZE: usize = 144; /// Failure while resolving an executable's optional guest-side interpreter. @@ -327,6 +328,7 @@ struct SuspendedParent { config: ProcessConfig, credentials: Credentials, pending_input: Option, + dormant_thread: Option, } #[derive(Clone, Copy, Debug, Eq, PartialEq)] @@ -335,6 +337,11 @@ struct ExitedChild { exit_code: u8, } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +struct DormantThread { + clear_child_tid: GuestAddress, +} + struct DisabledBlockExecutor; impl BlockExecutor for DisabledBlockExecutor { @@ -704,6 +711,7 @@ pub struct Process { parent_process_id: u64, suspended_parent: Option, exited_child: Option, + dormant_thread: Option, } impl Process { @@ -744,6 +752,7 @@ impl Process { parent_process_id: 0, suspended_parent: None, exited_child: None, + dormant_thread: None, }) } @@ -964,6 +973,7 @@ impl Process { Some(Syscall::Exit | Syscall::ExitGroup) => { let exit_code = self.register(0).to_le_bytes()[0]; self.clear_child_tid(); + self.clear_dormant_thread_tid(); self.trace.push(SyscallEvent { number, arguments, @@ -1040,6 +1050,10 @@ impl Process { } fn dispatch_clone(&mut self) { + if self.register(0) == DORMANT_THREAD_CLONE_FLAGS { + self.dispatch_dormant_thread_clone(); + return; + } if self.register(0) != SUPPORTED_CLONE_FLAGS || self.suspended_parent.is_some() || self.exited_child.is_some() @@ -1072,18 +1086,52 @@ impl Process { config: self.config, credentials: self.credentials, pending_input: self.pending_input, + dormant_thread: self.dormant_thread, }; self.suspended_parent = Some(parent); self.current_process_id = CHILD_PROCESS_ID; self.parent_process_id = INITIAL_PROCESS_ID; self.clear_child_tid = None; self.pending_input = None; + self.dormant_thread = None; if child_stack != 0 { self.state.set_sp(GuestAddress::new(child_stack)); } self.set_return(0); } + fn dispatch_dormant_thread_clone(&mut self) { + if self.suspended_parent.is_some() + || self.exited_child.is_some() + || self.dormant_thread.is_some() + { + self.set_return(Errno::TryAgain.return_value()); + return; + } + let stack = self.register(1); + let parent_tid = GuestAddress::new(self.register(2)); + let tls = self.register(3); + let clear_child_tid = GuestAddress::new(self.register(4)); + if stack == 0 + || parent_tid == GuestAddress::NULL + || tls == 0 + || clear_child_tid == GuestAddress::NULL + { + self.set_return(Errno::InvalidArgument.return_value()); + return; + } + let process_id = CHILD_PROCESS_ID; + let process_id_bytes = u32::try_from(process_id) + .expect("the bounded helper thread ID fits u32") + .to_le_bytes(); + if self.memory.write(parent_tid, &process_id_bytes).is_err() { + self.set_return(Errno::Fault.return_value()); + return; + } + self.dormant_thread = Some(DormantThread { clear_child_tid }); + self.set_return(process_id); + } + fn finish_child(&mut self, filesystem: &mut F, exit_code: u8) { self.close_all_descriptors(filesystem); let parent = self @@ -1109,6 +1157,7 @@ impl Process { self.config = parent.config; self.credentials = parent.credentials; self.pending_input = parent.pending_input; + self.dormant_thread = parent.dormant_thread; self.current_process_id = INITIAL_PROCESS_ID; self.parent_process_id = 0; self.exited_child = Some(ExitedChild { @@ -1216,6 +1265,7 @@ impl Process { let interpreter = Interpreter::new().expect("the AArch64 language was initialized for this process"); + self.clear_dormant_thread_tid(); self.close_on_exec_descriptors(filesystem); self.state = Aarch64State::new(image.initial_state.pc, image.initial_state.sp); self.memory = image.memory; @@ -1229,6 +1279,7 @@ impl Process { self.config = image.config; self.credentials = image.credentials; self.pending_input = None; + self.dormant_thread = None; } fn read_exec_vector(&self, address: GuestAddress) -> Result>, Errno> { @@ -2838,6 +2889,15 @@ impl Process { }; let _ = self.memory.write(address, &0_u32.to_le_bytes()); } + + fn clear_dormant_thread_tid(&mut self) { + let Some(thread) = self.dormant_thread.take() else { + return; + }; + let _ = self + .memory + .write(thread.clear_child_tid, &0_u32.to_le_bytes()); + } } fn directory_inode(name: &[u8]) -> u64 { @@ -2968,14 +3028,14 @@ mod tests { use binarrow_runtime_core::{GuestAddress, ResourceLimit, Trap}; use super::{ - AT_FDCWD, AnonymousPipe, CHILD_PROCESS_ID, DESCRIPTOR_CLOEXEC, ExecutionError, - ExecutionEvent, FCNTL_GET_DESCRIPTOR_FLAGS, FCNTL_GET_STATUS_FLAGS, FCNTL_SET_STATUS_FLAGS, - INITIAL_PROCESS_ID, IOCTL_CLEAR_CLOSE_ON_EXEC, IOCTL_SET_CLOSE_ON_EXEC, MAP_FIXED, - MAP_PRIVATE, OPEN_CLOEXEC, OPEN_DIRECTORY, OPEN_NOCTTY, OPEN_NOFOLLOW, OPEN_NONBLOCK, - OPEN_PATH, PAGE_SIZE, PIPE_CAPACITY_BYTES, PROT_EXECUTE, PROT_READ, PipeEnd, Process, - STANDARD_OUTPUT, STAT_CHARACTER_MODE, STAT_FIFO_MODE, STAT_FILE_SIZE_OFFSET, - STAT_MODE_OFFSET, STAT_REGULAR_MODE, STAT_SIZE, SUPPORTED_CLONE_FLAGS, SyscallEvent, - SyscallOutcome, load_process_with_guest_filesystem, + AT_FDCWD, AnonymousPipe, CHILD_PROCESS_ID, DESCRIPTOR_CLOEXEC, DORMANT_THREAD_CLONE_FLAGS, + ExecutionError, ExecutionEvent, FCNTL_GET_DESCRIPTOR_FLAGS, FCNTL_GET_STATUS_FLAGS, + FCNTL_SET_STATUS_FLAGS, INITIAL_PROCESS_ID, IOCTL_CLEAR_CLOSE_ON_EXEC, + IOCTL_SET_CLOSE_ON_EXEC, MAP_FIXED, MAP_PRIVATE, OPEN_CLOEXEC, OPEN_DIRECTORY, OPEN_NOCTTY, + OPEN_NOFOLLOW, OPEN_NONBLOCK, OPEN_PATH, PAGE_SIZE, PIPE_CAPACITY_BYTES, PROT_EXECUTE, + PROT_READ, PipeEnd, Process, STANDARD_OUTPUT, STAT_CHARACTER_MODE, STAT_FIFO_MODE, + STAT_FILE_SIZE_OFFSET, STAT_MODE_OFFSET, STAT_REGULAR_MODE, STAT_SIZE, + SUPPORTED_CLONE_FLAGS, SyscallEvent, SyscallOutcome, load_process_with_guest_filesystem, }; const MESSAGE: &[u8] = b"hello, world\n"; @@ -4135,6 +4195,45 @@ mod tests { assert_eq!(process.register(0), 0); } + #[test] + fn accepts_one_dormant_musl_signal_helper_thread() { + let image = load_hello(ProcessConfig::default(), 1, MESSAGE_ADDRESS); + let mut process = Process::new(image).unwrap(); + let parent_tid = process.state.sp().checked_sub(32).unwrap(); + let clear_child_tid = process.state.sp().checked_sub(28).unwrap(); + process + .memory + .write(clear_child_tid, &CHILD_PROCESS_ID.to_le_bytes()[..4]) + .unwrap(); + process.state.set_x(0, DORMANT_THREAD_CLONE_FLAGS).unwrap(); + process + .state + .set_x(1, process.state.sp().checked_sub(4096).unwrap().get()) + .unwrap(); + process.state.set_x(2, parent_tid.get()).unwrap(); + process.state.set_x(3, 0x1234).unwrap(); + process.state.set_x(4, clear_child_tid.get()).unwrap(); + process.dispatch_clone(); + assert_eq!(process.register(0), CHILD_PROCESS_ID); + let mut tid = [0; 4]; + process.memory.read_exact(parent_tid, &mut tid).unwrap(); + assert_eq!( + u32::from_le_bytes(tid), + u32::try_from(CHILD_PROCESS_ID).unwrap() + ); + + process.state.set_x(0, DORMANT_THREAD_CLONE_FLAGS).unwrap(); + process.dispatch_clone(); + assert_eq!(process.register(0), Errno::TryAgain.return_value()); + + process.clear_dormant_thread_tid(); + process + .memory + .read_exact(clear_child_tid, &mut tid) + .unwrap(); + assert_eq!(u32::from_le_bytes(tid), 0); + } + #[test] fn wait4_preserves_unreaped_status_on_fault() { let image = load_hello(ProcessConfig::default(), 1, MESSAGE_ADDRESS); diff --git a/toolchains/rust-musl/README.md b/toolchains/rust-musl/README.md new file mode 100644 index 0000000..090ede9 --- /dev/null +++ b/toolchains/rust-musl/README.md @@ -0,0 +1,34 @@ +# AArch64 Rust host toolchain + +This Phase 8 checkpoint packages the official Rust 1.93.0 +`aarch64-unknown-linux-musl` host compiler, Cargo, and target standard library +as an immutable Binarrow filesystem image. All three January 22, 2026 Rust +distribution archives are pinned by SHA-256. Downloads, extracted components, +the roughly 580 MiB stripped image, and every cache stay under the repository's +ignored `.tmp` directory. + +Prerequisites: + +- Zig 0.16.0 +- `llvm-objcopy` +- the repository Rust toolchain +- `curl`, `make`, `tar`, and `shasum` + +Build and run the initial host-tool verification: + +```sh +toolchains/rust-musl/build.sh +toolchains/rust-musl/verify.sh +``` + +Rust's host driver expects `libgcc_s.so.1`; the package produces that +compatibility DSO from the PIC LLVM `libunwind.a` already distributed in the +official musl standard-library component. The guest musl loader resolves the +driver, compiler proc-macro DSOs, libc, and unwind runtime. + +The current process checkpoint admits exactly one dormant detached pthread +shape used by rustc's Ctrl-C signal waiter. It writes the parent TID and clears +the child TID at process teardown, but deliberately does not schedule the +helper. General worker threads and futex scheduling remain unsupported. The +next checkpoint is a no-dependency source-to-static-ELF compile with a +single-codegen-unit policy and the packaged Clang/musl linker. diff --git a/toolchains/rust-musl/build.sh b/toolchains/rust-musl/build.sh new file mode 100755 index 0000000..123b6ae --- /dev/null +++ b/toolchains/rust-musl/build.sh @@ -0,0 +1,152 @@ +#!/bin/sh +set -eu + +toolchain_directory=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +workspace_directory=$(CDPATH= cd -- "$toolchain_directory/../.." && pwd) +work_directory=$workspace_directory/.tmp/rust-1.93.0-dist +archive_directory=$work_directory/archives +extract_directory=$work_directory/extracted +image_root=$work_directory/image-root +filesystem_image=$work_directory/rust-toolchain.bnfs +installation_marker=$work_directory/.binarrow-rust-1.93.0-installed +os_cache_directory=$work_directory/os-cache +os_temp_directory=$work_directory/os-temp +zig_cache_directory=$work_directory/zig-cache +cargo_home=$workspace_directory/.tmp/cargo-home +cargo_target=$workspace_directory/.tmp/cargo-target +rust_tmp=$workspace_directory/.tmp/rust-tmp +dist_url=https://static.rust-lang.org/dist/2026-01-22 +rustc_archive=rustc-1.93.0-aarch64-unknown-linux-musl.tar.xz +rustc_sha256=5371915850179d910d3eca32cb8f9240c336a1fdc830286242daf4b26227295f +cargo_archive=cargo-1.93.0-aarch64-unknown-linux-musl.tar.xz +cargo_sha256=da3d215cc53cb72e203c25db874e8140b71945d0be1d409045f9e2b32a6e2345 +stdlib_archive=rust-std-1.93.0-aarch64-unknown-linux-musl.tar.xz +stdlib_sha256=bab885a87da586040064064bd1c314d707164d8dc0fefee39d59be7f15ce6f7d + +zig=$(command -v zig || true) +if [ -z "$zig" ] || [ "$($zig version)" != "0.16.0" ]; then + echo "rust-musl requires Zig 0.16.0" >&2 + exit 1 +fi + +llvm_objcopy=${LLVM_OBJCOPY:-} +if [ -z "$llvm_objcopy" ]; then + llvm_objcopy=$(command -v llvm-objcopy || true) +fi +if [ -z "$llvm_objcopy" ]; then + for candidate in \ + /opt/homebrew/opt/llvm/bin/llvm-objcopy \ + /opt/homebrew/opt/llvm@21/bin/llvm-objcopy \ + /usr/local/opt/llvm/bin/llvm-objcopy \ + /usr/local/opt/llvm@21/bin/llvm-objcopy + do + if [ -x "$candidate" ]; then + llvm_objcopy=$candidate + break + fi + done +fi +if [ -z "$llvm_objcopy" ]; then + echo "rust-musl requires llvm-objcopy (or LLVM_OBJCOPY)" >&2 + exit 1 +fi + +mkdir -p \ + "$archive_directory" \ + "$extract_directory" \ + "$image_root" \ + "$os_cache_directory" \ + "$os_temp_directory" \ + "$zig_cache_directory/local" \ + "$zig_cache_directory/global" \ + "$cargo_home" \ + "$cargo_target" \ + "$rust_tmp" +export TMPDIR=$os_temp_directory +export TMP=$os_temp_directory +export TEMP=$os_temp_directory +export XDG_CACHE_HOME=$os_cache_directory +export DARWIN_USER_TEMP_DIR=$os_temp_directory +export DARWIN_USER_CACHE_DIR=$os_cache_directory +export ZIG_LOCAL_CACHE_DIR=$zig_cache_directory/local +export ZIG_GLOBAL_CACHE_DIR=$zig_cache_directory/global + +fetch_archive() { + name=$1 + expected_sha256=$2 + archive=$archive_directory/$name + if [ ! -f "$archive" ]; then + partial_archive=$archive.part + curl --fail --location --retry 3 --continue-at - \ + --output "$partial_archive" "$dist_url/$name" + mv "$partial_archive" "$archive" + fi + actual_sha256=$(shasum -a 256 "$archive" | awk '{print $1}') + if [ "$actual_sha256" != "$expected_sha256" ]; then + echo "unexpected $name checksum: $actual_sha256" >&2 + exit 1 + fi +} + +extract_component() { + name=$1 + directory=${name%.tar.xz} + if [ ! -x "$extract_directory/$directory/install.sh" ]; then + tar -xJf "$archive_directory/$name" -C "$extract_directory" + fi +} + +fetch_archive "$rustc_archive" "$rustc_sha256" +fetch_archive "$cargo_archive" "$cargo_sha256" +fetch_archive "$stdlib_archive" "$stdlib_sha256" +extract_component "$rustc_archive" +extract_component "$cargo_archive" +extract_component "$stdlib_archive" + +if [ ! -f "$installation_marker" ]; then + for component in \ + "${rustc_archive%.tar.xz}" \ + "${cargo_archive%.tar.xz}" \ + "${stdlib_archive%.tar.xz}" + do + sh "$extract_directory/$component/install.sh" \ + --destdir="$image_root" \ + --prefix=/usr/local \ + --disable-ldconfig + done + touch "$installation_marker" +fi + +# Rust's musl host driver expects libgcc_s, while its target standard library +# ships a PIC LLVM libunwind archive. Export that archive under the expected +# compatibility SONAME instead of adding a second C runtime distribution. +libunwind=$image_root/usr/local/lib/rustlib/aarch64-unknown-linux-musl/lib/self-contained/libunwind.a +"$zig" ld.lld \ + -shared \ + -soname libgcc_s.so.1 \ + --whole-archive "$libunwind" \ + --no-whole-archive \ + -o "$image_root/usr/local/lib/libgcc_s.so.1" + +find "$image_root/usr/local/lib" -maxdepth 1 -type f -name '*.so' \ + -exec "$llvm_objcopy" --strip-all {} \; +"$llvm_objcopy" --strip-all "$image_root/usr/local/bin/cargo" +"$llvm_objcopy" --strip-all "$image_root/usr/local/bin/rustc" +"$llvm_objcopy" --strip-all "$image_root/usr/local/lib/libgcc_s.so.1" +rm -f \ + "$image_root/usr/local/bin/rustdoc" \ + "$image_root/usr/local/bin/rust-gdb" \ + "$image_root/usr/local/bin/rust-gdbgui" \ + "$image_root/usr/local/bin/rust-lldb" + +"$workspace_directory/guest-tests/dynamic-musl/build.sh" +CARGO_HOME="$cargo_home" CARGO_TARGET_DIR="$cargo_target" \ + CARGO_NET_OFFLINE=true cargo build \ + --manifest-path "$workspace_directory/Cargo.toml" \ + --release -p binarrow-cli +"$cargo_target/release/binarrow" image pack \ + --guest-root / \ + "$image_root" \ + "$filesystem_image" + +echo "Rust toolchain filesystem image: $filesystem_image" diff --git a/toolchains/rust-musl/verify.sh b/toolchains/rust-musl/verify.sh new file mode 100755 index 0000000..9a006c0 --- /dev/null +++ b/toolchains/rust-musl/verify.sh @@ -0,0 +1,48 @@ +#!/bin/sh +set -eu + +toolchain_directory=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +workspace_directory=$(CDPATH= cd -- "$toolchain_directory/../.." && pwd) +work_directory=$workspace_directory/.tmp/rust-1.93.0-dist +filesystem_image=$work_directory/rust-toolchain.bnfs +rustc=$work_directory/image-root/usr/local/bin/rustc +system_image=$workspace_directory/guest-tests/dynamic-musl/dynamic-musl.bnfs +cargo_home=$workspace_directory/.tmp/cargo-home +cargo_target=$workspace_directory/.tmp/cargo-target +rust_tmp=$workspace_directory/.tmp/rust-tmp +runner=$cargo_target/release/binarrow + +for artifact in "$filesystem_image" "$rustc" "$system_image"; do + if [ ! -f "$artifact" ]; then + echo "build the Rust toolchain first with toolchains/rust-musl/build.sh" >&2 + exit 1 + fi +done + +mkdir -p "$cargo_home" "$cargo_target" "$rust_tmp" +export TMPDIR=$rust_tmp +export TMP=$rust_tmp +export TEMP=$rust_tmp + +CARGO_HOME="$cargo_home" CARGO_TARGET_DIR="$cargo_target" \ + CARGO_NET_OFFLINE=true cargo build \ + --manifest-path "$workspace_directory/Cargo.toml" \ + --release -p binarrow-cli + +version=$( + "$runner" run \ + --instruction-budget 50000000 \ + --filesystem-limit 838860800 \ + --filesystem-install "$system_image" \ + --filesystem-install "$filesystem_image" \ + --random-seed 1 \ + --argv0 /usr/local/bin/rustc \ + --env PATH=/usr/local/bin:/usr/bin:/bin \ + "$rustc" --version +) +if [ "$version" != "rustc 1.93.0 (254b59607 2026-01-19)" ]; then + echo "unexpected guest rustc version: $version" >&2 + exit 1 +fi + +echo "AArch64 musl rustc host fixture passed"