tsiry-sandratraina.com / app-elfloader
Load and execute Linux ELF binaries -> Unikraft
Star 0
Fork 0
Watch
Rust 71%
C 17%
8%
Shell 2%
Python 1%
Linker Script <1%
Makefile <1%
11 1 0
.github
autogen
defconfigs
example
glue
rust
scripts
support
vdso
.gitignore
Config.uk
exportsyms.uk
Kraftfile
Makefile
Makefile.uk
README.md
README.md

app-elfloader-rs #

A Rust rewrite of Unikraft's app-elfloader: the unikernel that loads an unmodified Linux ELF executable, side-loads its dynamic linker, and starts it.

It is a drop-in replacement. The Kconfig surface is deliberately identical, so an existing Kraftfile or defconfig needs no changes — with one subtraction: lib-libelf is no longer a dependency, because the loader parses ELF itself.

make test        # the loader's own tests, on your machine — no Unikraft needed

Why #

The C loader worked on x86_64 and did not work on arm64, and the reasons were hard to see because almost everything it did was correct for the one case it was exercised against: a position-independent, page-aligned, x86_64 binary. The rewrite is about the cases that are not that — 64 KiB segment alignment over a 4 KiB page, non-PIE, an image whose headers say something the file cannot back up — and about making them checkable without booting a unikernel.

What is different #

Each of these is a behaviour change, not a translation. The tests named in the right-hand column are the ones that pin it.
The image's address range is never left unreserved. The C loader sized a scratch mapping, munmap'd it, then mapped each segment into the address it had been handed. Unikraft's uk_vma_map() picks addresses first-fit from a fixed base (vmem_first_fit), so every moment the range was free — and every inter-segment gap, permanently — was an opportunity for an unrelated mmap(NULL, …) to be placed inside the program image. Here the reservation is taken once as PROT_NONE and each segment is mapped into it with MAP_FIXED. loader.rs image_range_is_never_left_unreserved, a_later_anonymous_mapping_cannot_land_inside_the_image
Segment protections follow PF_*. The C loader wrote if (phdr->p_flags & PROT_EXEC), testing ELF segment flags against an mmap constant. PROT_EXEC is 4, but PF_X is 1 and 4 is PF_R, so every readable segment came out executable and no segment's X bit was ever consulted. layout.rs prot_bits_follow_pf_flags_not_prot_constants, final_protections_match_the_segment_flags
Non-PIE (ET_EXEC) loads. The C loader rejected anything that was not ET_DYN. It also computed the image length as PAGE_ALIGN_UP(highest vaddr), ignoring the lowest — so a binary linked at 0x400000 would have reserved its link address plus its size, from zero. layout.rs non_pie_keeps_its_link_address, non_pie_loads_at_its_link_address
.bss is cleared even in a read-only segment. The C loader zeroed the tail of the last file page only when the segment was writable, leaving whatever followed the segment in the file visible to the program. Here the mapping is made writable for as long as the clearing takes and then narrowed. loader.rs read_only_segment_is_narrowed_after_zeroing, bss_tail_is_zeroed_and_file_bytes_do_not_leak
AT_BASE is the interpreter's load bias, not the address of its first mapped segment. Those coincide only while the first PT_LOAD has p_vaddr == 0, which every shipped dynamic linker does and nothing guarantees. program.rs at_base_is_the_interpreter_bias_not_its_first_segment
AT_RANDOM points into the application's own stack. The C loader pointed it at a local variable of its main(); in the binfmt path that function returned before the application ever read it, so glibc's stack canary and pointer-mangling key came out of a dead kernel stack frame. auxv.rs at_random_points_at_sixteen_readable_bytes_on_the_stack
AT_HWCAP describes the machine. The C loader passed 0. glibc and OpenSSL both dispatch on it, so zero means node uses the generic AES and SHA paths on a CPU that has the instructions. Now derived from ID_AA64ISAR0_EL1, and gated on CONFIG_FPSIMD — advertising SIMD features while FP/SIMD is trapped would turn a slow path into a crash. glue/shim.c —
AT_PAGESZ is the real page size rather than a hard-coded 4096, and AT_PHDR prefers PT_PHDR before falling back to the covering PT_LOAD. AT_PHDR/PHENT/PHNUM are omitted entirely when the headers are not mapped, instead of being reported as 0 — musl treats a present-but-null AT_PHDR as a valid pointer. auxv.rs, layout.rs omits_phdr_entries_when_the_headers_are_not_mapped, phdr_addr_prefers_pt_phdr_then_falls_back
The stack cannot be overrun. Every write is bounds-checked, and the size policy counts NUL terminators and the pointer arrays — the C version summed strlen() only, so a large number of short environment variables could pass the check and still not fit. auxv.rs oversized_environment_is_rejected_not_overflowed, size_check_counts_terminators_and_pointers
Headers are validated against the file. p_offset + p_filesz past the end, p_filesz > p_memsz, p_vaddr + p_memsz overflowing, overlapping PT_LOADs, a p_vaddr/p_offset pair that is not page-congruent (which cannot be produced by one mmap, and which the C loader mapped anyway, shifted) — all rejected before anything is mapped. PT_INTERP is read through the normal file interface rather than through libelf's private e_rawfile, which is only populated for in-memory images and was a dangling read from the elf_open(fd) path. elf.rs 13 rejection tests
arm64 instruction-cache maintenance. Instruction and data caches are not coherent on arm64: code written through the data side is not visible to instruction fetch until it has been cleaned to the point of unification and the I-cache invalidated. The loader now does this for anything it writes itself. The demand-paging path needs the same thing one level down, in Unikraft — see below. glue/shim.c, loader.rs copy_path_loads_and_syncs_the_icache
An executable-stack request is refused rather than silently ignored. Unikraft's thread stacks are not executable and cannot be made so. loader.rs an_executable_stack_request_is_refused
CONFIG_APPELFLOADER_DEBUG actually prints. The kernel log level is a Kconfig choice, which a Kraftfile cannot set, so it sits at KLVL_ERR and filtered every message the option enabled. Loader messages are now forced through when the option is on. glue/shim.c —

Two Unikraft-side fixes go with this; they live in bsdkrun's library/unikraft-base/patches/apply.sh because they are not this repository's to make:

  • invalidate_icache_range() invalidated the I-cache before cleaning the D-cache for the same line, and strode by the I-cache line size while doing both. Both halves are wrong for publishing new code.
  • ukvmem never did any I-cache maintenance when it populated a page, so demand-paged .text could be fetched stale from whatever previously occupied the physical frame. Linux does this in set_pte_at().

Layout #
rust/src/elf.rs bounds-checked ELF64 header parsing
rust/src/layout.rs headers → a concrete plan of mappings
rust/src/loader.rs executes a plan against a Vm
rust/src/auxv.rs builds the initial stack (argv/envp/auxv)
rust/src/program.rs program + interpreter, and the auxv they imply
rust/src/sys.rs the Unikraft implementations of those traits
rust/src/app.rs the main() the unikernel boots into
rust/src/binfmt.rs libukbinfmt registration, for execve
glue/shim.c the only C: everything Unikraft exposes as a macro, a static inline or a struct layout

The split is the point. Everything that decides what to map is pure and runs under cargo test on a developer machine, against a model address space that reproduces Unikraft's first-fit placement. The cases that are painful to reach by booting a unikernel — a non-PIE at its link address, 64 KiB segment alignment over a 4 KiB page, a truncated or hostile header, an interpreter with a non-zero first p_vaddr — are covered there.

glue/shim.c exists so that no Rust code encodes a Unikraft struct offset. A Unikraft API change is then a compile error in that file, at the line that uses it, rather than a silently wrong offset at run time.

Building #

The Rust core is a no_std staticlib for aarch64-unknown-none-softfloat or x86_64-unknown-none — bare-metal targets, because Unikraft compiles kernel code without FP/SIMD (-mgeneral-regs-only on arm64, -mno-sse on x86_64) and those targets match that ABI. Makefile.uk drives cargo and hands Unikraft the archive through ALIBS-y; the crate has no dependencies, so the build is offline.

You need cargo on the machine that runs the Unikraft build. Makefile.uk fails with an install hint if it is missing.

Status #

x86_64: builds and boots.

arm64: builds and boots, and dynamically linked C and C++ programs run.

Getting there turned up two arm64 bugs in Unikraft, neither of them in the loader — the C app-elfloader failed identically:

  • struct stat is the x86_64 layout on every architecture. vfscore's vn_stat() does memset(st, 0, sizeof(struct stat)) on the application's buffer, and Unikraft's definition is 144 bytes where arm64 uses 128, so every stat()/fstat() overwrote 16 bytes past the end of it. In musl's load_library() that is the saved frame pointer and return address, so any binary loading a second shared object returned to address 0.

  • The arm64 signal trampoline has never assembled — it uses SP as an operand of AND — so CONFIG_LIBPOSIX_PROCESS_SIGNAL could not be enabled and no CPU fault reached the application as a signal.

Both fixes live in bsdkrun's library/unikraft-base/patches/apply.sh. node now reaches OpenSSL's SM3 probe, which executes an undefined instruction on purpose and expects SIGILL; that hits a third gap in the same arm64 signal path. See ../bsdkrun/examples/unikraft-expressjs/repro/README.md.