simdutf #

OCaml bindings to simdutf for SIMD-accelerated Unicode validation, transcoding, encoding detection, code point counting, and Base64 encoding/decoding.

This project follows simdjsont: Dune builds C++17 stubs against a pinned, vendored upstream amalgamation. It includes the official simdutf 9.2.1 header and implementation, with no dependency on an installed simdutf library and no downloads during builds. Provenance and SHA-256 checksums are recorded in lib/simdutf_vendor.md.

Installation #

You need OCaml 5.2 or newer, Dune 3.20 or newer, and a C++17 compiler with the C++ standard library. The build uses -lstdc++, following simdjsont's toolchain convention. SIMD support is selected at runtime; no -march=native flag is required.

Install from this checkout:

opam install .

For development, install the test and documentation dependencies and build:

opam install . --deps-only --with-test --with-doc
dune build
dune runtest
dune build @doc

Add the library to your application's Dune stanza:

(executable
 (name example)
 (libraries simdutf))

Quick start #

let () =
  assert (Simdutf.is_valid "Hello, 世界 🌍");
  assert (Simdutf.count_codepoints "Hello, 世界 🌍" = Ok 11);

  let utf16 =
    Simdutf.transcode_exn ~src:Utf8 ~dst:Utf16le "Hello, 世界 🌍"
  in
  assert (
    Simdutf.transcode ~src:Utf16le ~dst:Utf8 utf16 = Ok "Hello, 世界 🌍");

  assert (Simdutf.Base64.encode "hello" = "aGVsbG8=");
  assert (Simdutf.Base64.decode "aGVsbG8=" = Ok "hello")

Encodings and validation #

Simdutf.Encoding.t supports all conversions between these encodings:

Constructor	Representation in an OCaml string
Utf8	UTF-8 bytes
Utf16le	UTF-16 bytes, little endian
Utf16be	UTF-16 bytes, big endian
Utf32le	UTF-32 bytes, little endian
Utf32be	UTF-32 bytes, big endian
Latin1	ISO-8859-1 bytes

Strings are length-delimited bytes: embedded NULs are preserved, and no special padding or terminator is needed. UTF-16/32 byte order is explicit and independent of the host. Latin-1 accepts every byte value. Use is_ascii or validate_ascii to check the ASCII subset.

validate, is_valid, and count_codepoints default to UTF-8. Pass ~encoding to process another encoding:

let () =
  assert (Simdutf.is_valid ~encoding:Utf16le "A\x00");
  assert (Simdutf.is_ascii "hello");
  match Simdutf.validate "ok\xff" with
  | Ok () -> ()
  | Error e -> Format.eprintf "%a\n" Simdutf.pp_error e
  (* Invalid UTF-8 leading byte at byte 2 *)

Errors contain { code; offset; message }. Unicode offsets are zero-based input byte offsets, including for UTF-16 and UTF-32. Partial UTF-16/32 code units produce Incomplete_code_unit at the start of the trailing partial unit. Conversions validate even when source and destination encodings match. Invalid Unicode and characters unrepresentable in Latin-1 return an error, with no partial output exposed. The _exn helpers raise Simdutf.Error with the same error record.

Code point counting validates first. It counts Unicode scalar values, rather than bytes, UTF-16 units, or user-perceived characters (grapheme clusters).

Byte order marks are preserved as ordinary code points. Conversions do not add or strip them, and do not perform Unicode normalization.

Slices #

Validation, counting, transcoding, and Base64 decoding accept ~off and ~len to process a range of a larger string without copying it:

let () =
  let message = "Subject: caf\xc3\xa9\r\n" in
  assert (Simdutf.is_valid ~off:9 ~len:5 message);
  match Simdutf.validate ~off:9 ~len:4 message with
  | Error { code = Too_short; offset = 3; _ } -> ()
  | _ -> failwith "truncated slice"

off defaults to zero and len to the remaining bytes. Bounds are checked; negative or out-of-range slices raise Invalid_argument. Error offsets are relative to the start of the slice.

Byte search #

find returns the index of the first occurrence of a byte, or -1. The index is counted from the start of the string, as with String.index_from, so a scan continues after each match:

let count_lines s =
  let rec go off n =
    match Simdutf.find ~off s '\n' with -1 -> n | i -> go (i + 1) (n + 1)
  in
  go 0 0

find compares bytes exactly and does not decode UTF-8; for an ASCII target such as '\n' this is also safe on UTF-8 text, whose multi-byte sequences contain no ASCII bytes. Slices shorter than 1 KiB use the C library's memchr, since upstream's search spends most of a short call aligning to 64 bytes; longer slices use upstream's SIMD search. find never releases the runtime lock.

find_utf16 ~encoding searches UTF-16LE or UTF-16BE data for a 16-bit code unit, stepping two bytes at a time from off, and returns its byte index. It does not validate the input. Both functions accept ~off and ~len, have bigstring counterparts, and do not allocate. Unaligned UTF-16 slices are searched through an aligned buffer, without converting the data.

Encoding detection #

let candidates = Simdutf.Encoding.detect "hello"

This exposes upstream's heuristic detector. It may report several possible Unicode encodings, or none; a byte order mark takes precedence. Detection does not validate the text after a BOM, so validate using the selected encoding before use. Latin-1 is never inferred because any byte string is valid Latin-1.

Base64 #

let encoded = Simdutf.Base64.encode ~alphabet:Url "\xfb\xff"
(* "-_8" *)

let padded = Simdutf.Base64.encode ~alphabet:Url ~padding:true "\xfb\xff"
(* "-_8=" *)

let decoded = Simdutf.Base64.decode ~alphabet:Url "-_8"
(* Ok "\xfb\xff" *)

let strict = Simdutf.Base64.decode ~last_chunk:Strict "Zg=="
(* Ok "f" *)

The standard alphabet uses + and / and defaults to padding. The URL alphabet uses - and _ and defaults to no padding. ~padding overrides either default.

Decoding defaults to the standard alphabet and Loose final-group handling. Both modes accept exactly this input:

• ASCII whitespace (space, tab, LF, CR, form feed) is ignored anywhere, including between and after padding characters. Any other byte outside the alphabet, including vertical tab and NUL, is an error.
• The other characters form groups of four alphabet characters, followed by an optional final group. A final group of one character is an error.
• A final group of two or three characters may be completed with = padding to four characters: xx== or xxx=. Partial padding (xx=) and padding after a full group (xxxx=) are errors.
• Only whitespace may follow padding, so xx== followed by more data is an error.
• Empty or whitespace-only input decodes to "".

Loose accepts a final group of two or three characters with or without padding, and ignores the unused low bits of its last character (Zh decodes to f). Strict rejects an unpadded final group and non-zero unused bits. Neither mode accepts the other alphabet's characters. Base64 errors use upstream's input byte position, which can identify the start of a final group or the end of the input. test/test_slices.ml checks these rules against an independent reference decoder on random input.

Base64.decode allocates its result at the exact size. To decode a range of a larger string into a reusable buffer, size it with Base64.decoded_length, which is exact for valid input, including whitespace:

let decode_body message ~off ~len buffer =
  let n = Simdutf.Base64.decoded_length ~off ~len message in
  if Bytes.length !buffer < n then buffer := Bytes.create n;
  match Simdutf.Base64.decode_into ~off ~len ~output:!buffer message with
  | Ok written -> Ok (Bytes.sub_string !buffer 0 written)
  | Error e -> Error e

Base64.decode_into has the same output rules as the bigstring _into functions described below: ~output_off and ~output_len select the output slice, insufficient capacity returns Output_buffer_too_small without writing, and malformed Base64 may leave garbage within the output slice.

Bigstrings #

Every text/binary processing operation has a counterpart under Simdutf.Bigstring. These use the standard bigstring type, compatible with Bigstringaf, Lwt_bytes, Core.Bigstring, and simdjsont:

(char, Bigarray.int8_unsigned_elt, Bigarray.c_layout) Bigarray.Array1.t

You can consume network buffers or memory-mapped files and produce bigstrings without copying through an OCaml string. All input operations accept checked off/len slices:

let () =
  let b = Simdutf.Bigstring.of_string "!Hello, 世界 🌍!" in
  assert (
    Simdutf.Bigstring.validate
      ~off:1 ~len:(Bigarray.Array1.dim b - 2) b = Ok ());

  let utf16 =
    Simdutf.Bigstring.transcode_exn ~src:Utf8 ~dst:Utf16le
      ~off:1 ~len:(Bigarray.Array1.dim b - 2) b
  in
  let encoded = Simdutf.Bigstring.Base64.encode utf16 in
  let decoded = Simdutf.Bigstring.Base64.decode_exn encoded in
  assert (Simdutf.Bigstring.is_valid ~encoding:Utf16le decoded)

String operation	Bigstring counterpart
validate, is_valid, count_codepoints	Bigstring.validate, Bigstring.is_valid, Bigstring.count_codepoints
validate_ascii, is_ascii	Bigstring.validate_ascii, Bigstring.is_ascii
Encoding.detect	Bigstring.Encoding.detect
find, find_utf16	Bigstring.find, Bigstring.find_utf16
transcode, transcode_exn	Bigstring.transcode, Bigstring.transcode_exn
Base64.encode, Base64.decode, Base64.decode_exn	Bigstring.Base64.encode, Bigstring.Base64.decode, Bigstring.Base64.decode_exn
Base64.decoded_length, Base64.decode_into	Bigstring.Base64.decoded_length, Bigstring.Base64.decode_into

Bigstring.Base64.decode_to_bytes decodes a bigstring slice directly into OCaml bytes, with the same rules as Base64.decode_into.

Encoding types, Base64 options, and error handling are shared with the string API. Allocating transcoding/Base64 functions return fresh bigstrings whose dimensions are the actual output byte lengths.

Caller-provided output buffers #

For reusable output storage, use Bigstring.transcode_into, Bigstring.Base64.encode_into, or Bigstring.Base64.decode_into:

let transcode_into_buffer input =
  let open Simdutf in
  match Bigstring.transcoded_length ~src:Utf8 ~dst:Utf16le input with
  | Error e -> Result.Error e
  | Ok capacity ->
      let output = Bigstring.create capacity in
      match Bigstring.transcode_into ~src:Utf8 ~dst:Utf16le ~output input with
      | Error e -> Result.Error e
      | Ok written -> Ok (output, written)

Bigstring.transcoded_length validates the input and destination representability before returning the exact output byte length. Bigstring.Base64.encoded_length takes an input byte length and the encoding options. Bigstring.Base64.decoded_length takes a bigstring slice and computes a sufficient output size, exact for valid Base64 including whitespace and padding; it does not validate input.

The _into functions accept ~output, optional ~output_off (default zero), and ~output_len (default remaining capacity). They return Ok written, in bytes, and leave bytes outside the written prefix unchanged on success. An exact-sized buffer suffices. Insufficient capacity returns Error { code = Output_buffer_too_small; offset = 0; ... } without writing. Malformed Unicode or unrepresentable characters also leave output unchanged. Malformed Base64 may modify bytes within the output slice before reporting an error; discard that output.

Overlapping input/output slices raise Invalid_argument, including overlaps through different Bigarray.Array1.sub views. Disjoint slices of the same backing allocation are supported. Bigstring.create allocates uninitialized storage; only read bytes after writing them.

off defaults to zero and len to the remaining bytes. Bounds are checked; negative or out-of-range slices raise Invalid_argument. Error offsets are relative to the slice. No alignment or extra padding is required. UTF-8 and Latin-1 validation/counting and Base64 read the bigstring directly. UTF-16/32 use aligned native temporary storage, and detection may copy unaligned input. Processing never stages data through an OCaml string. The explicit of_string, to_string, and sub_string helpers copy with memcpy; sub_string copies a slice, such as the written prefix of an output buffer.

Runtime behavior #

Simdutf.version reports the vendored version. Simdutf.implementation () reports the selected CPU implementation, which can also be a scalar fallback.

Operations are stateless and can be called from different OCaml domains. Do not mutate input or access output from another domain during an operation.

Operations on OCaml strings and bytes retain the runtime lock, since the garbage collector could otherwise move them. Bigstring operations release the lock when their input slice is at least 64 KiB long, so other threads and domains can run meanwhile; the bigstrings are kept alive for the duration. Shorter bigstring inputs, encoding detection, Bigstring.Base64.decode_to_bytes, and the copying helpers retain it.

Validation, decoded_length, and the _into and decode_to_bytes functions are [@@noalloc] stubs returning a single integer. On success, validate and is_valid allocate nothing, and functions returning (int, error) result allocate only the Ok block. Errors allocate their record. ~off and ~len arguments are unboxed by cross-module inlining in native builds; Dune's dev profile compiles with -opaque, which prevents that. Transcoding allocates an output string or bigstring (unless using _into) and, for wide encodings, aligned C++ buffers. C++ allocation failures are translated to OCaml Out_of_memory; output-size overflow raises Invalid_argument.

These bindings cover the validating whole-buffer APIs described above. Streaming conversion, unchecked conversion, and replacement of malformed Unicode are not currently exposed.

Tests and benchmark #

dune runtest
dune runtest --profile release   # also runs the allocation checks
dune exec --profile release bench/bench_utf.exe -- 1000
dune exec --profile release bench/bench_mime.exe

Tests cover Unicode boundaries and all encoding pairs, byte order and BOM preservation, malformed input and byte offsets, Latin-1 representability, Base64 vectors/options, bigstring bounds/alignment, exact output capacities, buffer overlap and shared views, mmap, independent randomized scalar references, GC pressure, and multiple OCaml domains. test/test_slices.ml checks every sliced operation at slice edges and against bad slices, compares the string, bytes, and bigstring variants on random valid and malformed input, and checks UTF-8 validation against String.is_valid_utf_8 and Base64 decoding against a reference decoder. test/test_find.ml checks byte and UTF-16 search against naive references on random slices, including unaligned bigstring views. test/test_alloc.ml checks per-call allocation in native release builds. The bigstring and slice suites run in native and bytecode modes. README examples are compiled and checked in test/test_readme.ml.

bench_utf reports input MiB/s for validation, counting, transcoding, and Base64, including OCaml allocation overhead. bench_mime measures operations on ranges of larger buffers: bigstring copies, decoding a 12 MiB CRLF-wrapped Base64 body from a string, byte search, and validating short slices. Its optional argument is the number of iterations; results depend on the machine and input.

License #

The bindings are licensed under ISC, with the same author and maintainer metadata as simdjsont. See LICENSE.

Vendored simdutf is licensed under MIT OR Apache-2.0. Upstream notices and both license texts are preserved in lib/ and included in the installed documentation.