bytecursor #

A sliding window over a Bytesrw byte reader, for text parsers.

A text parser needs a buffer holding the bytes it is looking at, a way to ask for more, and an end-of-input flag, and none of it depends on the format. A Bytesrw reader hands out slices that are valid only until the next read, with boundaries wherever they happen to fall, so a parser cannot work on the slices directly. Bytecursor gives it a window whose extent the parser controls.

The main functions are:

• Bytecursor.v, which creates a cursor on a reader, with the window size, a cap on how far look-ahead may grow it, and an on_bytes observer for a streaming validator or checksum;
• Bytecursor.ensure, the only way to widen the window, and so the one place where a parser's look-ahead bound is written down;
• pos, length, available, discarded and eof, which say where the parse is;
• get, sub_string, add_to_buffer and skip, which read and advance.

A cursor has two kinds of position. pos is an index into the window, and discarded is the offset in the stream of the window's first byte. Only stream offsets are stable, because ensure may slide the window.

Installation #

Install with opam:

$ opam install nox-bytecursor

If opam cannot find the package, it may not yet be released in the public opam-repository. Add the overlay repository, then install it:

$ opam repo add samoht https://tangled.org/gazagnaire.org/opam-overlay.git
$ opam update
$ opam install nox-bytecursor

Requires OCaml >= 4.14 and bytesrw.

Usage #

Read a token that spans several refills. The window here is smaller than the input, so it slides while the token accumulates, and add_to_buffer copies each byte out before it goes:

let token src =
  let c = Bytecursor.v ~size:8 (Bytesrw.Bytes.Reader.of_string src) in
  let b = Buffer.create 16 in
  let rec go () =
    Bytecursor.ensure c 1;
    if Bytecursor.available c = 0 then Buffer.contents b
    else if Bytecursor.get c (Bytecursor.pos c) = ' ' then Buffer.contents b
    else begin
      Bytecursor.add_to_buffer c b ~first:(Bytecursor.pos c) ~length:1;
      Bytecursor.skip c 1;
      go ()
    end
  in
  go ()

let () = assert (token "description = \"Demo\"" = "description")

Report a position. A window index means nothing once the window has slid, so an error records the stream offset discarded c + pos c:

let offset_of src ch =
  let c = Bytecursor.v ~size:4 (Bytesrw.Bytes.Reader.of_string src) in
  let rec go () =
    Bytecursor.ensure c 1;
    if Bytecursor.available c = 0 then None
    else if Bytecursor.get c (Bytecursor.pos c) = ch then
      Some (Bytecursor.discarded c + Bytecursor.pos c)
    else begin
      Bytecursor.skip c 1;
      go ()
    end
  in
  go ()

let () = assert (offset_of "description = \"Demo\"" '"' = Some 14)

API #

lib/bytecursor.mli documents the full interface.

Licence #

ISC. See LICENSE.md.