Something went wrong. Try again.
Sliding byte-reader window
Something went wrong. Try again.
4.6 kB · 102 lines
OCaml
at main
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103(*--------------------------------------------------------------------------- Copyright (c) 2026 Thomas Gazagnaire. All rights reserved. SPDX-License-Identifier: ISC ---------------------------------------------------------------------------*)
(** A sliding window over a {!Bytesrw.Bytes.Reader.t}.
A parser for a text format needs a buffer holding the bytes it is looking at, a way to ask for more when it runs out, and an end-of-input flag, and none of it depends on the format. A reader hands out slices that are valid only until the next read, with boundaries wherever they fall, so a parser cannot work on slices directly and needs a window whose extent it controls.
The window is the cursor's own buffer and never aliases a caller's [string]. To parse a string, use a cursor on {!Bytesrw.Bytes.Reader.of_string}.
{1:pos Positions}
A cursor has two kinds of position. {!pos} is an index into the window, and {!discarded} is the absolute offset of the window's first byte in the stream, so a byte's stream offset is [discarded c + i] for a window index [i]. Only stream offsets are stable: {!ensure} may slide the window, after which an index taken before it names a different byte, or none.
{1:lookahead Look-ahead}
{!ensure} is the only way to widen the window and the only thing that can invalidate an index, so a parser's look-ahead bound is explicit: it is the largest [n] the parser passes to {!ensure} while holding an index it uses afterwards. A well-behaved parser holds no index across {!ensure}. *)
type t(** A cursor part-way through a byte stream. *)
val v : ?size:int -> ?max_size:int -> ?on_bytes:(bytes -> int -> int -> unit) -> Bytesrw.Bytes.Reader.t -> t(** [v r] is a cursor on [r] with an empty window.
[size] (defaults to [8192]) is the window's initial and usual size: a refill takes what fits and pushes the rest back on [r], so a reader handing out large slices does not decide how much the cursor holds. [max_size] (defaults to [Sys.max_string_length]) caps what {!ensure} may grow it to.
[on_bytes b first length] is called on each range of bytes as it enters the window, in stream order and exactly once per byte, which is where a streaming validator or checksum belongs. *)
val pos : t -> int(** [pos c] is the window index of the next unread byte.
This and the other accessors are marked [[@inline]] because a parser reads one byte at a time through them: reading a megabyte through [pos], {!get} and {!skip} costs what reading it through [String.get] costs, and a call per byte would not. {!ensure} is the one a caller should keep off that path, by testing {!available} before asking. *)
val length : t -> int(** [length c] is the number of valid bytes in the window; indices in [\[0;length c)] may be read, and reading is only meaningful from [pos c] on.*)
val available : t -> int(** [available c] is [length c - pos c], the bytes readable without a refill. *)
val discarded : t -> int(** [discarded c] is the stream offset of window index [0], so [discarded c + pos c] is the stream offset of the next unread byte. *)
val eof : t -> bool(** [eof c] is [true] once the reader has reported end of input. It says nothing about the window, which may still hold unread bytes. *)
val get : t -> int -> char(** [get c i] is the byte at window index [i]. Requires [i < length c]; a higher index reads whatever the buffer happens to hold. *)
val sub_string : t -> first:int -> length:int -> string(** [sub_string c ~first ~length] copies a range of the window out. Requires [first + length <= length c]. *)
val add_to_buffer : t -> Buffer.t -> first:int -> length:int -> unit(** [add_to_buffer c b ~first ~length] appends a range of the window to [b], which is how a token that straddles a refill is kept while the window slides out from under it. *)
val skip : t -> int -> unit(** [skip c n] advances {!pos} by [n]. Requires [n <= available c]. *)
val ensure : t -> int -> unit(** [ensure c n] makes [n] bytes readable from {!pos} where the stream has them: afterwards [available c >= n], or [eof c] is [true], or [n] is past [max_size]. It may slide the window, so an index taken before it must not be used after it. *)
val ensure_all : t -> unit(** [ensure_all c] grows the window until it holds the whole rest of the stream, which is what a caller wanting the remaining input as one string has to do. It defeats the point of the window and is not for the parse path. *)