Something went wrong. Try again.
An experimental library which implements a subset of the apache arrow format in ocaml.
Something went wrong. Try again.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346(** A typed, columnar Arrow array — the value object that pairs a logical {!Datatype.t} with the {!Arrow.Internal.Buffer}s that physically back its slots.
An array is, conceptually, the record
{v { dtype; length; null_count; buffers; children; dictionary } v}
- [dtype] — the logical column type that dictates the {e physical layout} (how many buffers there are and what each holds). - [length] — the number of slots (logical values) in the array. - [null_count] — how many of those slots are null. Always [0 <= null_count <= length]. - [buffers] — the raw byte regions, in Arrow's canonical order. The {e validity bitmap}, when present, is [buffers]'s first entry; the data / offset buffers follow, in the order the layout dictates. - [children] — sub-arrays for the nested layouts ([List]/[Map] element array, [Struct] field arrays). Empty for the leaf layouts. - [dictionary] — for a [Dictionary] array, the value array its indices point into; [None] otherwise.
{2 The "no validity buffer" optimization}
A validity buffer is present {b iff} the array actually has nulls ([null_count > 0]). An all-valid array carries no validity buffer at all — its accessors simply treat every slot as non-null. The smart constructors apply this automatically: pass an all-valid (or no) {!Validity.t} and the resulting array has [null_count = 0] and no validity buffer.
{2 Physical layout per [dtype]}
Beyond the optional leading validity buffer, the data buffers are:
- fixed-width primitives ([Boolean], [Int32], [Int64], [UInt8]/[16]/[32]/[64], [Float64], [Timestamp]) and [Fixed_size_binary]: one data buffer. [Boolean] packs one bit per slot (LSB-first); the others pack one fixed-width little-endian value per slot. - [Utf8] and [Binary]: an offsets buffer ([length + 1] int32 entries) then a data buffer of the concatenated bytes. - [Dictionary]: one data buffer of indices (width set by the index type); the values live in {!dictionary}. - [List] (and [Map], which is [List<Struct<key, value>>]): one offsets buffer; the element values are {!child} [0]. - [Struct]: no data buffer; each field is a {!child}.
The type is kept abstract: build an array with a per-type smart constructor and read it back through the accessors. The constructors enforce every invariant above, so a value of this type is always well-formed.
{2 Example}
In practice the easiest way to obtain an array is to append values to a {!Arrow.Builder} and {!Arrow.Builder.finish} it; this page's accessors then read it back. Every typed accessor returns [None] for a null slot, and the "no validity buffer" optimization means an all-valid array reports [null_count = 0]:
{[ # open Arrow;; # let col = let b = Builder.utf8 () in Builder.append_utf8 b "alice"; Builder.append_null b; Builder.append_utf8 b "carol"; Builder.finish b;; val col : Arrow.Array.t = <abstr> # Datatype.to_string (Array.dtype col);; - : string = "Utf8" # Array.length col;; - : int = 3 # Array.null_count col;; - : int = 1 # Array.is_null col 1;; - : bool = true # Array.get_utf8 col 0;; - : string option = Some "alice" # Array.get_utf8 col 1;; - : string option = None ]}
The smart constructors are the lower-level alternative, taking the typed building blocks directly. For a fixed-width layout that is a data {!Arrow.Internal.Buffer} of little-endian values — here two [Int32]s, [7] and [8], with no nulls:
{[ # let raw = let data = Arrow.Internal.Buffer.create ~len:8 in Arrow.Internal.Buffer.blit_from_string ~src:"\007\000\000\000\008\000\000\000" ~src_pos:0 data ~dst_pos:0 ~len:8; Array.int32 ~length:2 ~data ();; val raw : Arrow.Array.t = <abstr> # Array.get_int32 raw 0;; - : int32 option = Some 7l # Array.get_int32 raw 1;; - : int32 option = Some 8l ]} *)
type t
(** {1 Construction}
One smart constructor per layout. Each takes the typed building blocks ({!Validity.t}, {!Offsets.t}, data {!Buffer.t}, child arrays) rather than a bag of raw buffers, so the layout can never be assembled wrong. [?validity] is omitted for an all-valid array; when given, it must describe exactly [length] slots, and an all-valid bitmap is dropped (the optimization above).
Every constructor raises [Invalid_argument] on a precondition violation (negative length, a data buffer too small for the slot count, an offsets buffer whose slot count or extent disagrees with the data, a validity bitmap of the wrong length, a dictionary index out of range). These are programmer errors, not expected failures. *)
val boolean : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [boolean ~length ~data ()] is a [Boolean] array; [data] holds the bit-packed values (LSB-first, [ceil (length / 8)] bytes). *)
val int32 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [int32 ~length ~data ()] is an [Int32] array; [data] holds [length] signed little-endian 32-bit values. *)
val int64 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [int64 ~length ~data ()] is an [Int64] array; [data] holds [length] signed little-endian 64-bit values. *)
val uint8 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [uint8 ~length ~data ()] is a [UInt8] array; one byte per slot. *)
val uint16 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [uint16 ~length ~data ()] is a [UInt16] array; two little-endian bytes per slot. *)
val uint32 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [uint32 ~length ~data ()] is a [UInt32] array; four little-endian bytes per slot. *)
val uint64 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [uint64 ~length ~data ()] is a [UInt64] array; eight little-endian bytes per slot. *)
val float64 : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [float64 ~length ~data ()] is a [Float64] array; [length] IEEE-754 little-endian doubles. *)
val timestamp : ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [timestamp ~length ~data ()] is a [Timestamp] array; stored exactly like {!int64} (nanoseconds since the epoch). *)
val timestamp_tz : unit:Datatype.time_unit -> ?tz:string -> ?validity:Validity.t -> length:int -> data:Buffer.t -> unit -> t(** [timestamp_tz ~unit ?tz ~length ~data ()] is a [Timestamp_tz] array of [unit] (optionally in [tz]); stored exactly like {!int64}. *)
val utf8 : ?validity:Validity.t -> length:int -> offsets:Offsets.t -> data:Buffer.t -> unit -> t(** [utf8 ~length ~offsets ~data ()] is a [Utf8] array. [offsets] must describe exactly [length] slots ({!Offsets.length} [= length]); slot [i] spans [data] bytes [\[Offsets.get offsets i, Offsets.get offsets (i + 1))]. @raise Invalid_argument if [Offsets.length offsets <> length] or [data] is smaller than {!Offsets.last_offset}. *)
val binary : ?validity:Validity.t -> length:int -> offsets:Offsets.t -> data:Buffer.t -> unit -> t(** [binary ~length ~offsets ~data ()] is a [Binary] array: the {!utf8} layout over arbitrary bytes (no encoding is implied or checked). @raise Invalid_argument as {!utf8}. *)
val fixed_size_binary : ?validity:Validity.t -> length:int -> byte_width:int -> data:Buffer.t -> unit -> t(** [fixed_size_binary ~length ~byte_width ~data ()] is a [Fixed_size_binary byte_width] array; each slot is [byte_width] consecutive bytes of [data]. @raise Invalid_argument if [byte_width < 0] or [data] is too small. *)
val list : ?validity:Validity.t -> length:int -> offsets:Offsets.t -> values:t -> unit -> t(** [list ~length ~offsets ~values ()] is a [List] array over element array [values]. Slot [i] is the sub-range [\[Offsets.get offsets i, Offsets.get offsets (i + 1))] of [values]. A [Map] array is just this with [values] a {!struct_} of two fields, so [Datatype.to_string (dtype result)] is the [Map] rendering — no separate [map] constructor is needed. @raise Invalid_argument if [Offsets.length offsets <> length] or {!Offsets.last_offset} exceeds [length values]. *)
val struct_ : ?validity:Validity.t -> length:int -> children:t list -> unit -> t(** [struct_ ~length ~children ()] is a [Struct] array. Every child must have the same [length] as the struct. @raise Invalid_argument if any child's length differs from [length]. *)
val dictionary : ?validity:Validity.t -> length:int -> index:Datatype.t -> indices:Buffer.t -> values:t -> unit -> t(** [dictionary ~length ~index ~indices ~values ()] is a [Dictionary] array: [indices] holds [length] unsigned integers (width set by [index], which must be one of [UInt8]/[UInt16]/[UInt32]/[Int32]/[Int64]) that point into the [values] array. Every non-null index must lie in [\[0, length values)]. @raise Invalid_argument if [index] is not an integer type, [indices] is too small, or any non-null index is out of range. *)
(** {1 Layout accessors} *)
val dtype : t -> Datatype.t(** The array's logical type. *)
val length : t -> int(** The number of slots in the array. *)
val null_count : t -> int(** The number of null slots; always in [[0, length t]]. *)
val buffers : t -> Buffer.t list(** The raw backing buffers in Arrow's canonical order: the validity bitmap first when [null_count t > 0], then the layout's data/offset buffers. Ready to hand to an IPC writer. *)
val children : t -> t list(** The child sub-arrays: the element array for [List]/[Map], the field arrays for [Struct]; empty for the leaf and [Dictionary] layouts. *)
val child : t -> int -> t(** [child t i] is the [i]-th child sub-array. @raise Invalid_argument if [i] is out of range. *)
val dictionary_values : t -> t option(** The value array a [Dictionary] array's indices point into; [None] for every other layout. *)
(** {1 Validity} *)
val is_valid : t -> int -> bool(** [is_valid t i] is [true] iff slot [i] is non-null. @raise Invalid_argument if [i] is outside [\[0, length t)]. *)
val is_null : t -> int -> bool(** [is_null t i] is [not (is_valid t i)]. *)
(** {1 Typed accessors}
Each returns [None] for a null slot and [Some v] for a value. Calling an accessor whose type does not match the array's [dtype] is a programmer error and raises [Invalid_argument]; an out-of-range index likewise raises. *)
val get_bool : t -> int -> bool optionval get_int32 : t -> int -> int32 optionval get_int64 : t -> int -> int64 optionval get_uint8 : t -> int -> int optionval get_uint16 : t -> int -> int optionval get_uint32 : t -> int -> int optionval get_float64 : t -> int -> float optionval get_timestamp : t -> int -> int64 option
val get_timestamp_tz : t -> int -> int64 option(** Like {!get_timestamp}, for a [Timestamp_tz] array of any unit/timezone. *)
val get_utf8 : t -> int -> string option
val get_uint64 : t -> int -> int64 option(** [get_uint64 t i] is slot [i]'s unsigned 64-bit value as the [int64] with the same bit pattern (values [>= 2^63] come back negative). *)
val get_binary : t -> int -> string option(** [get_binary t i] is slot [i]'s raw bytes ([None] if null). *)
val get_fixed_size_binary : t -> int -> string option(** [get_fixed_size_binary t i] is the [byte_width] raw bytes of slot [i] as a string ([None] if null). *)
val get_list : t -> int -> (int * int) option(** [get_list t i] is the [\[start, stop)] half-open range of slot [i] into [child t 0] (the element array), or [None] if the slot is null. Works for both [List] and [Map]. *)
val get_dictionary_index : t -> int -> int option(** [get_dictionary_index t i] is the raw dictionary index stored at slot [i] ([None] if null); resolve it with [dictionary_values t] and the value array's own accessor. *)
(** {1 Logical equality} *)
val equal_values : t -> t -> bool(** [equal_values a b] is [true] iff [a] and [b] have the same {!dtype} and {!length}, and every slot is null in both or holds the same value in both (for [Float64], compared with [Float.equal]; for a [Dictionary], the resolved values, not the indices; for a [List], the elements).
Unlike byte-level comparison ({!Record_batch.equal}), it ignores bytes that carry no meaning: validity padding bits, the values stored under null slots, and spare capacity at the end of buffers. Use it to compare arrays built over foreign memory with arrays built by this library.
{[ # let a = let b = Builder.int64 () in Builder.append_int64 b 1L; Builder.append_null b; Builder.finish b;; val a : Arrow.Array.t = <abstr> # let b = (* Same slots, but a garbage value under the null and a larger data buffer. *) let data = Internal.Buffer.create ~len:32 in Internal.Buffer.set data 0 1; Internal.Buffer.set data 8 0xff; let validity = Internal.Validity.init ~len:2 (fun i -> i = 0) in Array.int64 ~validity ~length:2 ~data ();; val b : Arrow.Array.t = <abstr> # Array.equal_values a b;; - : bool = true ]} *)