# text

> written against zig 0.16.

zig has no string type. text is `[]const u8`, and the language makes no promise
about what encoding those bytes are in. indexing and slicing are byte
operations, which is correct for a byte slice and wrong for most of the things
people write with it.

## the two vocabularies

almost every text operation is described in one of two units, and they are easy
to mix up because ASCII makes them identical:

- **bytes** — buffer sizes, length caps, wire formats. slicing is right here.
- **characters** — "first character", "last three characters", "one character
  per class". slicing is wrong here the moment input leaves ASCII.

`"’s"` is four bytes: `E2 80 99 73`. taking `token[0]` yields `0xE2`, which is
not a character. it is the lead byte of one, and alone it identifies nothing —
the same first byte begins thousands of different characters.

## what this looks like in practice

spacez extracts four features per token for spaCy's NER model, three of which
are defined over characters — the first character, the last three characters,
and a per-character shape string. all three were implemented with byte slices:

```zig
prefix_buf[0] = token[0];            // first byte
var suffix_buf: [3]u8 = undefined;   // last three bytes
for (token) |c| { ... }              // shape, per byte
```

for ASCII this is exactly right and every test passed. for a curly apostrophe
the prefix hashed a lone `0xE2`, the suffix came from the middle of a character,
and the shape emitted three symbols where spaCy emitted one. the model got
embeddings for tokens that do not exist, and answered confidently.

nothing crashed. bytes are always valid bytes.

## decoding

`std.unicode` has what you need to work in characters:

```zig
const n = try std.unicode.utf8ByteSequenceLength(bytes[0]);  // 1..4
const cp = try std.unicode.utf8Decode(bytes[0..n]);
var it = (try std.unicode.Utf8View.init(s)).iterator();       // codepoint walk
```

going backwards has no helper. continuation bytes are `10xxxxxx`, so walking
back to a character boundary is a loop over `b & 0xC0 == 0x80`.

## classification stops at ASCII

`std.ascii` answers `isAlphabetic`, `isUpper`, `toLower` for bytes below 128.
above that, std has no case mapping and no character categories — those are
large tables and zig does not ship them.

this matters when a spec is written in terms of "letter" or "uppercase".
spaCy's shape feature calls `Проверка` five letters and lowercases `Ω` to `ω`;
reproducing that needs unicode tables from somewhere. the honest options are to
carry the tables, take a dependency, or document the gap. picking one of ASCII's
answers and hoping is the option that looks like it works.

## truncation

a byte cap that lands mid-character produces invalid UTF-8, which json encoders
and terminals both object to. walk characters and stop before the one that would
cross the limit:

```zig
/// truncate to at most `max` bytes without splitting a codepoint
pub fn utf8TruncateLen(s: []const u8, max: usize) usize {
    const limit = @min(s.len, max);
    var i: usize = 0;
    while (i < limit) {
        const seq_len = std.unicode.utf8ByteSequenceLength(s[i]) catch {
            i += 1; // invalid lead byte
            continue;
        };
        if (i + seq_len > limit) break;
        i += seq_len;
    }
    return i;
}
```

any fixed-size field holding user text wants this — entity names, log lines,
anything copied into a `[64]u8`.

## rules of thumb

- if a spec says "character", decode. if it says "byte", slice.
- a fixed-size buffer is a byte count; the loop that fills it is a character
  loop. keeping those straight in the same function is most of the work.
- test with a curly apostrophe, an emoji, and a Cyrillic word. ASCII tests pass
  either way, which is what makes this class of bug survive a test suite.

## related

- [structs](./structs.md) — slices are views; the same discipline applies to
  what a slice points at
- [json](./json.md) — invalid UTF-8 surfaces first at the encoder

## sources

- [spacez](https://tangled.org/zzstoatzz.io/spacez) `src/embed.zig` — prefix,
  suffix and shape computed over bytes; fixed in `be0f81f` (2026-08-18)
- [coral](https://tangled.org/zzstoatzz.io/coral) `backend/src/utf8.zig` — the
  truncation helper