russ.fugl.dev / kiri
Multi-runtime Japanese morphological tokenizer in TypeScript, inspired by Sudachi. Performs Viterbi tokenization with NFKC normalization.
Star
Fork 0
Watch

Split replace_map into char-keyed and string-keyed, fix key_lengths master

Split the single HashMap<String, String> into replace_single (HashMap<char, String>) for single-codepoint rules and replace_multi (HashMap<String, String>) for multi-codepoint rules. Single-char lookups now avoid allocating a String key entirely. key_lengths now stores char count instead of byte length, and only tracks multi-codepoint keys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author Russ T Fugal Co-author Claude Opus 4.6 Date Commit 4bb4e5bd Parent 05fb4647
+46 -17
1 changed file