russ.fugl.dev / kiri
Multi-runtime Japanese morphological tokenizer in TypeScript, inspired by Sudachi. Performs Viterbi tokenization with NFKC normalization.
Star
Fork 0
Watch

Commits on main

Format kiri-native with prettier main
Russ T Fugal committed
Precompute NFKC+lowercase table, bump kiri-core to 0.2.0
Russ T Fugal committed
Bump kiri-yaiba to 0.2.0, update kiri-kotoba dep, add CHANGELOG
Russ T Fugal committed
Bump kiri-kotoba to 0.2.0, add CHANGELOG
Russ T Fugal committed
Add tests for NFKC table, replace_map split, and key_lengths fix
Russ T Fugal committed
Split replace_map into char-keyed and string-keyed, fix key_lengths
Russ T Fugal committed
Precompute NFKC+lowercase table at load time, eliminate per-char normalization
Russ T Fugal committed
Deduplicate resolved_form(), reuse path_rewrite version
Russ T Fugal committed
Remove dead params from concatenate_nodes()
Russ T Fugal committed
Bump version to 0.3.0, update CHANGELOG and audit log
Russ T Fugal committed
Unify word_info return type to %WordInfo{} struct across all code paths
Russ T Fugal committed
Use integer-keyed lookup for single-codepoint rewrite rules
Russ T Fugal committed
Precompute NFKC+lowercase table at load time, eliminate all unicode_util from hot path
Russ T Fugal committed
Add range guards to skip unicode_util for CJK/kana codepoints
Russ T Fugal committed
Remove dead get_text calls from normalize loop (saves O(M×N))
Russ T Fugal committed
Remove Kiri.InputText.BuilderTest (12 tests for deleted module)
Russ T Fugal committed
Remove deprecated Kiri.InputText.Builder module
Russ T Fugal committed
Remove legacy Kiri.Tokenizer GenServer (replaced in v0.2.0)
Russ T Fugal committed
Precompute charlist once in build_morphemes instead of per-morpheme
Russ T Fugal committed
Delegate Kiri.Backend.Nif morpheme building to shared Kiri.Backend
Russ T Fugal committed
Extract shared build_morphemes pipeline into Kiri.Backend
Russ T Fugal committed
Minor cleanups: remove dead code, move require Logger to module level
Russ T Fugal committed
Fix O(n^2) list access in Oov.Mecab by using tuple with elem/2
Russ T Fugal committed
Fix stale Kiri.create_tokenizer/2 doc references in tokenizer.ex
Russ T Fugal committed
Add benchmark results (HTML, PNG, markdown)
Russ T Fugal committed
Update READMEs for v0.2.0
Russ T Fugal committed
Update benchmark suite
Russ T Fugal committed
Update kiri Elixir to 0.2.0
Russ T Fugal committed
Update kiri_nif to 0.2.0
Russ T Fugal committed
Update kiri-native to 0.2.0
Russ T Fugal committed
Add kiri-engine 0.2.0, kiri-kotoba 0.1.0, kiri-yaiba 0.1.0
Russ T Fugal committed
Add .gitignore patterns for Erlang/debug artifacts
Russ T Fugal committed
Eliminate unicode_util bottleneck and fix all Credo issues
Russ T Fugal committed
Rewrite Elixir package as pure Elixir tokenizer (v0.2.0)
Russ T Fugal committed
Add shared dictionary, auto MeCab config, and path_rewrite performance for Elixir
Russ T Fugal committed
Add shared dictionary API, performance improvements, and standardize dict paths
Russ T Fugal committed
Add shared dictionary support to share the ~150 MB trie across tokenizers via Arc
Russ T Fugal committed
Update benchmark results and visualization
Russ T Fugal committed
Add benchmark results, visualization, and Playwright screenshot
Russ T Fugal committed
Publish Rust crates to crates.io and kiri-native to npm
Russ T Fugal committed
Add HexDocs documentation link and update package metadata
Russ T Fugal committed
Rename packages for registry availability
Russ T Fugal committed
Remove explicit return type annotations from all implementations
Russ T Fugal committed
Remove @returns tags from all JSDoc
Russ T Fugal committed
Add JSDoc to undocumented items across codebase
Russ T Fugal committed
Small focused fixes: validation, dead code, naming, docs, loop style
Russ T Fugal committed
Extract shared getResourcePath utility
Russ T Fugal committed
Extract tryFinalizeNumeric from joinNumericMorphemes
Russ T Fugal committed
Unify concatenateOovNodes and concatenateNodes
Russ T Fugal committed
Consume resolvedForm across codebase
Russ T Fugal committed
Consume createOovWordInfo in all OOV construction sites
Russ T Fugal committed
Add createOovWordInfo helper and resolvedForm utility
Russ T Fugal committed
Harden and simplify kiri-core: fix 13 code review issues
Russ T Fugal committed
Add TypeScript surrogate pair tests for ignore-yomigana plugin
Russ T Fugal committed
Add Elixir test coverage for Types, Builder.replace, Normalize, SN.format
Russ T Fugal committed
Elixir minor fixes: persistent_term caches, Bitwise import, SN.format
Russ T Fugal committed
Fix O(n^2) performance in Elixir builder and ignore_yomigana
Russ T Fugal committed
TypeScript performance improvements and surrogate pair fix
Russ T Fugal committed
Fix i32-to-i16 silent truncation and oov_pos_id API inconsistency
Russ T Fugal committed
Change LexiconSet to Arc<Vec<u8>> and NIF Mutex to RwLock
Russ T Fugal committed