Profiling Kiri Elixir Tokenizer #

bench/profile.exs runs OTP profilers (cprof, eprof, fprof) against realistic tokenization workloads using the same corpus files as bench/tokenize.exs.

Prerequisites #

1. A converted .kiri dictionary:

   cd packages/kiri
   mix kiri.convert ~/.kiri-ji/dict/system_core.dic ~/.kiri-ji/system_core.kiri
   

2. A corpus JSON file (array of strings):

   bun bench/gen-corpus.ts
   

Usage #

cd packages/kiri
mix run ../../bench/profile.exs --tool TOOL [--dict DICT] [--corpus CORPUS] [--limit N]

Flag	Values	Default	Description
--tool	cprof, eprof, fprof	(required)	Which OTP profiler to run
--dict	small, core, full	core	Dictionary size
--corpus	short, long	short	Corpus scale
--limit	integer	all texts	Only profile first N texts

Tools #

cprof — Call frequency #

Counts how many times each traced function is called. Low overhead (~10%), safe on the full corpus. Best as a first pass to find call-count anomalies (e.g. O(n^2) where you expect O(n)).

Output: sorted call counts to stdout, grouped by module.

mix run ../../bench/profile.exs --tool cprof
mix run ../../bench/profile.exs --tool cprof --dict full --corpus long

eprof — Time distribution #

Measures wall-clock time per function across all calls. Shows what percentage of total time each function consumes. Higher overhead than cprof — use --limit for large corpora.

Output: table to stdout + bench/profile_eprof.txt

mix run ../../bench/profile.exs --tool eprof --limit 20

fprof — Caller/callee timing #

Full call-graph profiling with own-time vs accumulated-time breakdown. Shows which callers drive the most time into each function. Highest overhead — always use --limit.

Output: first 80 lines to stdout + bench/profile_fprof.txt (full Erlang terms)

mix run ../../bench/profile.exs --tool fprof --limit 20

Reading the output #

cprof #

Look for modules with unexpectedly high call counts. The output is grouped by module and sorted by total calls descending, with per-function breakdowns:

--- Kiri.Lattice (85326877 calls) ---
  -connect_node/4-fun-0-/5: 55196711
  connect_node/4: 7056792

If a function is called O(n^2) times when you expect O(n), that's a structural issue worth investigating.

eprof #

The table is sorted by time. Key columns:

Column	Meaning
CALLS	Total invocations
%	Percentage of total profiled time
TIME	Total microseconds spent in this function
[uS / CALLS]	Average microseconds per call

FUNCTION                              CALLS        %     TIME  [uS / CALLS]
'Elixir.Dictionary.Grammar':get_connect_cost/4  1221249  1.96   111550  [      0.09]

• High % + low uS/CALLS = called too often (reduce call count)
• High uS/CALLS = expensive implementation (optimize the function body)

fprof #

Each entry in the output file is an Erlang term with three parts:

{[{CallerMFA, CallerCNT, CallerACC, CallerOWN}, ...],   %% callers
 { {Module, Function, Arity}, CNT, ACC, OWN},            %% this function
 [{CalleeMFA, CalleeCNT, CalleeACC, CalleeOWN}, ...]}.   %% callees

Field	Meaning
CNT	Number of calls
ACC	Accumulated time (this function + all callees)
OWN	Time spent in the function itself (excluding callees)

• High ACC but low OWN = the function is a pass-through; optimize its callees
• High OWN = the function itself is the bottleneck
• The callers list shows who is driving calls into this function

Recommended workflow #

1. cprof on full corpus — find call-count anomalies

   mix run ../../bench/profile.exs --tool cprof
   

2. eprof with --limit — see where wall-clock time goes

   mix run ../../bench/profile.exs --tool eprof --limit 20
   

3. fprof with --limit — deep-dive on the top bottleneck's caller/callee chains

   mix run ../../bench/profile.exs --tool fprof --limit 20
   

4. Optimize, then benchmark — measure improvement with the benchmark script

   mix run ../../bench/tokenize.exs --dict core --corpus short
   

Traced modules #

cprof traces these Kiri modules (eprof and fprof trace all functions in the profiled process):

• Kiri, Kiri.Backend.Elixir
• Kiri.Lattice, Kiri.Lattice.Node
• Kiri.Dictionary.Trie, Kiri.Dictionary.Lexicon, Kiri.Dictionary.Grammar
• Kiri.Dictionary.WordParams, Kiri.Dictionary.WordIdTable, Kiri.Dictionary.WordInfo
• Kiri.InputText, Kiri.InputText.Builder, Kiri.InputText.CharCategory
• Kiri.InputText.Normalize, Kiri.InputText.ProlongedSoundMark, Kiri.InputText.IgnoreYomigana
• Kiri.Oov.Mecab, Kiri.Oov.Simple
• Kiri.PathRewrite, Kiri.PathRewrite.NumericParser
• Kiri.MecabConfig