diphones/ — the shippable AC-native jeffrey TTS. jeffrey-pvc ElevenLabs as teacher → forced-aligned diphone bank → C/WASM concat synth. ~3-5 MB shippable assets. Sounds like jeffrey because the diphones ARE jeffrey.
PT lane (this README + the rest of pop/voice/) — Pink Trombone as a timbral musical instrument. Confirmed audibly that PT can't reach phoneme-level intelligibility from external trajectory fitting alone — it's a vocal-tract music tool, not a speech synthesizer. The fit pipeline is still useful: anthropometric priors, single-pose vowel matching, and big-pictures-style sustained-vowel timbres.

The PT lane was the first attempt; the pivot below records why it can't ship as jeffrey's voice but stays useful for music. The diphones lane is the new mainline for shippable jeffrey speech.

PT lane (timbral instrument) #

original goal — a tiny, runnable, anatomically grounded jeffrey voice — a physical model of jeffrey's vocal tract that we can synthesize from in C/WASM at AC-native budgets — derived from two existing AC corpora:

the jeffrey-pvc ElevenLabs clone (the audio side — already canonical, already wired to /api/say)
the jeffrey-platter photographic corpus (55 AV-shoot headshots + 38 candid masters + 90 first-person POV captures) — the visual side, already POI-annotated

the synthesis layer is pink trombone (neil thapen, 2017) — a browser-side physical model of the vocal tract, ~500 lines of JS, MIT-ish, ports cleanly to C and WASM. PT is the structural prior. jeffrey-pvc is the acoustic target. the platter photos give us the anatomical prior so the search lives inside a realistic envelope, not a hacky one.

"the gear was the gate, the gate was the price tag" — pop/big-pictures/plork.txt

the gate here is generic-adult-male defaults. the gate is the price tag.

why this lane exists #

the prior frankenstein attempt — slicing jeffrey-pvc per phoneme and stitching — produced incoherent monsters because the slices come from different acoustic contexts. PT is a physical model: optimizing tract parameters lands on something coherent by construction. and once it's a physical model, the parameters mean something — they're tongue position, lip aperture, glottal tension — not opaque embedding deltas.

dropped 2026-05-03: the AC-native formant-resonator path. it sounded like tones, not a voice.

picked up 2026-05-03: the pink trombone path, with anatomical priors derived from the platter, with the question of whether the data can also improve PT itself.

pipeline #

platter photos ──▶ measure-jeffrey.py  (mediapipe / insightface 3D landmarks)
                       │
                       ▼
                  PHYSIOLOGY.md priors
                  (VTL, lip aperture range, mandible ROM, nasal scale)
                       │
                       ▼
jeffrey-pvc /api/say ──▶ record-corpus.mjs ──▶ corpus/raw/*.wav
                       │
                       ▼
                  fit.py  (CMA-ES + MFCC distance, bounded by priors)
                       │
                       ▼
                  fits/<phoneme>.json   (per-phoneme PT param trajectories)
                       │
                       ▼
                  render.{c,wasm,mjs}    (jeffrey voice at synth time)

every step caches to disk. reruns cost $0 (the only paid step is the corpus recording, and that's content-hashed).

file map #

README.md             — this file
CORPUS.md             — phoneme target list + recording protocol
PHYSIOLOGY.md         — anthropometric pipeline (platter → tract priors)
MODEL-EXTENSIONS.md   — research outline: how the physiology data could
                        improve pink trombone itself, not just bound it
NOTICE.md             — third-party license tracking (PT vendor)

bin/
  vendor-pt.sh        — clone neil thapen's pink trombone into vendor/
  record-corpus.mjs   — hits /api/say with jeffrey-pvc per phoneme
  measure-jeffrey.py  — extracts anthropometry from platter photos
  fit.py              — CMA-ES fit of PT params to recorded WAV
  render-pt.mjs       — node-side PT renderer (params → wav, headless)

corpus/
  phonemes.json       — machine-readable target list
  raw/                — recorded jeffrey-pvc WAVs (gitignored)

fits/                 — per-phoneme PT param trajectories (gitignored)
vendor/
  pinktrombone/       — vendored thapen source (gitignored; see NOTICE.md)
jeffrey-anthropometry.json  — derived priors (committed; small JSON)

status #

2026-05-04 — lane scaffolded. all docs landed, all bin/ scripts are stubs.
2026-05-04 — vendored from dood.al/pinktrombone directly (single-file HTML, MIT, Neil Thapen 2017). A github mirror (zakaton/Pink-Trombone) was tried first but turned out GPL-3.0 after refactoring; rejected. License decision resolved — see NOTICE.md.
2026-05-04 — bin/render-pt.mjs working. Loads PT inline source via node vm sandbox (drift-free; re-extracts from upstream HTML on every call). Renders six sustained vowels (ah ee oo aa schwa hum) to WAV in ~90 ms each at 22050 Hz. Smoke set on ~/Desktop is the audible proof.
2026-05-04 — bin/fit.py working. CMA-ES + librosa MFCC distance, calls render-pt.mjs as a subprocess. PT→PT self-test recovers every articulator within tolerance. Real fit on the smoke pt-ee.wav with iy seed: loss 27.8 → 2.5 over 200 evals (~36 s). Recovered params lined up tightly: tongueDiameter Δ -0.02, f0 Δ +0.6 Hz, tenseness Δ +0.006, lipMul ≈ 1.0.
2026-05-04 — bin/measure-jeffrey.py working (mediapipe Tasks API). Ran on 521 confirmed-jeffrey IG photos → 363 measurements (74% detection rate). jeffrey-anthropometry.json committed: 8 facial measurements (medians + IQRs) + Fitch-Giedd VTL prior (17.5 cm ± 0.8 σ; population male median, no stated body height yet). lip_mul_upper_bound = 1.0 consumed by fit.py's bounds clamp.
2026-05-04 — smoke recording: 3 jeffrey-pvc clips (/iy/ /ah/ /uw/, iso context) — server returns raw mp3 without with-timestamps for jeffrey-pvc, recorder falls back gracefully.
2026-05-04 — fit loss v0.1 → v0.2: replaced MFCC time-mean (20-d) with log-mel mean+std (128-d) + librosa.pyin F0 distance in log-cents. v0.1 reported f0=120 Hz on jeffrey's /iy/ but the actual jeffrey-pvc F0 was 144.94 Hz — v0.1 was 25 Hz off and didn't know it. v0.2 recovers F0 to within 1.4 Hz. Throughput 3.7 → 1.8 evals/s (pyin overhead).
2026-05-04 — fit-basin finding: with --sigma0 0.08 --init iy the optimizer recovers the anatomically correct front-tongue /iy/ pose (tongueIndex=12.04, tongueDiameter=2.67). With wider sigma it falls into a back-close basin at slightly lower loss — PT can produce similar vowel-like timbres from multiple tract configurations, so the acoustic loss is multimodal. POSE_INITS + tight sigma is the right disambiguator, not loss-function complexity.
2026-05-04 — trajectory rendering landed. render-pt.mjs accepts --keyframes arrays of {t, ...params} interpolated linearly across the duration; PT's tract.movementSpeed adds another smoothing layer. --word eye shortcut + WORD_RECIPES table for /aɪ eɪ oʊ aʊ wi/. 2-keyframe joint fit: take 4 ElevenLabs requests for "eye"/"owe"/"we"/"ay", then run fit.py twice per word with --window-start/--window-end to recover start + end poses independently. Composed trajectory recovers anatomically correct front/back tongue positions AND the natural rising F0 contour (e.g. "eye" goes 143 → 198 Hz under stress). PT speaking the fitted-from-jeffrey "eye" is pt-word-eye-fitted.wav on Desktop.
2026-05-04 — pivot: Audible test of PT-saying-words confirmed PT cannot reach phoneme-level intelligibility. New mainline for shippable jeffrey speech is diphones/ — ElevenLabs-distilled diphone bank, ~3-5 MB shippable, pure C concat at synth time. Diphone target list generated: 1527 ARPABet pairs, 532 in tier-1. PT remains useful for big-pictures sustained-vowel timbres; it's just not the speech engine.

# render a hand-crafted word
node bin/render-pt.mjs --word eye --duration 1.0 --out ~/Desktop/pt-eye.wav
# fit a word from jeffrey's recording (two-pass, start + end)
.venv/bin/python bin/fit.py --target jeff-eye.mp3 --init ah --window-end 0.45 --out fits/eye-start.json
.venv/bin/python bin/fit.py --target jeff-eye.mp3 --init iy --window-start 0.55 --out fits/eye-end.json
# (compose keyframes + render — see one-liner in fits/ pipeline)

# quick smoke run
node bin/render-pt.mjs --smoke ~/Desktop
# single vowel with arbitrary params
node bin/render-pt.mjs --params '{"pose":"iy","f0":110}' --duration 2 --out ~/Desktop/pt-ee.wav
# fit any target WAV
.venv/bin/python bin/fit.py --target ~/Desktop/pt-ee.wav --init iy --max-evals 200
# end-to-end self-test (no target needed)
.venv/bin/python bin/fit.py --self-test --max-evals 200

memory: reference_pink_trombone.md — origin notes
memory: feedback_jeffrey_pvc_settings.md — stability ≥ 0.5 keeps voice identity
memory: feedback_pop_dual_vocal.md — AC-native vocal lane history (formant attempt dropped)
repo: pop/RESEARCH-DIRECTION.md — vocal lane in the wider pop/ research
repo: papers/jeffrey-platter/ — photographic corpus index (the data this lane reads)
repo: portraits/jeffrey/bin/ — existing face-match / face-describe pipeline (insightface, GPT-4o); shares dependency surface with measure-jeffrey.py

maintained by @jeffrey — update STATUS as the lane advances

voice — the jeffrey synth lanes #

PT lane (timbral instrument) #

why this lane exists #

pipeline #

file map #

status #

related #