From 1ae39cd1f3eb58cba468eecc470d9842ea02c690 Mon Sep 17 00:00:00 2001 From: "prompt.ac/@jeffrey" Date: Sun, 19 Jul 2026 13:51:50 -0700 Subject: [PATCH 1/2] =?UTF-8?q?video:=20net-time=20unison,=20deck=20keys,?= =?UTF-8?q?=20sineline,=20steady=20dial,=20gesture=20vector=20=E2=80=94=20?= =?UTF-8?q?the=20tape=20joins=20the=20band?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - πŸ•°οΈ Net-time phase lock: loop boundaries anchor to UTC multiples of the tape length (bios) and the at-rest drive continuously lerps phase toward the AC network clock (clock.time() / /api/clock) with a gentle Β±5% tempo lean β€” every player of the same tape converges into global unison. Verified live: two simultaneous Chrome windows (sineline + house) locked to sync +20ms / converging. Top-right 'sync Β±Nms' readout, green when locked. - Absolute audio rate (tape:audio-rate β†’ sampleSpeed) replaces relative shifts: no estimate drift after heavy interaction. tape:audio-pos relocates the free-running loop for jumps/chops. - Seamless everywhere: every landing (park, wheel, dip, brake, friction, scroll) keeps the scrub drive running β€” driving at exactly 1.0Γ— IS playback, so the 0.9β†’1.0 quantized jump is gone. Video duration now includes the last frame (loop parity with the 8.000s audio), and the bios RAF clock no longer advances frames at 1Γ— between scrub seeks. - πŸŽ›οΈ Pioneer-style deck keys: ← β†’ one-beat jumps, hold ↑ quarter-beat chop repeat, hold ↓ eighth-beat glitch chop; on-screen keys legend. - πŸ“Ÿ Steady-rate dial: drag the top-right readout vertically to set a held rate friction won't touch. Two-finger scroll range up to Β±24Γ—. - 🧭 Gesture vector debug: anchorβ†’finger line with the rate-driving component and live Γ—-readout drawn at the fingertip. - 〰️ New 'sine' synthtape (legato pentatonic sineline) + ^tapes bag entry; the raising position-sweep test tone is removed from all styles. - Autopilot: 14 segments (adds fast 4Hz scratch); docs/video-scrub-rubric.md records the full test rubric and invariants. --- docs/video-scrub-rubric.md | 89 +++++ system/public/aesthetic.computer/bags.json | 5 + system/public/aesthetic.computer/bios.mjs | 99 +++-- .../public/aesthetic.computer/disks/video.mjs | 350 ++++++++++++++---- 4 files changed, 429 insertions(+), 114 deletions(-) create mode 100644 docs/video-scrub-rubric.md diff --git a/docs/video-scrub-rubric.md b/docs/video-scrub-rubric.md new file mode 100644 index 000000000..65d140e7d --- /dev/null +++ b/docs/video-scrub-rubric.md @@ -0,0 +1,89 @@ +# Video Scrub Rubric β€” synthtapes + the scrub instrument + +The tape player (`disks/video.mjs` + tape machinery in `bios.mjs`) is an +instrument: rate is the controlled quantity, position follows. This rubric +defines what "working" means and how it is checked. + +## Test surfaces + +- **`video scrub auto`** β€” the in-piece autopilot: scripted gesture segments + driven through the *same state the finger drives*, graded against + wall-clock ground truth. All output is console-greppable by `πŸ§ͺ`. + Variants compose: `video scrub break auto`, `video scrub 2 auto` (short + tape β†’ seam-crossing stress), etc. +- **Synthtapes** (`tape:play-synth` in bios) β€” deterministic tapes whose + frames render the soundtrack's own waveform on a scrolling ruler, so + scrub velocity, direction, and AV alignment are visible by eye. + Styles: `bed` (melodic), `break` (16th breakbeat), `house`, `dub`, + `sine` (legato sineline). All pitched voices are quantized to whole + cycles per loop β†’ the seam is phase-continuous by construction. +- **Headless driving** β€” wheel gestures via `dispatchEvent(new WheelEvent(...))`, + taps via CDP clicks; `window.__speaker_telemetry.runningCount` asserts the + audio source stayed alive. + +## The rubric (autopilot segments Γ— criteria) + +| Segment | Drives | Pass criteria | +|---|---|---| +| roll | nothing (normal play) | position advances | +| fast forward | rate 3Γ— held 90 ticks | effective rate within Β±25% of 3Γ— (motion-integral, wrap-proof) | +| release into inertia | release at speed | inertia decays and converges | +| reverse | rate βˆ’2Γ— held | effective βˆ’2Γ— Β±25%, seam wrap keeps direction | +| slow crawl | rate 0.5Γ— | effective 0.5Γ— Β±25% (slow-rate accuracy) | +| tap dip | single-tap dip | rate sags below 0.3Γ—, returns ≀1.05Γ—, negligible net motion | +| scratch | Β±3.2Γ— @ 2Hz rocking | both directions exceeded, net drift < 0.7 tape-seconds | +| fast scratch | Β±2.2Γ— @ 4Hz | same, amplitude-scaled | +| touch brake | hold from 1Γ— | rate sags to < 0.05Γ— | +| brake release spin-up | lift after brake | ramps back to pre-brake rate | + +Every scrub-driven segment also records: physics tick count, integrated +wall time (must β‰ˆ segment wall time), speed min/max, and the tape duration +used β€” so a failure names its layer. + +## Invariants (the things that regress) + +1. **Wall-clock honesty** β€” commanded rate means tape-seconds per + wall-second at any sim tick rate (measured `simDt`, burst-tolerant). +2. **One audio source** β€” tape audio is a single `loop: true` sample; the + worklet wraps the read in both directions (stample ring). Nothing + kills/restarts it during interaction; `runningCount` stays 1 through + seam crossings both ways. +3. **Absolute rate** β€” `tape:audio-rate` sets the worklet speed absolutely; + no relative-shift accumulation, no estimate drift. +4. **No handoffs** β€” every landing (park, wheel, dip, brake release, + friction) keeps the scrub drive running; driving at exactly 1.0Γ— *is* + playback, so there is never a restart jump near 1Γ—. +5. **Loop parity** β€” video duration = audio duration exactly + (`mediaRecorderDuration`, includes the last frame's display time), so + video and audio wrap at the same point at any rate. +6. **Scrub owns frames** β€” while scrubbing, the bios RAF clock must not + advance frames at 1Γ— between seeks. +7. **Musical loop grid** β€” at rest, loop periods converge to β‰ˆΒ±2ms of the + tape length (grid-anchored, logged as `πŸ•°οΈ`). + +## Gesture map under test + +tap = dip Β· hold = brake Β· grab+drag = displacement-is-rate scratch +(with an on-screen anchorβ†’finger vector showing direction and energy) Β· +flick = prize-wheel spin-down to pre-flick rate Β· two-finger scroll = +shuttle (Β±24Γ—) Β· top-right readout drag = steady-rate dial (friction-free +hold) Β· ←/β†’ = one-beat jump (audio relocated via `tape:audio-pos`) Β· +hold ↑ = quarter-beat chop repeat, hold ↓ = eighth-beat glitch chop Β· +release = seamless park + bearing friction home to 1Γ— Β· +spacebar = full reset/pause. A keys legend renders bottom-right. + +## Net-time unison + +Loop boundaries anchor to UTC multiples of the tape length (bios), and the +at-rest drive continuously lerps phase toward the AC network clock +(`clock.time()`, synced via `/api/clock`) with a Β±5% tempo lean β€” so every +player of the same tape converges into global unison without jumps. The +top-right `sync Β±Nms` readout is green when locked (<60ms). + +## Open items + +- Independent pitch-shift while playing (granular in the speaker worklet β€” + `targetDuration`/grain machinery exists in `lib/sound/synth.mjs`, needs a + live pitch param + a gesture axis). +- Seam-scratch segment (scratching across the loop point) in the autopilot. +- UTC/`net.time` phase alignment of the loop grid across devices. diff --git a/system/public/aesthetic.computer/bags.json b/system/public/aesthetic.computer/bags.json index 3baba9ee0..ceaf15772 100644 --- a/system/public/aesthetic.computer/bags.json +++ b/system/public/aesthetic.computer/bags.json @@ -26,6 +26,11 @@ "code": "video~scrub~dub", "name": "dub" }, + { + "type": "piece", + "code": "video~scrub~sine", + "name": "sineline" + }, { "type": "piece", "code": "video~scrub~break~2", diff --git a/system/public/aesthetic.computer/bios.mjs b/system/public/aesthetic.computer/bios.mjs index 0dee16701..90881d889 100644 --- a/system/public/aesthetic.computer/bios.mjs +++ b/system/public/aesthetic.computer/bios.mjs @@ -11546,12 +11546,12 @@ async function boot(parsed, bpm = 60, resolution, debug) { const bassRoots = [82.5, 65.5, 98, 73.5]; // Plucks quantized to 4Hz β€” whole cycles per quarter-beat note. const pent = [328, 392, 440, 492, 588]; - let sweepPhase = 0; // Phase-increment, not sin(TAU*f*t) β€” no drift on long tapes. let bassPhase = 0; let melPhase = 0; let ph1 = 0; // Generic per-style phase accumulators let ph2 = 0; let ph3 = 0; + let sineF = 164; // Sineline's gliding frequency (~E3 start) // πŸ₯ "break" style: one bar of 16th-note breakbeat, repeated β€” // scratch material with hard transients on a funk grid. const STEP = BEAT / 4; // 16ths at 120 BPM @@ -11600,6 +11600,20 @@ async function boot(parsed, bpm = 60, resolution, debug) { (Math.sin(ph1) + Math.sin(ph1 * 2) * 0.4) * Math.exp(-inHalf * 12) * 0.42; // Bass stab } + } else if (style === "sine") { + // 〰️ Sineline: one continuous legato sine gliding through a + // pentatonic line, with a soft sub octave β€” pure pitch + // material, lovely under a scrub. The glide keeps frequency + // continuous; a 15ms edge fade softens the loop seam. + const noteIdx = Math.floor(tt / BEAT); + const target = + pent[(noteIdx * 2 + Math.floor(noteIdx / 3)) % pent.length] / 2; + sineF += (target - sineF) * 0.0004; // ~50ms portamento + ph1 += (TAU * sineF) / sampleRate; + ph2 += (TAU * sineF) / 2 / sampleRate; + const sineEdge = Math.min(1, tt / 0.015, (duration - tt) / 0.015); + v += Math.sin(ph1) * 0.48 * sineEdge; + v += Math.sin(ph2) * 0.16 * sineEdge; } else if (style === "dub") { // 🌫️ Halftime dub: deep kicks on 1 & 3, rim on 3, a sub line // per half-bar, and offbeat skank chords. @@ -11650,12 +11664,8 @@ async function boot(parsed, bpm = 60, resolution, debug) { v += Math.sin(melPhase) * Math.exp(-inNote * 9) * 0.22; } - // Position sweep, quiet underneath, with a 50ms edge fade so - // the 880β†’220Hz jump at the loop seam can't click. - sweepPhase += (TAU * (220 + pos * 660)) / sampleRate; - const edge = Math.min(1, tt / 0.05, (duration - tt) / 0.05); - v += Math.sin(sweepPhase) * 0.07 * edge; - + // (The 220β†’880Hz position-sweep diagnostic is gone β€” the tapes + // are musical material now; position lives in the visuals.) pcm[s] = Math.tanh(v) * 32767; // Soft-clip the mix. } @@ -11930,11 +11940,16 @@ async function boot(parsed, bpm = 60, resolution, debug) { cachedCode: window.currentRecordingOptions?.cachedCode || null, }; - // Calculate total duration from frame timestamps - if (recordedFrames.length > 1) { + // Total duration must include the LAST frame's display time β€” + // mediaRecorderDuration has it. The timestamp span alone runs one + // frame short, which made the piece's wrap point land ~a frame + // before the audio loop's and drift by that much every lap. + if (mediaRecorderDuration > 0) { + info.totalDuration = mediaRecorderDuration / 1000; + } else if (recordedFrames.length > 1) { const firstTimestamp = recordedFrames[0][0]; const lastTimestamp = recordedFrames[recordedFrames.length - 1][0]; - info.totalDuration = (lastTimestamp - firstTimestamp) / 1000; // Convert to seconds + info.totalDuration = (lastTimestamp - firstTimestamp) / 1000; } // Reply back to the disk @@ -12308,6 +12323,32 @@ async function boot(parsed, bpm = 60, resolution, debug) { return; } + if (type === "tape:audio-pos") { + // Absolute read-head position (0..1) for the tape audio β€” used by + // beat jumps and chop repeats to relocate the free-running loop. + const tapeAudioId = Object.keys(sfxPlaying).find((id) => + id.startsWith("tape:audio_"), + ); + if (tapeAudioId && sfxPlaying[tapeAudioId]) { + const p = typeof content === "number" ? Math.max(0, Math.min(1, content)) : 0; + sfxPlaying[tapeAudioId].update({ samplePosition: p }); + } + return; + } + + if (type === "tape:audio-rate") { + // Absolute playback rate for the tape audio β€” drift-proof, unlike + // accumulating relative shifts against a piece-side estimate. + const tapeAudioId = Object.keys(sfxPlaying).find((id) => + id.startsWith("tape:audio_"), + ); + if (tapeAudioId && sfxPlaying[tapeAudioId]) { + const rate = typeof content === "number" ? content : 1; + sfxPlaying[tapeAudioId].update({ sampleSpeed: rate }); + } + return; + } + // πŸ“Ό Load and parse tape ZIP from URL (for replay piece) if (type === "tape:load-zip") { console.log("πŸ“Ό Loading tape ZIP from:", content); @@ -15321,10 +15362,6 @@ async function boot(parsed, bpm = 60, resolution, debug) { let tapeSoundId; - // πŸ•°οΈ Absolute grid for musical loop timing β€” see the loop branch. - let tapeLoopGridEpoch = null; - let tapeLoopGridCount = 0; - stopTapePlayback = () => { continuePlaying = false; stopped = true; @@ -15419,9 +15456,6 @@ async function boot(parsed, bpm = 60, resolution, debug) { // Adjust playback start time to match new position const currentTime = performance.now(); playbackStart = currentTime - (progress * playbackDurationMs); - // A seek breaks the musical loop grid β€” re-anchor at next loop. - tapeLoopGridEpoch = null; - tapeLoopGridCount = 0; // Update display with new frame if (recordedFrames[f]) { @@ -15617,7 +15651,11 @@ async function boot(parsed, bpm = 60, resolution, debug) { } // Advance frames while playback has progressed past the current frame's time - if (f >= recordedFrames.length - 1) { + if (isScrubbing && !render) { + // While the piece is speed-scrubbing, its seeks own the frame + // index β€” the RAF clock must not advance frames at 1Γ— between + // seeks (at slow rates that read as jitter and inaccuracy). + } else if (f >= recordedFrames.length - 1) { // For video export, don't loop - complete when all frames are processed if (doneCb && render) { console.log(`🎬 πŸ“Ή Video export reaching completion - final frame processed`); @@ -15627,25 +15665,12 @@ async function boot(parsed, bpm = 60, resolution, debug) { // For normal playback, loop f = 0; - // πŸ•°οΈ Musical loop grid: anchor each loop start to an absolute - // grid so the period can't accumulate RAF drift (measured - // ~-31ms/loop before). The next loop simply begins at - // epoch + N Γ— duration; if the frames arrive early the grid - // holds tempo, if late they catch up. + // πŸ•°οΈ Net-time phase lock: every loop boundary anchors to UTC + // multiples of the tape length, so any two players of the + // same tape converge into phase β€” and the local period can't + // accumulate RAF drift either. const gridNow = performance.now(); - if ( - tapeLoopGridEpoch === null || - Math.abs(gridNow - (tapeLoopGridEpoch + (tapeLoopGridCount + 1) * mediaRecorderDuration)) > - mediaRecorderDuration / 2 - ) { - // First loop or a broken grid (seek/suspend) β€” re-anchor. - tapeLoopGridEpoch = gridNow; - tapeLoopGridCount = 0; - playbackStart = gridNow; - } else { - tapeLoopGridCount += 1; - playbackStart = tapeLoopGridEpoch + tapeLoopGridCount * mediaRecorderDuration; - } + playbackStart = gridNow - (Date.now() % mediaRecorderDuration); playbackProgress = 0; // The next update's f===0 branch would reset playbackStart to // "now", clobbering the grid β€” isResuming makes it skip once. @@ -15670,7 +15695,7 @@ async function boot(parsed, bpm = 60, resolution, debug) { window.__lastTapeLoopAt = gridNow; if (loopPeriod) { console.log( - `πŸ•°οΈ Video loop period ${loopPeriod.toFixed(1)}ms vs tape ${mediaRecorderDuration.toFixed(1)}ms (drift ${(loopPeriod - mediaRecorderDuration).toFixed(1)}ms, grid loop #${tapeLoopGridCount})`, + `πŸ•°οΈ Video loop period ${loopPeriod.toFixed(1)}ms vs tape ${mediaRecorderDuration.toFixed(1)}ms (drift ${(loopPeriod - mediaRecorderDuration).toFixed(1)}ms, utc phase ${(Date.now() % mediaRecorderDuration).toFixed(0)}ms)`, ); } } diff --git a/system/public/aesthetic.computer/disks/video.mjs b/system/public/aesthetic.computer/disks/video.mjs index 9846c227f..43e6e0586 100644 --- a/system/public/aesthetic.computer/disks/video.mjs +++ b/system/public/aesthetic.computer/disks/video.mjs @@ -136,6 +136,26 @@ let flickVel = 0; // Low-passed drag velocity, for telling flicks from lets-go const FLICK_THRESHOLD = 5; // px/event of recent drag velocity const FLICK_KICK = 0.12; // Extra rate per px of flick velocity +// πŸ“Ÿ Steady-rate dial: dragging the top-right rate readout vertically sets +// a held steady rate β€” friction leaves it alone until it's brought back to +// 1Γ— (or spacebar resets). +let rateBtn = null; +let steadyHold = false; +let rateDragStartY = 0; +let rateDragStartRate = 1; + +// 🧭 Gesture vector debug: the grab anchor and current finger, so the +// stretch reads as a drawn vector (direction + energy) on screen. +let elasticAnchorY = null; +let penX = null; +let penY = null; + +// πŸŽ›οΈ Deck keys (Pioneer-style): ← β†’ beat-jump; holding ↑/↓ chop-repeats a +// beat fraction. chopActive holds the slice length in beats (0 = off). +let chopActive = 0; +let chopStart = 0; +const BEAT_SEC = 0.5; // 120 BPM + // Scrub physics runs on measured wall time, not an assumed tick rate β€” // sim ticks at ~120Hz here, and a fixed 1/60 dt made every commanded // speed land at roughly half its real value. @@ -207,10 +227,11 @@ function ensureScrubStripButton(ui, screen, enabled) { } function nudgeTapeAudioSpeed(send, targetSpeed) { - const clampedTarget = Math.max(-16, Math.min(16, targetSpeed)); - const delta = clampedTarget - scrubAudioSpeed; - if (Math.abs(delta) < 0.0005) return; - send({ type: "tape:audio-shift", content: delta }); + // Absolute rate: the worklet is set to exactly this speed, so repeated + // gestures can never accumulate drift the way relative shifts did. + const clampedTarget = Math.max(-24, Math.min(24, targetSpeed)); + if (Math.abs(clampedTarget - scrubAudioSpeed) < 0.0005) return; + send({ type: "tape:audio-rate", content: clampedTarget }); scrubAudioSpeed = clampedTarget; } @@ -313,6 +334,9 @@ function boot({ wipe, rec, gizmo, jump, notice, store, params, send, hud }) { lastScrollAt = 0; wheelActive = false; sustained = false; + steadyHold = false; + rateBtn = null; + chopActive = 0; flickVel = 0; resumeTarget = 1; dipBase = 1; @@ -444,7 +468,9 @@ function boot({ wipe, rec, gizmo, jump, notice, store, params, send, hud }) { ? "house" : rest.includes("dub") ? "dub" - : "bed"; + : rest.includes("sine") || rest.includes("line") + ? "sine" + : "bed"; const nums = rest.map(parseFloat).filter((n) => Number.isFinite(n)); let duration = nums[0] || 8; // Four bars by default, break included if (rest.includes("bar")) duration = 2; @@ -485,6 +511,7 @@ function paint({ sound, send, num, + clock, }) { if (typeof needsPaint === "function") { requestPaint = needsPaint; @@ -848,11 +875,21 @@ function paint({ sustained || tapDipTime >= 0; const liveRate = scrubDriven ? scrubSpeed : playing ? 1 : 0; - ink(255, 255, 0).write(`${liveRate.toFixed(2)}x`, { - x: screen.width - 6, - y: 6, - right: true, - }); + ink(steadyHold ? [0, 255, 180] : [255, 255, 0]).write( + `${liveRate.toFixed(2)}x`, + { x: screen.width - 6, y: 6, right: true }, + ); + + // πŸ“Ÿ The readout is also a dial β€” keep its hit area in place. + if (!rateBtn) { + rateBtn = new ui.Button(screen.width - 64, 0, 64, 24); + rateBtn.stickyScrubbing = true; + rateBtn.noRolloverActivation = true; + } + rateBtn.box.x = screen.width - 64; + rateBtn.box.y = 0; + rateBtn.box.w = 64; + rateBtn.box.h = 24; // πŸ”΄ The red marker rides the bottom edge at the actual playback // position β€” drawn from live state so it never lies. @@ -865,6 +902,56 @@ function paint({ 6, 4, ); + + // πŸ•°οΈ Net-time unison readout: phase offset vs the AC network clock's + // grid β€” green when locked, amber while converging. + if (tapeInfo?.totalDuration) { + const durMs = tapeInfo.totalDuration * 1000; + const nowMs = clock?.time?.()?.getTime?.() ?? Date.now(); + let phaseErr = (nowMs % durMs) / durMs - livePos; + if (phaseErr > 0.5) phaseErr -= 1; + else if (phaseErr < -0.5) phaseErr += 1; + const errMs = Math.round(phaseErr * durMs); + const locked = Math.abs(errMs) < 60; + ink(locked ? [0, 255, 120] : [255, 170, 0]).write( + `sync ${errMs >= 0 ? "+" : ""}${errMs}ms`, + { x: screen.width - 6, y: 18, right: true }, + ); + } + + // 🧭 Gesture vector: anchor β†’ finger, with the horizontal component + // (the part that drives the rate) emphasized. + if (isScrubbing && elasticAnchorX !== null && penX !== null) { + const ay = elasticAnchorY ?? penY ?? 0; + ink(255, 255, 255, 70).line(elasticAnchorX, ay, penX, penY ?? ay); + const fwd = penX >= elasticAnchorX; + ink(fwd ? [0, 255, 120, 200] : [255, 80, 80, 200]).box( + Math.min(elasticAnchorX, penX), + (ay) - 1, + Math.max(1, Math.abs(penX - elasticAnchorX)), + 3, + ); + ink(255, 255, 255, 220).box(elasticAnchorX - 2, ay - 2, 5, 5); + ink(255, 255, 255).write(`${scrubSpeed.toFixed(2)}x`, { + x: penX + 8, + y: (penY ?? ay) - 4, + }); + } + + // ⌨️ Deck keys legend, bottom-right. + const keyLines = [ + "<- -> beat jump", + "hold ^ 1/4 chop", + "hold v 1/8 chop", + "space reset", + ]; + keyLines.forEach((l, i) => { + ink(255, 255, 255, 110).write(l, { + x: screen.width - 6, + y: screen.height - 10 - (keyLines.length - i) * 10, + right: true, + }); + }); } // Scrub overlay (STAMPLE-style speed-based) @@ -950,7 +1037,7 @@ function paint({ return true; // Always keep painting } -function sim({ needsPaint, rec, send }) { +function sim({ needsPaint, rec, send, clock }) { ellipsisTicker?.sim(); frameCount++; // Increment frame counter for animations @@ -1033,18 +1120,10 @@ function sim({ needsPaint, rec, send }) { const phase = Math.min(1, tapDipTime / TAP_DIP_DURATION); scrubSpeed = dipBase * (1 - TAP_DIP_DEPTH * Math.sin(Math.PI * phase)); if (phase >= 1) { + // Lands as a seamless park at the departed rate β€” no handoff jump. tapDipTime = -1; - if (Math.abs(dipBase - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = dipBase; - sustained = true; - } + scrubSpeed = dipBase; + sustained = true; } } @@ -1091,21 +1170,14 @@ function sim({ needsPaint, rec, send }) { if (scrubSpeed < 0.02) scrubSpeed = 0; } - // πŸ–±οΈ Scroll release: the wheel went quiet β€” park at the landed rate. + // πŸ–±οΈ Scroll release: the wheel went quiet β€” a seamless park at the + // landed rate (pinned to exactly 1 when close). if (scrollScrubbing && isScrubbing && performance.now() - lastScrollAt > 150) { scrollScrubbing = false; tapDipTime = -1; isScrubbing = false; - if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - sustained = true; - } + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) scrubSpeed = 1; + sustained = true; } // 🎰 Wheel spin-down: after a flick the platter runs free, then eases @@ -1114,33 +1186,50 @@ function sim({ needsPaint, rec, send }) { scrubSpeed = resumeTarget + (scrubSpeed - resumeTarget) * Math.pow(0.975, rate); if (Math.abs(scrubSpeed - resumeTarget) < 0.04) { + // Lands as a seamless park β€” the drive keeps running, no handoff. wheelActive = false; - if (Math.abs(resumeTarget - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = resumeTarget; - sustained = true; - } + scrubSpeed = resumeTarget; + sustained = true; } } // πŸŒ€ Friction: even a parked rate isn't forever β€” it glides home to // 1Γ— over a few seconds, like a wheel that always feels the bearing. - if (sustained && !isScrubbing) { + // It never hands off: the drive just converges to exactly 1.0 and + // keeps driving β€” seamless by construction. + if (sustained && !steadyHold && !isScrubbing && !chopActive) { scrubSpeed += (1 - scrubSpeed) * (1 - Math.pow(0.9965, rate)); - if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - sustained = false; - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); + if (Math.abs(scrubSpeed - 1) < 0.005) scrubSpeed = 1; + + // πŸ•°οΈ Net-time phase pull: at rest the loop always lerps toward the + // AC network clock's phase grid (clock.mjs / /api/clock synced), so + // every player of this tape converges into unison β€” a gentle Β±5% + // tempo lean, never a jump. + if (Math.abs(scrubSpeed - 1) < 0.01) { + const durMs = totalDuration * 1000; + const nowMs = clock?.time?.()?.getTime?.() ?? Date.now(); + const target = (nowMs % durMs) / durMs; + let phaseErr = target - scrubCurrentProgress; + if (phaseErr > 0.5) phaseErr -= 1; + else if (phaseErr < -0.5) phaseErr += 1; + scrubSpeed = 1 + Math.max(-0.05, Math.min(0.05, phaseErr * 0.15)); + } + } + + // πŸŒ€ Chop repeat: while ↑/↓ is held, loop a beat-fraction slice β€” + // glitch stutter, Pioneer-style. + if (chopActive && rec?.presenting) { + const chopLen = (chopActive * BEAT_SEC) / totalDuration; + let rel = scrubCurrentProgress - chopStart; + if (rel < -0.5) rel += 1; + else if (rel > 0.5) rel -= 1; + if (rel >= chopLen || rel < 0) { + scrubCurrentProgress = chopStart; send({ type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, + content: { progress: chopStart, speedScrub: true }, }); + send({ type: "tape:audio-pos", content: chopStart }); } } @@ -1148,18 +1237,10 @@ function sim({ needsPaint, rec, send }) { if (brakeResume && !isScrubbing) { scrubSpeed += (resumeTarget - scrubSpeed) * (1 - Math.pow(0.88, rate)); if (Math.abs(scrubSpeed - resumeTarget) < 0.05) { + // Lands as a seamless park β€” the drive keeps running, no handoff. brakeResume = false; - if (Math.abs(resumeTarget - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = resumeTarget; - sustained = true; - } + scrubSpeed = resumeTarget; + sustained = true; } } @@ -2263,6 +2344,8 @@ function act({ sustained = false; scrollScrubbing = false; tapDipTime = -1; + steadyHold = false; + chopActive = 0; scrubSpeed = 0; nudgeTapeAudioSpeed(send, 1); if (rec.playing) rec.pause(); @@ -2271,6 +2354,41 @@ function act({ return; } + // πŸŽ›οΈ Deck keys, Pioneer-style: ← β†’ beat-jump the tape (audio included); + // holding ↑ chop-repeats a quarter beat, ↓ an eighth β€” release to run on. + if (rec.presenting && tapeInfo?.totalDuration) { + const dur = tapeInfo.totalDuration; + const beatJump = (beats) => { + ensureDriven(rec, send); + let p = scrubCurrentProgress + (beats * BEAT_SEC) / dur; + p = ((p % 1) + 1) % 1; + scrubCurrentProgress = p; + send({ + type: "recorder:present:seek", + content: { progress: p, speedScrub: true }, + }); + send({ type: "tape:audio-pos", content: p }); + triggerRender(); + }; + if (e.is("keyboard:down:arrowleft")) beatJump(-1); + if (e.is("keyboard:down:arrowright")) beatJump(1); + if (e.is("keyboard:down:arrowup") && !chopActive) { + ensureDriven(rec, send); + chopActive = 0.25; + chopStart = scrubCurrentProgress; + send({ type: "tape:audio-pos", content: chopStart }); + } + if (e.is("keyboard:down:arrowdown") && !chopActive) { + ensureDriven(rec, send); + chopActive = 0.125; + chopStart = scrubCurrentProgress; + send({ type: "tape:audio-pos", content: chopStart }); + } + if (e.is("keyboard:up:arrowup") || e.is("keyboard:up:arrowdown")) { + chopActive = 0; + } + } + // πŸ–±οΈ Two-finger scroll scrubs directly β€” no tap-drag needed. Wheel // deltas bend the rate; when the gesture stops it springs back to 1Γ—. if (e.is("scroll") && rec.presenting) { @@ -2297,15 +2415,61 @@ function act({ } scrubMoved = true; // Expressive range: two-finger scroll can push way past the drag's - // reach β€” up to Β±12Γ—. Negated so scroll direction matches drag + // reach β€” up to Β±24Γ—. Negated so scroll direction matches drag // direction (natural scrolling inverts the wheel deltas). - scrubSpeed = Math.max(-12, Math.min(12, scrubSpeed * 0.6 - d * 0.35)); + scrubSpeed = Math.max(-24, Math.min(24, scrubSpeed * 0.6 - d * 0.35)); lastScrollAt = performance.now(); nudgeTapeAudioSpeed(send, scrubSpeed); triggerRender(); } } + // πŸ“Ÿ Steady-rate dial: vertical drag on the top-right readout sets a + // held rate the friction won't touch. + if (rateBtn && rec.presenting && !isPrinting && !isPostingTape) { + rateBtn.act(e, { + down: () => { + rateDragStartY = e.y ?? 0; + rateDragStartRate = + isScrubbing || inertiaActive || brakeResume || wheelActive || sustained || tapDipTime >= 0 + ? scrubSpeed + : rec.playing + ? 1 + : 0; + }, + scrub: () => { + if (e.y === undefined) return; + if (!sustained && !isScrubbing) { + // Engage the drive so the dialed rate actually plays. + scrubCurrentProgress = rec.presentProgress || scrubCurrentProgress || 0; + if (!rec.playing) rec.play(); + send({ + type: "recorder:present:seek", + content: { progress: scrubCurrentProgress, speedScrub: true, scrubbing: true }, + }); + } + inertiaActive = false; + brakeResume = false; + wheelActive = false; + tapDipTime = -1; + sustained = true; + steadyHold = true; + const dyRate = (rateDragStartY - e.y) * 0.03; + scrubSpeed = Math.max(-24, Math.min(24, rateDragStartRate + dyRate)); + nudgeTapeAudioSpeed(send, scrubSpeed); + triggerRender(); + }, + up: () => { + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { + scrubSpeed = 1; + steadyHold = false; // Back at play speed β€” friction may hold it + } + triggerRender(); + }, + }); + if (rateBtn.down) return; // The dial owns this gesture + } + if (!anyButtonDown && !isPrinting && !isPostingTape && rec.presenting) { ensureScrubStripButton(ui, screen, true); scrubStripBtn?.act(e, { @@ -2319,6 +2483,9 @@ function act({ holdTime = 0; flickVel = 0; elasticAnchorX = e.x ?? null; + elasticAnchorY = e.y ?? null; + penX = e.x ?? null; + penY = e.y ?? null; // πŸ…ΏοΈ A parked, spinning, or ramping rate survives a new touch: // the gesture starts FROM it β€” so you can grab a fast wheel and // drag it slower β€” and returns TO it. @@ -2362,6 +2529,8 @@ function act({ // grab point IS the rate β€” hold still and it holds still, a third // of the screen right drags at 3Γ—, left drags in reverse. const dx = e.x - elasticAnchorX; + penX = e.x; + penY = e.y ?? penY; if (Math.abs(dx) > 2) scrubMoved = true; flickVel = flickVel * 0.6 + (e.delta?.x || 0) * 0.4; const K = 9 / screen.width; @@ -2382,22 +2551,18 @@ function act({ // 🎰 Flick: the platter runs free with the throw's momentum, // then eases down like a prize wheel to the pre-flick rate. scrubSpeed = Math.max( - -12, - Math.min(12, scrubSpeed + flickVel * FLICK_KICK), + -24, + Math.min(24, scrubSpeed + flickVel * FLICK_KICK), ); nudgeTapeAudioSpeed(send, scrubSpeed); wheelActive = true; - } else if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - // Close enough to play speed β€” hand back to normal playback. - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); } else { - // πŸ…ΏοΈ Gentle release: the rate parks where you left it. + // πŸ…ΏοΈ Release parks where you left it. Near 1Γ— it pins to + // exactly 1 β€” the scrub drive at 1.0 IS normal playback, so + // there's no handoff and no jump, ever. + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) scrubSpeed = 1; sustained = true; + nudgeTapeAudioSpeed(send, scrubSpeed); } triggerRender(); return true; @@ -3048,6 +3213,7 @@ function leave({ send }) { tapDipTime = -1; wheelActive = false; sustained = false; + steadyHold = false; flickVel = 0; scrubSpeed = 0; scrubCurrentProgress = 0; @@ -3069,6 +3235,29 @@ export { boot, paint, sim, act, signal, receive, leave }; // πŸ“š Library (Useful functions used throughout the piece) +// πŸŽ›οΈ Make sure the scrub drive owns playback (used by deck keys) β€” engages +// a sustained 1Γ— drive if nothing else is driving. +function ensureDriven(rec, send) { + if ( + isScrubbing || + sustained || + wheelActive || + brakeResume || + inertiaActive || + tapDipTime >= 0 + ) { + return; + } + scrubCurrentProgress = rec.presentProgress || 0; + scrubSpeed = 1; + sustained = true; + if (!rec.playing) rec.play(); + send({ + type: "recorder:present:seek", + content: { progress: scrubCurrentProgress, speedScrub: true, scrubbing: true }, + }); +} + // πŸ§ͺ Synthtape autopilot β€” a scripted stand-in for the finger. Each segment // either lets the tape roll, holds a scrub speed, or releases into inertia, // then the observed progress delta is graded against the physics. All output @@ -3092,6 +3281,8 @@ function autopilot(rec, send, simDt) { { name: "tap dip", mode: "dip" }, { name: "scratch", mode: "scratch", frames: 150 }, { name: "scratch release", mode: "coast" }, + { name: "fast scratch", mode: "scratch", frames: 120, hz: 4, amp: 2.2 }, + { name: "fast scratch release", mode: "coast" }, { name: "touch brake", mode: "brake", frames: 90 }, { name: "brake release spin-up", mode: "brakeRelease" }, ]; @@ -3200,8 +3391,11 @@ function autopilot(rec, send, simDt) { autoTimer += 1; if (autoSeg.mode === "scratch") { - // 🎚️ Musical scratch: the rate rocks Β±3.2Γ— at ~2Hz like a hand on the reel. - scrubSpeed = 3.2 * Math.sin((autoTimer / 60) * Math.PI * 2); + // 🎚️ Musical scratch: the rate rocks like a hand on the reel β€” + // amplitude and tempo per segment (default Β±3.2Γ— at 2Hz). + const amp = autoSeg.amp || 3.2; + const hz = autoSeg.hz || 2; + scrubSpeed = amp * Math.sin((autoTimer / 120) * Math.PI * 2 * hz); } if (autoSeg.mode === "scrub") { scrubSpeed = autoSeg.speed; // Hold the drag @@ -3268,8 +3462,10 @@ function autopilot(rec, send, simDt) { } } else if (autoSeg.mode === "scratch") { const totalDuration = tapeInfo?.totalDuration || 10; + const amp = autoSeg.amp || 3.2; const driftSecs = Math.abs(autoMotionSum * totalDuration); - pass = autoSpeedMax > 1.5 && autoSpeedMin < -1.5 && driftSecs < 0.7; + pass = + autoSpeedMax > amp * 0.45 && autoSpeedMin < -amp * 0.45 && driftSecs < 0.7; detail += ` (rate swung ${autoSpeedMin.toFixed(1)}..${autoSpeedMax.toFixed(1)}Γ—, net drift ${driftSecs.toFixed(2)}s)`; } else if (autoSeg.mode === "dip") { const totalDuration = tapeInfo?.totalDuration || 10; -- 2.51.2 From 75e477d5b61370e02f555993a411a96ca4033bb1 Mon Sep 17 00:00:00 2001 From: "prompt.ac/@jeffrey" Date: Sun, 19 Jul 2026 14:28:42 -0700 Subject: [PATCH 2/2] =?UTF-8?q?spinging:=20round-6=20=E2=80=94=20scales=20?= =?UTF-8?q?singalong=20reel,=20+12=20register=20lift=20with=20fallback=20l?= =?UTF-8?q?adder?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- pop/menuband/bin/reel-lib.mjs | 9 +- pop/menuband/bin/render-jingles.mjs | 98 ++++++++- pop/menuband/bin/sim-scales.mjs | 200 +++++++++++++++++ pop/menuband/bin/sing-jingle.mjs | 318 ++++++++++++++++++++++++---- spinging/README.md | 38 ++++ spinging/cache/pronounce.json | 98 +++++++++ spinging/lib/pronounce.mjs | 16 +- spinging/lib/sing_line_world.py | 102 +++++++-- spinging/lib/vocal_shapes.py | 32 ++- 9 files changed, 842 insertions(+), 69 deletions(-) create mode 100644 pop/menuband/bin/sim-scales.mjs diff --git a/pop/menuband/bin/reel-lib.mjs b/pop/menuband/bin/reel-lib.mjs index 1c77099a2..3eab92cbd 100644 --- a/pop/menuband/bin/reel-lib.mjs +++ b/pop/menuband/bin/reel-lib.mjs @@ -165,9 +165,12 @@ export function makeParticles(ctx) { } // ── the strip rig β€” real captured strip, re-lightable per note set ───────── -/// Midis the cache holds single-note captures for (the waltz's white keys, -/// G4..B5). Any melody is folded onto these before lighting. -export const STRIP_MIDIS = [67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; +/// Midis the cache holds single-note captures for: the waltz's white keys +/// (G4..B5) + the low C4..F4 the scales ladder added (round 6 β€” captured via +/// `MenuBand --render-menubar --notes --light`, same args as sim.mjs). +/// Together that's every white key on the strip, C4..B5. Any melody is +/// folded onto these before lighting. +export const STRIP_MIDIS = [60, 62, 64, 65, 67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; const PC_TO_STRIP = new Map([[0, 72], [2, 74], [4, 76], [5, 77], [7, 79], [9, 69], [11, 71]]); /// Fold any (white-key) midi onto a strip key: same pitch class, the cached diff --git a/pop/menuband/bin/render-jingles.mjs b/pop/menuband/bin/render-jingles.mjs index 88b131d1f..c207a4ae6 100644 --- a/pop/menuband/bin/render-jingles.mjs +++ b/pop/menuband/bin/render-jingles.mjs @@ -1,10 +1,13 @@ #!/usr/bin/env node -// render-jingles.mjs β€” the three campaign jingles for the Menu Band promo -// reels (announce / features / chords), built on the same /pop lullaby engine -// as the launch waltz (render-waltz.mjs). Each jingle writes an mp3 + a -// notes.json its sim choreographs to; the chords jingle also writes a -// segment score (menuband-chords.score.json) so the audio and the on-screen -// modifier keycaps agree to the frame. +// render-jingles.mjs β€” the four campaign jingles for the Menu Band promo +// reels (announce / features / chords / scales), built on the same /pop +// lullaby engine as the launch waltz (render-waltz.mjs). Each jingle writes +// an mp3 + a notes.json its sim choreographs to; the chords jingle also +// writes a segment score (menuband-chords.score.json) so the audio and the +// on-screen modifier keycaps agree to the frame; the scales jingle writes +// menuband-scales.score.json (spoken cue times + the letter ladder) that +// sing-jingle.mjs and sim-scales.mjs both read, so the voice, the bed and +// the visuals share ONE timeline. // // EVERY lead note is a WHITE KEY: the strip rig re-lights the real captured // menu-bar piano from the single-note captures sim.mjs cached (G4..B5, @@ -201,3 +204,86 @@ function master(name, events, opts) { JSON.stringify({ durationSec: +durationSec.toFixed(4), segs }, null, 2)); console.log(` ${segs.length} chord segments`); } + +// ════════════════════════════════════════════════════════════════════════ +// 4 Β· SCALES β€” the teaching singalong (round 6). Spoken frame + the notepat +// two-octave letter ladder (c d e f g a b Β· h i j k l m n = the second +// octave, straight from notepat's NOTE_TO_KEYBOARD_KEY) sung up then +// down at 108 BPM over a soft drone bed. The `lead` lane carries the +// ladder at the STRIP's midis (C4..B5 β€” every one a real cached key); +// jeffrey sings it an octave below (C3..B4, bright top, unstrained). +// The bed stays light β€” a pedal drone, gentle kalimba, a quiet +// vibraphone doubling the ladder β€” never burying the voice. +// ════════════════════════════════════════════════════════════════════════ +{ + const BPM = 108, BEAT = 60 / BPM, BAR = 4 * BEAT; + const { events, push } = makeTrack(); + + // ── the shared timeline (spoken cues measured off the cached TTS takes) ── + const INTRO_T = 0.8; // "Here's how to type out the C scale." (~2.1s) + const RUN_T = 3.5; // "C, D, E, F, G, A, B. Now sing it!" (~2.9s) + const SING0 = 6.95; // the ladder's first beat + const LETTERS_ASC = ["c", "d", "e", "f", "g", "a", "b", + "h", "i", "j", "k", "l", "m", "n"]; + const STRIP_ASC = [60, 62, 64, 65, 67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; + const HOLD_TOP = 2.0, HOLD_BOTTOM = 2.5; // beats: top n rings, home c lands + const ladder = []; + LETTERS_ASC.forEach((letter, i) => { + const top = i === LETTERS_ASC.length - 1; + ladder.push({ letter, t: +(SING0 + i * BEAT).toFixed(4), + dur: +((top ? HOLD_TOP : 0.92) * BEAT).toFixed(4), + strip: STRIP_ASC[i], vocal: STRIP_ASC[i] - 12, dir: "up" }); + }); + // descending starts on m after the top-n hold (2 beats of ring) + const DESC0 = SING0 + (LETTERS_ASC.length + 1) * BEAT; + for (let j = 0; j < 13; j++) { + const i = 12 - j; // m l k j i h b a g f e d c + const bottom = j === 12; + ladder.push({ letter: LETTERS_ASC[i], t: +(DESC0 + j * BEAT).toFixed(4), + dur: +((bottom ? HOLD_BOTTOM : 0.92) * BEAT).toFixed(4), + strip: STRIP_ASC[i], vocal: STRIP_ASC[i] - 12, dir: "down" }); + } + const SING1 = DESC0 + 12 * BEAT + HOLD_BOTTOM * BEAT; // ladder ends + const OUTRO_T = +(SING1 + 0.9).toFixed(3); // "Wanna type it yourself? …" + + // ── the bed ────────────────────────────────────────────────────────────── + // pedal drone: soft C2/G2 bass, one note per bar, entering under the intro + const LAST_BAR = Math.ceil((OUTRO_T + 3.2) / BAR); + for (let bar = 0; bar < LAST_BAR; bar++) { + const t = bar * BAR; + push("bass", "bass", t, bar % 4 === 3 ? "G2" : "C2", 3.6 * BEAT, 0.20, 0); + } + // gentle kalimba broken chord, quiet, from the spoken run onward + const ARP = ["C4", "E4", "G4", "A3"]; + for (let bar = 1; bar < LAST_BAR - 1; bar++) { + for (let s = 0; s < 4; s++) { + push("harmony", "kalimba", bar * BAR + s * BEAT, ARP[s % 4], + 0.9 * BEAT, 0.085, s % 2 ? 0.2 : -0.2, 1.0); + } + } + // the ladder itself: lead lane at STRIP midis β€” lights the real keys and + // doubles the voice an octave up, quietly + for (const n of ladder) { + push("lead", "vibraphone", n.t, n.strip, n.dur * 1.15, 0.22, 0.05, 1.25); + } + // soft woodblock ticks keep the singalong honest through the ladder bars + for (let t = SING0; t < SING1 - BEAT; t += 2 * BEAT) { + push("perc", "woodblock", +t.toFixed(4), "E5", 0.12, 0.06, 0.12); + } + // home-chord sparkle under the outro CTA + push("harmony", "glockenspiel", OUTRO_T + 0.3, m("E6"), 2.0, 0.08, 0.28, 1.5); + push("harmony", "glockenspiel", OUTRO_T + 0.9, m("C6"), 2.4, 0.09, -0.2, 1.5); + + const durationSec = master("menuband-scales", events, { title: "Menu Band C Scale Singalong" }); + writeScore("menuband-scales", events, {}, { + bpm: BPM, beatSec: +BEAT.toFixed(6), barSec: +BAR.toFixed(6), + durationSec: +durationSec.toFixed(4), + }); + writeFileSync(resolve(OUT_DIR, "menuband-scales.score.json"), JSON.stringify({ + bpm: BPM, beatSec: +BEAT.toFixed(6), durationSec: +durationSec.toFixed(4), + spoken: { intro: { t: INTRO_T }, run: { t: RUN_T }, outro: { t: OUTRO_T } }, + sing: { t0: +SING0.toFixed(4), t1: +SING1.toFixed(4) }, + ladder, + }, null, 2)); + console.log(` ladder ${ladder.length} letters Β· sing ${SING0.toFixed(2)}–${SING1.toFixed(2)}s Β· outro @ ${OUTRO_T}s`); +} diff --git a/pop/menuband/bin/sim-scales.mjs b/pop/menuband/bin/sim-scales.mjs new file mode 100644 index 000000000..93dc98566 --- /dev/null +++ b/pop/menuband/bin/sim-scales.mjs @@ -0,0 +1,200 @@ +#!/usr/bin/env node +// sim-scales.mjs β€” the Menu Band SCALES teaching-singalong reel base (v6). +// +// The reel that literally demonstrates the spinging speakβ†’sing transform: +// a spoken frame ("Here's how to type out the C scale." … "Now sing it!") +// around the notepat two-octave letter ladder β€” c d e f g a b h i j k l m n +// up and back down (h..n = notepat's second octave) β€” each letter sung on +// its own scale degree while the REAL captured menu-bar strip lights that +// key (strip rig, now covering every white key C4..B5) and a BIG keycap +// shows the letter karaoke-chart style. Spoken lines get normal captions. +// +// Timeline comes from out/menuband-scales.score.json (render-jingles.mjs) β€” +// the same sidecar sing-jingle.mjs sings from, so audio, voice and visuals +// share ONE clock. Word timings: out/menuband-scales.words.sung.json. +// +// This reel only exists sung β€” run with --sung (the suffix keeps the +// naming pattern of the other campaign reels): +// node pop/menuband/bin/sing-jingle.mjs menuband-scales +// node pop/menuband/bin/sim-scales.mjs --sung +// node pop/menuband/bin/chrome-reel.mjs menuband-scales-sung + +import { readFileSync } from "node:fs"; +import { + W, H, FPS, OUT, INK, INK_RGB, easeOut, clamp01, rgb, + makeStage, roundRect, text, drawDesktop, vignette, drawIcon, + makeParticles, loadStripRig, drawStrip, stripKeyX, stripKeyColor, + loadScore, leadOf, litAt, makeOnsets, + renderVideo, writeMeta, makeScenes, sungMode, loadSungWords, makeKaraoke, +} from "./reel-lib.mjs"; + +const SLUG = "menuband-scales"; +const { sung: SUNG, suffix: VAR } = sungMode(); +if (!SUNG) { + console.error("βœ— the scales reel only exists sung β€” run with --sung"); + process.exit(1); +} +const score = loadScore(SLUG); +const sc = JSON.parse(readFileSync(`${OUT}/${SLUG}.score.json`, "utf8")); +const TOTAL = score.durationSec; +const lead = leadOf(score); +const onsetsBetween = makeOnsets(lead); + +// spoken words β†’ normal bottom captions; sung letters drive the big keycap +const allWords = loadSungWords(SLUG); +const karaoke = makeKaraoke(allWords.filter((w) => w.spoken), { y: H * 0.925 }); + +const { canvas, ctx } = makeStage(); +const rig = await loadStripRig(); +const particles = makeParticles(ctx); + +const SING0 = sc.sing.t0, SING1 = sc.sing.t1; +const { scenes: SCENES, sceneAt } = makeScenes([ + { name: "intro", from: 0, to: sc.spoken.run.t / TOTAL, tint: [97, 158, 255] }, + { name: "speak", from: sc.spoken.run.t / TOTAL, to: SING0 / TOTAL, tint: [255, 214, 56] }, + { name: "sing-up", from: SING0 / TOTAL, to: (SING0 + (SING1 - SING0) * 0.52) / TOTAL, tint: [51, 209, 179] }, + { name: "sing-down", from: (SING0 + (SING1 - SING0) * 0.52) / TOTAL, to: (SING1 + 0.8) / TOTAL, tint: [167, 139, 250] }, + { name: "end", from: (SING1 + 0.8) / TOTAL, to: 1.0, tint: [255, 77, 107] }, +], TOTAL); + +// ── the strip: parked upper area, playing the ladder ─────────────────────── +const HERO_W = W * 0.96, HERO_X = (W - HERO_W) / 2; +function heroRect(t) { + const h = HERO_W / rig.aspect; + const enter = easeOut(clamp01(t / 1.1)); + const bob = Math.sin(t * 1.5) * 7 * enter; + const rest = H * 0.165 - h / 2; + return { x: HERO_X, y: (-h - 40) + (rest - (-h - 40)) * enter + bob, w: HERO_W, h }; +} + +// ── the ladder clock: which letter is active at t ────────────────────────── +const LADDER = sc.ladder; // [{ letter, t, dur, strip, vocal, dir }] +function activeNote(t) { + let cur = null; + for (const n of LADDER) if (t >= n.t) cur = n; else break; + if (!cur) return null; + if (t > cur.t + Math.max(cur.dur, 0.5) + 0.35) return null; // ladder done + return cur; +} +const LETTERS = [...new Set(LADDER.map((n) => n.letter))]; // c..n in order + +// ── the BIG teaching keycap β€” the letter you type, huge, in the key's own +// strip color, pressing on every onset ───────────────────────────────────── +function drawKeycap(t) { + const n = activeNote(t); + if (!n) return; + const local = clamp01((t - n.t) / 0.14); + const press = 1 - easeOut(local); // 1 at strike β†’ 0 settled + const inA = easeOut(clamp01((t - SING0 + 0.4) / 0.5)) * + (1 - easeOut(clamp01((t - (SING1 + 0.35)) / 0.5))); + if (inA <= 0) return; + const size = 430; + const x = W / 2 - size / 2, y = H * 0.335 - size / 2 + press * 14; + const col = stripKeyColor(rig, n.strip); + ctx.save(); + ctx.globalAlpha = inA; + ctx.shadowColor = "rgba(0,0,0,0.32)"; + ctx.shadowBlur = 34 * (1 - press * 0.7); ctx.shadowOffsetY = 16 * (1 - press * 0.7); + roundRect(ctx, x, y, size, size, 64); + ctx.fillStyle = "rgb(250,249,253)"; ctx.fill(); + ctx.shadowColor = "transparent"; + // the key's strip color washes the cap as it presses + ctx.globalAlpha = inA * (0.16 + 0.22 * press); + roundRect(ctx, x, y, size, size, 64); + ctx.fillStyle = rgb(col); ctx.fill(); + ctx.globalAlpha = inA; + ctx.lineWidth = 3; ctx.strokeStyle = "rgba(20,18,28,0.22)"; + roundRect(ctx, x, y, size, size, 64); ctx.stroke(); + const pop = 1 + 0.09 * press; + ctx.translate(W / 2, y + size * 0.47); ctx.scale(pop, pop); + text(ctx, n.letter.toUpperCase(), 0, 0, 250, INK, 800); + ctx.scale(1 / pop, 1 / pop); + text(ctx, n.dir === "up" ? "↑" : "↓", 0, size * 0.36, 54, rgb(col), 800); + ctx.restore(); +} + +// ── the ladder chart β€” all 14 letters, the singalong teaching row ────────── +function drawChart(t) { + const a = easeOut(clamp01((t - SING0 + 0.6) / 0.5)) * + (1 - easeOut(clamp01((t - (SING1 + 0.5)) / 0.5))); + if (a <= 0) return; + const n = activeNote(t); + const activeIdx = n ? LETTERS.indexOf(n.letter) : -1; + const goingUp = !n || n.dir === "up"; + const cellW = 68, y = H * 0.585; + const x0 = W / 2 - (LETTERS.length * cellW) / 2; + ctx.save(); + ctx.globalAlpha = a; + for (let i = 0; i < LETTERS.length; i++) { + const cx = x0 + i * cellW + cellW / 2; + const isActive = i === activeIdx; + const sungAlready = goingUp ? i < activeIdx : i > activeIdx; + if (isActive) { + const col = stripKeyColor(rig, LADDER[i].strip); + roundRect(ctx, cx - 28, y - 34, 56, 68, 16); + ctx.fillStyle = rgb(col); ctx.fill(); + text(ctx, LETTERS[i], cx, y + 1, 44, "rgba(255,255,255,0.98)", 800); + } else { + text(ctx, LETTERS[i], cx, y + 1, 40, + sungAlready ? INK : "rgba(20,18,28,0.32)", 700); + } + } + text(ctx, "c major Β· two octaves Β· h through n = the next c to b", + W / 2, y + 74, 34, "rgba(60,50,80,0.85)", 600); + ctx.restore(); +} + +// ── the end card β€” app icon + menuband.app ───────────────────────────────── +function drawEndCard(t) { + const e = easeOut(clamp01((t - (SING1 + 1.0)) / 0.7)); + if (e <= 0) return; + const cw = W * 0.78, chh = H * 0.40; + const cx = (W - cw) / 2, cy = H * 0.44 - chh / 2 + (1 - e) * H * 0.35; + ctx.save(); ctx.globalAlpha = e; + ctx.shadowColor = "rgba(0,0,0,0.45)"; ctx.shadowBlur = 60; ctx.shadowOffsetY = 24; + roundRect(ctx, cx, cy, cw, chh, 40); ctx.fillStyle = "rgba(250,249,253,0.99)"; ctx.fill(); + ctx.shadowColor = "transparent"; + const litIcon = new Set(litAt(lead, t).map((mm) => ((mm % 12) + 12) % 12 % 5)); + const ipx = 330; + drawIcon(ctx, W / 2 - ipx / 2, cy + 40, ipx, litIcon); + text(ctx, "menuband.app", W / 2, cy + 450, 88, INK, 800); + text(ctx, "free on the Mac App Store", W / 2, cy + 540, 44, "rgba(60,50,80,0.9)", 700); + text(ctx, "type it yourself", W / 2, cy + 610, 40, "rgba(60,50,80,0.75)", 600); + ctx.restore(); +} + +function drawFrame(t) { + drawDesktop(ctx); + const dt = 1 / FPS; + + const hero = heroRect(t); + const hRect = drawStrip(ctx, rig, litAt(lead, t, 0.3), hero.x, hero.y, hero.w); + for (const n of onsetsBetween(t - dt, t)) { + particles.spawnNote(stripKeyX(rig, n.midi, hRect), hRect.y + hRect.h + 6, + stripKeyColor(rig, n.midi), true); + } + + // kicker under the strip β€” it has its say during the spoken frame, then + // clears the stage for the big teaching keycap + const ka = easeOut(clamp01((t - 0.9) / 0.6)) * + (1 - easeOut(clamp01((t - (SING0 - 0.5)) / 0.5))); + if (ka > 0) { + ctx.save(); ctx.globalAlpha = ka; + text(ctx, "your keyboard knows the C scale", W / 2, H * 0.245, 52, INK, 800); + ctx.restore(); + } + + drawKeycap(t); + drawChart(t); + drawEndCard(t); + + particles.stepAndDraw(dt); + vignette(ctx); + karaoke.draw(ctx, t); +} + +await renderVideo({ + canvas, audioPath: `${OUT}/${SLUG}${VAR}.mp3`, outPath: `${OUT}/base-${SLUG}${VAR}.mp4`, + total: TOTAL, drawFrame, label: `menuband scales sim${VAR}`, +}); +writeMeta(`${SLUG}${VAR}`, TOTAL, SCENES); diff --git a/pop/menuband/bin/sing-jingle.mjs b/pop/menuband/bin/sing-jingle.mjs index f5e36e666..da086b0b4 100644 --- a/pop/menuband/bin/sing-jingle.mjs +++ b/pop/menuband/bin/sing-jingle.mjs @@ -1,6 +1,30 @@ #!/usr/bin/env node // sing-jingle.mjs β€” jeffrey SINGS the Menu Band campaign jingles. // +// v6 β€” the REGISTER round + the SCALES teaching singalong: +// Β· REGISTER β€” `--register ` (default +12) lifts every line +// that many semitones ABOVE the engine's minimal-|shift| octave fit +// (jeffrey: "I could be higher octave?"). The formant envelope is never +// touched (no kermit by construction) and the goalpost conformance is +// register-aware (f0-linked bands widen/shift with the lift; duration/ +// energy/click gates unchanged). Per-line FALLBACK LADDER: if a line +// fails the WER gate at the asked register it re-renders at +7 then 0 +// and the most intelligible take wins (ties prefer the higher register); +// fallbacks are reported per line. +// Β· SCALES (menuband-scales) β€” a spoken frame around a sung two-octave +// notepat letter ladder (c d e f g a b h i j k l m n up, back down; +// h..n = notepat's second octave). Spoken lines are placed verbatim +// (natural TTS, unpitched, level-matched to the sung lines) with +// whisper word timings for captions; the ladder is explicit-mode with +// absolute pitches C3..B4 (register 0, octave_opt off β€” the ladder IS +// the register) and letter-name lyrics ("see dee ee …") for curated +// IPA. WER scoring letter-folds both sides so "C-D-E" == "see dee ee". +// Β· Engine fixes this round (spinging/lib/sing_line_world.py R6Β·2-4): +// final unstressed syllables stop starving ("diminished" β†’ "deman"), +// real ~100 ms phrase-boundary silence before phrase-initial fricatives +// ("keys. Sus" β†’ "kisses"), phrase-medial onset bursts sit prouder +// ("control" β†’ "Troll"). +// // v5 β€” the DICTION round: consonant time-stretching the way trained choirs // handle it (round 4's whisper gate was failing on swallowed consonants): // Β· STRETCHED CONSONANTS β€” the engine (spinging/lib/sing_line_world.py) @@ -102,6 +126,12 @@ const SR = 48_000; // Round 3 defaults to a ~0.875 lock β€” "drift more into the perfect harmony". const hIdx = process.argv.indexOf("--harmony"); const HARMONY = hIdx > 0 ? parseFloat(process.argv[hIdx + 1]) : 0.875; +// --register: semitones ABOVE the minimal-shift octave fit (v6 default +12 β€” +// "I could be higher octave?"). Lines that fail the WER gate up there fall +// back down the ladder (+7, then 0) and the best take wins. +const rIdx = process.argv.indexOf("--register"); +const REGISTER = rIdx > 0 ? parseInt(process.argv[rIdx + 1], 10) : 12; +const REGISTER_FALLBACKS = [7, 0]; const QA_PASSES = 3; // re-render budget per line (percentile gate) const CLARITY_PASSES = 2; // extra re-renders if the whisper WER gate fails const BRIDGE_MAX_S = 0.45; // keep in sync with sing_line_world.BRIDGE_MAX_S @@ -183,7 +213,28 @@ const LYRICS = { ["keys", [[18.8, 1.6, 48]]]] }, ], }, + // v6 β€” the SCALES teaching singalong. A spoken frame around the notepat + // two-octave letter ladder; timings + pitches come from the shared + // menuband-scales.score.json sidecar (render-jingles.mjs), so buildLines + // assembles the words there. `register: 0` β€” the ladder places its own + // absolute pitches (C3..B4, bright top, unstrained); octave_opt off. + "menuband-scales": { + mode: "scales", register: 0, + intro: "Here's how to type out the C scale.", + run: "C, D, E, F, G, A, B. Now sing it!", + asc: "C. D. E. F. G. A. B. H. I. J. K. L. M. N.", + desc: "M, L, K, J, I, H, B, A, G, F, E, D, C!", + outro: "Wanna type it yourself? Try Menu Band.", + }, +}; + +// notepat letter β†’ letter-NAME word (the lyric the ladder is sung on β€” +// real dictionary words so pronounce.mjs serves curated GenAm IPA). +const LETTER_WORDS = { + a: "ay", b: "bee", c: "see", d: "dee", e: "ee", f: "eff", g: "gee", + h: "aitch", i: "eye", j: "jay", k: "kay", l: "ell", m: "em", n: "en", }; +const LETTER_NAME_SET = new Set(Object.values(LETTER_WORDS)); // ── helpers ──────────────────────────────────────────────────────────────── // Percentile-gate feedback: nudge the engine's tweak knobs toward the @@ -357,6 +408,9 @@ const HOMOPHONES = { won: "one", banned: "band", cord: "chord", cords: "chords", suss: "sus", knew: "new", oar: "or", ore: "or", write: "right", rite: "right", }; +// casual contractions whisper expands ("wanna" β†’ "want to") β€” canonicalize +// BOTH sides to the expansion so the outro CTA scores verbatim +const CONTRACTIONS = { wanna: ["want", "to"], gonna: ["going", "to"], gotta: ["got", "to"] }; function normTokens(text) { const rough = String(text).toLowerCase() .replace(/[β€™β€˜]/g, "'") @@ -366,10 +420,25 @@ function normTokens(text) { const out = []; for (const w of rough) { if (/^\d+$/.test(w)) out.push(...numberToWords(parseInt(w, 10))); + else if (CONTRACTIONS[w]) out.push(...CONTRACTIONS[w]); else out.push(w.replace(/'/g, "")); } return out.map((w) => HOMOPHONES[w] || w); } + +// v6 letter folding (the scales ladder): "C-D-E" and "see dee ee" must score +// as the same thing. Single tokens a–n become their letter names; welded +// letter runs ("cde") split first. Only applied on lettered lines. +function letterFoldTokens(toks) { + const out = []; + for (const t of toks) { + if (/^[a-n]$/.test(t)) out.push(LETTER_WORDS[t]); + else if (/^[a-n]{2,}$/.test(t) && !LETTER_NAME_SET.has(t) && !HOMOPHONES[t]) + out.push(...t.split("").map((c) => LETTER_WORDS[c])); + else out.push(t); + } + return out; +} const tokEq = (a, b) => a === b || (a.length >= 3 && b.length >= 3 && editDist(a, b) <= 1); // merge hyp tokens that SPLIT one ref token ("full screen" vs "fullscreen", @@ -425,9 +494,11 @@ function werScore(refToks, hypToks) { } return m ? prev[n] / m : 0; } -function evalWER(refText, hypText) { - const ref = normTokens(refText); - const hyp = dewedge(ref, rewedge(ref, normTokens(hypText))); +function evalWER(refText, hypText, { lettered = false } = {}) { + let ref = normTokens(refText); + let hypRaw = normTokens(hypText); + if (lettered) { ref = letterFoldTokens(ref); hypRaw = letterFoldTokens(hypRaw); } + const hyp = dewedge(ref, rewedge(ref, hypRaw)); const wer = +werScore(ref, hyp).toFixed(3); const missing = ref.filter((w) => w.length >= 3 && !STOPWORDS.has(w) && !hyp.some((h) => tokEq(w, h))); @@ -504,6 +575,30 @@ function presplitHeard(scoreWords, heard) { return out; } +// v6 (scales): before alignment, fold whisper's heard letters onto the +// spelled letter-name score words β€” "C" β†’ "see", and welded runs ("CDE", +// or dashes normed away: "c-d-e" β†’ "cde") split char-proportionally into +// one window per letter. Only for lettered lines. +function letterizeHeard(heard) { + const out = []; + for (const h of heard) { + const t = norm(h.text); + if (/^[a-n]$/.test(t)) { out.push({ ...h, text: LETTER_WORDS[t] }); continue; } + if (/^[a-n]{2,}$/.test(t) && !LETTER_NAME_SET.has(t)) { + const span = h.toMs - h.fromMs; + let t0 = h.fromMs; + for (let c = 0; c < t.length; c++) { + const w = span / t.length; + out.push({ text: LETTER_WORDS[t[c]], fromMs: Math.round(t0), toMs: Math.round(t0 + w) }); + t0 += w; + } + continue; + } + out.push(h); + } + return out; +} + // Whisper stamps the words at a clip's hard end zero-width and buries their // audio inside the PREVIOUS window ("Type" 110-1020 actually holds "type to // play"). Clamp every window into the audio, then repair each RUN of @@ -651,6 +746,30 @@ function buildLines(slug) { }); lines.push({ tts: line.tts, words }); } + } else if (spec.mode === "scales") { + // spoken frame + the sung letter ladder, all timed by the shared + // score sidecar (render-jingles.mjs writes it; sim-scales reads it too) + const sc = JSON.parse(readFileSync(`${OUT}/${slug}.score.json`, "utf8")); + const asc = sc.ladder.filter((n) => n.dir === "up"); + const desc = sc.ladder.filter((n) => n.dir === "down"); + // notationText: the spelled letter names WITH commas β€” notation.mjs + // matches its tokens to the lyric words, so every letter becomes its own + // phrase and the engine ARTICULATES the ladder (tiny onset gaps, no + // melisma blur) instead of welding 14 letters into one glissando. + const sungLine = (tts, notes) => ({ + tts, lettered: true, + notationText: notes.map((n) => LETTER_WORDS[n.letter]).join(", ") + ".", + words: notes.map((n) => ({ + w: LETTER_WORDS[n.letter], display: n.letter, + slots: [{ t: n.t, dur: n.dur, midi: n.vocal }], + })), + }); + lines.push({ tts: spec.intro, spoken: true, t: sc.spoken.intro.t, words: [] }); + // lettered: the run's welded "C-D-E-F-G-A-B" caption splits into letters + lines.push({ tts: spec.run, spoken: true, lettered: true, t: sc.spoken.run.t, words: [] }); + lines.push(sungLine(spec.asc, asc)); + lines.push(sungLine(spec.desc, desc)); + lines.push({ tts: spec.outro, spoken: true, t: sc.spoken.outro.t, words: [] }); } else { for (const line of spec.lines) { lines.push({ @@ -659,13 +778,19 @@ function buildLines(slug) { }); } } - return { lines, durationSec: score.durationSec }; + return { lines, durationSec: score.durationSec, spec }; } // ── main ─────────────────────────────────────────────────────────────────── async function singOne(slug) { console.log(`\nβ–Έ ${slug} β€” jeffrey sings (line-continuous WORLD)`); - const { lines, durationSec } = buildLines(slug); + const { lines, durationSec, spec } = buildLines(slug); + // v6: register ladder β€” the asked lift first, then the kermit/strain + // fallbacks; a line stops falling as soon as it passes the WER gate. + const specRegister = spec.register ?? REGISTER; + const regLadder = [specRegister, ...REGISTER_FALLBACKS.filter((r) => r < specRegister)]; + const OCT_OPT = spec.mode !== "scales"; // the ladder places its own pitches + console.log(` register +${specRegister} (fallback ladder ${regLadder.join(" β†’ ")})`); const dir = `${OUT}/sung/${slug}`; mkdirSync(`${dir}/words`, { recursive: true }); if (!existsSync(WHISPER_MODEL)) throw new Error(`whisper model missing: ${WHISPER_MODEL}`); @@ -677,6 +802,8 @@ async function singOne(slug) { const qaLines = []; const lineSpans = []; // absolute spans for the per-line stem re-transcribe const consSpans = []; // v5: absolute consonant spans β†’ extra bed duck + const spokenPlacements = []; // v6: spoken lines placed after level match + const regFallbacks = []; // v6: per-line register fallbacks (reported) if (!existsSync(GOALPOSTS)) { throw new Error(`goalposts missing: ${GOALPOSTS} β€” build with spinging goalposts`); } @@ -691,6 +818,63 @@ async function singOne(slug) { const mp3 = `${dir}/line-${li}-${hash}.mp3`; await ttsLine(line.tts, mp3); + // ── v6 spoken lines (the scales frame): natural TTS placed verbatim at + // its absolute time β€” no engine, no pitch; whisper word timings become + // the caption windows and the transcript must be essentially verbatim ── + if (line.spoken) { + const w16s = mp3.replace(/\.mp3$/, "-16k.wav"); + if (!existsSync(w16s)) { + sh("ffmpeg", ["-y", "-v", "error", "-i", mp3, "-ac", "1", "-ar", "16000", w16s]); + } + const wjs = mp3.replace(/\.mp3$/, "-words"); + if (!existsSync(`${wjs}.json`)) { + sh("whisper-cli", ["-m", WHISPER_MODEL, "-f", w16s, "-ml", "1", "-oj", "-ojf", "-of", wjs], + { stdio: ["ignore", "ignore", "pipe"] }); + } + const heard = wordsFromWhisper(`${wjs}.json`); + const { audio: spokenAudio } = decodeAudioMono(mp3, SR); + spokenPlacements.push({ audio: spokenAudio, at: Math.floor(line.t * SR) }); + // caption words: strip trailing commas/periods; a lettered spoken line + // (the "C, D, E…" run) splits welded letter runs into single capitals + let capWords = heard; + if (line.lettered) { + capWords = []; + for (const h of heard) { + const tN = norm(h.text); + if (/^[a-n]{2,}$/.test(tN) && !LETTER_NAME_SET.has(tN)) { + const span = (h.toMs - h.fromMs) / tN.length; + let t0 = h.fromMs; + for (const c of tN) { + capWords.push({ text: c.toUpperCase(), fromMs: Math.round(t0), toMs: Math.round(t0 + span) }); + t0 += span; + } + } else if (/^[a-n]$/.test(tN)) capWords.push({ ...h, text: tN.toUpperCase() }); + else capWords.push(h); + } + } + for (const h of capWords) { + const text = h.text.replace(/[.,;:]+$/, ""); + if (!text) continue; + sungWords.push({ + text, fromMs: Math.round(line.t * 1000 + h.fromMs), + toMs: Math.round(line.t * 1000 + h.toMs), line: li, spoken: true, + }); + } + const tx = whisperTranscribe(mp3); + const wr = evalWER(line.tts, tx, { lettered: !!line.lettered }); + console.log(` line ${li} (spoken): "${line.tts}" β†’ heard "${tx}" Β· WER ${wr.wer} ${wr.pass ? "βœ“" : "βœ—"}`); + qaLines.push({ + line: li, text: line.tts, spoken: true, + whisper: { transcript: tx, wer: wr.wer, missing: wr.missing, pass: wr.pass }, + }); + lineSpans.push({ + li, text: line.tts, lettered: !!line.lettered, spoken: true, + t0: Math.max(0, line.t - 0.1), + t1: Math.min(durationSec, line.t + spokenAudio.length / SR + 0.15), + }); + continue; + } + // 16k mono for whisper (unpadded β€” trailing silence makes whisper smear // word timestamps into it; repairWindows handles the zero-width final // word it stamps at a hard file end), 48k mono for the WORLD engine @@ -710,8 +894,10 @@ async function singOne(slug) { const { audio: lineAudio } = decodeAudioMono(mp3, SR); const lineLen = lineAudio.length / SR; const mapWords = line.words.map((w) => w.w); + let heardRaw = expandDigitWords(wordsFromWhisper(`${wj}.json`)); + if (line.lettered) heardRaw = letterizeHeard(heardRaw); // v6 scales const heard = presplitHeard(mapWords, - rescaleHeard(expandDigitWords(wordsFromWhisper(`${wj}.json`)), lineAudio, lineLen * 1000)); + rescaleHeard(heardRaw, lineAudio, lineLen * 1000)); const windows = repairWindows(alignWords(mapWords, heard), lineLen * 1000, lineAudio); console.log(` line ${li}: "${line.tts}" Β· whisper heard "${heard.map((h) => h.text).join(" ")}"`); @@ -720,7 +906,7 @@ async function singOne(slug) { // punctuation + melody rests β‰₯ 0.4 s; the engine bridges legato inside a // phrase and breathes only at its edges). Phonemes from curated US IPA. const score = await buildLineScore({ - text: line.tts, + text: line.notationText ?? line.tts, // v6: scales phrases per letter words: line.words.map((w) => ({ w: w.w, slots: w.slots })), }); const scorePath = `${dir}/words/line-${li}-score.json`; @@ -763,7 +949,8 @@ async function singOne(slug) { slots, hardEnd: +tEnd.toFixed(4), phraseStart: phraseStartOf[wi], }); sungWords.push({ - text: word.w, fromMs: Math.round(tStart * 1000), toMs: Math.round(tEnd * 1000), line: li, + text: word.display ?? word.w, // v6: the scales karaoke shows LETTERS + fromMs: Math.round(tStart * 1000), toMs: Math.round(tEnd * 1000), line: li, }); } @@ -789,7 +976,7 @@ async function singOne(slug) { // the reference plateau-drift band on pass 1 const tweaks = { drift_scale: 1.6, glide_scale: 1, vib_depth_scale: 1, beta_scale: 1, air_scale: 1, cons_stretch_scale: 1 }; // v5: consonant diction stretch (engine caps at 2.5Γ—) - const renderPlan = () => { + const renderPlan = (register) => { const plan = { line_wav: w48, out_wav: outWav, @@ -801,7 +988,8 @@ async function singOne(slug) { f0_floor: 60, f0_ceil: 300, // jeffrey's real range β€” de-kermit // (the octave optimizer weighs words by voiced evidence instead of // narrowing this ceiling β€” the TTS exclamations genuinely peak high) - octave_opt: true, choir: true, + octave_opt: OCT_OPT, choir: true, + register, // v6: applied AFTER the octave fit tweaks, words: planWords, }; @@ -823,7 +1011,7 @@ async function singOne(slug) { let lastIsBest = false; const consider = (st) => { const tx = whisperTranscribe(outWav); - const wr = evalWER(line.tts, tx); + const wr = evalWER(line.tts, tx, { lettered: !!line.lettered }); const cf = confOf(st); const wins = best.stats === null || (wr.wer < best.res.wer && (cf || !best.conf)) @@ -838,35 +1026,49 @@ async function singOne(slug) { }; let stats = {}; let passes = 0; - for (let pass = 1; pass <= QA_PASSES; pass++) { - passes = pass; - stats = renderPlan(); + let clarityPasses = 0; + const registersTried = []; + // ── v6: register ladder β€” full QA + clarity budget per register; a line + // stops falling as soon as the whisper gate passes. consider() keeps the + // best take across ALL registers (ties prefer the earlier = higher one). + for (const reg of regLadder) { + registersTried.push(reg); + Object.assign(tweaks, { drift_scale: 1.6, glide_scale: 1, vib_depth_scale: 1, + beta_scale: 1, air_scale: 1, cons_stretch_scale: 1 }); + for (let pass = 1; pass <= QA_PASSES; pass++) { + passes++; + stats = renderPlan(reg); + if (stats.error) break; + const clean = stats.clicks && stats.clicks.clicks === 0 && stats.clicks.flux_spikes === 0; + if ((!stats.conformance || stats.conformance._pass) && clean) { consider(stats); break; } + consider(stats); + if (pass === QA_PASSES) break; + adjustTweaks(tweaks, stats.conformance); + console.log(` ↻ pass ${pass} (reg +${reg}): out of band β€” retweak ` + + Object.entries(tweaks).map(([k, v]) => `${k}=${v.toFixed(2)}`).join(" ")); + } if (stats.error) break; - const clean = stats.clicks && stats.clicks.clicks === 0 && stats.clicks.flux_spikes === 0; - if ((!stats.conformance || stats.conformance._pass) && clean) { consider(stats); break; } - consider(stats); - if (pass === QA_PASSES) break; - adjustTweaks(tweaks, stats.conformance); - console.log(` ↻ pass ${pass}: out of band β€” retweak ` + - Object.entries(tweaks).map(([k, v]) => `${k}=${v.toFixed(2)}`).join(" ")); + for (let cp = 1; cp <= CLARITY_PASSES && !best.res.pass; cp++) { + clarityPasses++; + tweaks.air_scale *= 0.6; + tweaks.vib_depth_scale *= 0.7; + // v5: more diction, not just less air β€” lean the stretch in harder + tweaks.cons_stretch_scale = Math.min(1.3, tweaks.cons_stretch_scale * 1.15); + console.log(` ↻ clarity pass ${cp} (reg +${reg}): WER ${best.res.wer} ` + + `(heard "${best.transcript}") β€” re-render with less air/vibrato, more stretch`); + const st = renderPlan(reg); + if (st.error) break; + consider(st); + } + if (best.res?.pass) break; + if (reg !== regLadder[regLadder.length - 1]) { + console.log(` ↧ register +${reg} missed the WER gate (${best.res?.wer}) β€” falling back`); + } } if (stats.error) { report.push({ slug, line: li, word: "(line)", note: stats.error }); continue; } - let clarityPasses = 0; - for (let cp = 1; cp <= CLARITY_PASSES && !best.res.pass; cp++) { - clarityPasses = cp; - tweaks.air_scale *= 0.6; - tweaks.vib_depth_scale *= 0.7; - // v5: more diction, not just less air β€” lean the stretch in harder - tweaks.cons_stretch_scale = Math.min(1.3, tweaks.cons_stretch_scale * 1.15); - console.log(` ↻ clarity pass ${cp}: WER ${best.res.wer} ` + - `(heard "${best.transcript}") β€” re-render with less air/vibrato, more stretch`); - const st = renderPlan(); - if (st.error) break; - consider(st); - } if (!lastIsBest) { // the final re-render didn't win β€” restore the best-scoring take copyFileSync(bestWav, outWav); @@ -875,10 +1077,18 @@ async function singOne(slug) { let werRes = best.res; let transcript = best.transcript; stats = best.stats; + // v6: which register actually won β€” a drop below the asked one is a + // reported fallback (kermit/strain safety valve) + const finalReg = best.stats?.register ?? regLadder[0]; + if (finalReg !== specRegister) { + regFallbacks.push({ slug, line: li, text: line.tts, + asked: specRegister, used: finalReg, wer: best.res.wer }); + console.log(` ‡ line ${li} register fallback: +${specRegister} β†’ +${finalReg}`); + } // v5 diagnostic: transcribe the choir-less LEAD stem too β€” separates // diction gains from choir masking in the QA sidecar const leadTranscript = whisperTranscribe(leadWav); - const leadWer = evalWER(line.tts, leadTranscript); + const leadWer = evalWER(line.tts, leadTranscript, { lettered: !!line.lettered }); // v5: absolute consonant spans drive the bed's extra diction duck for (const [a, b] of stats.consonant_spans || []) { consSpans.push([lineT0 + a, lineT0 + b]); @@ -914,6 +1124,8 @@ async function singOne(slug) { console.log(` lead-only diagnostic: WER ${leadWer.wer} β€” heard "${leadTranscript}"`); qaLines.push({ line: li, text: line.tts, passes, clarityPasses, tweaks, + register: finalReg, registersTried, + registerFallback: finalReg !== specRegister, lineTranspose: stats.line_transpose, beta: stats.beta, harmony: HARMONY, consStretchScale: stats.cons_stretch_scale, f0JumpMaxCents: stats.f0_jump_max_cents, f0JumpP95Cents: stats.f0_jump_p95_cents, @@ -925,7 +1137,7 @@ async function singOne(slug) { conformance: stats.conformance, clicks: stats.clicks, }); lineSpans.push({ - li, text: line.tts, + li, text: line.tts, lettered: !!line.lettered, t0: Math.max(0, planWords[0].slots[0].t - 0.15), t1: Math.min(durationSec, planWords[planWords.length - 1].hardEnd + 0.3), }); @@ -937,6 +1149,28 @@ async function singOne(slug) { for (let i = 0; i < sung.length && at + i < master.length; i++) master[at + i] += sung[i]; } + // v6: place the spoken lines now, level-matched to the sung material β€” + // active-frame RMS of the assembled sung vocal vs each spoken take, spoken + // sitting just under the singing so the frame never shouts over the song + if (spokenPlacements.length) { + const rmsActive = (buf) => { + let e = 0, n = 0; + for (let i = 0; i < buf.length; i++) { + const v = buf[i]; + if (Math.abs(v) > 1e-3) { e += v * v; n++; } + } + return n ? Math.sqrt(e / n) : 0; + }; + const sungRms = rmsActive(master); + for (const p of spokenPlacements) { + const own = rmsActive(p.audio); + const g = sungRms > 0 && own > 0 ? Math.min(1.6, (sungRms * 0.9) / own) : 1; + for (let i = 0; i < p.audio.length && p.at + i < master.length; i++) { + master[p.at + i] += p.audio[i] * g; + } + } + } + // normalize the vocal, write it, then MASTER the mix let peak = 0; for (let i = 0; i < master.length; i++) peak = Math.max(peak, Math.abs(master[i])); @@ -1020,7 +1254,7 @@ async function singOne(slug) { console.log(` whisper end-to-end on the vocal stem …`); const stemLines = lineSpans.map((s) => { const tx = whisperTranscribeSpan(vocalWav, s.t0, s.t1, `${dir}/words/stem-line-${s.li}`); - const w = evalWER(s.text, tx); + const w = evalWER(s.text, tx, { lettered: !!s.lettered }); const mark = w.pass ? "βœ“" : "βœ—"; console.log(` ${mark} L${s.li} WER ${w.wer} "${s.text}" β†’ heard "${tx}"`); return { line: s.li, text: s.text, transcript: tx, wer: w.wer, missing: w.missing, pass: w.pass }; @@ -1030,8 +1264,9 @@ async function singOne(slug) { writeFileSync(`${OUT}/${slug}.words.sung.json`, JSON.stringify(sungWords, null, 2)); writeFileSync(`${OUT}/${slug}-sung-qa.json`, JSON.stringify({ - slug, harmony: HARMONY, engine: "spinging/lib/sing_line_world.py (round 5)", + slug, harmony: HARMONY, engine: "spinging/lib/sing_line_world.py (round 6)", goalposts: GOALPOSTS, + register: { asked: specRegister, ladder: regLadder, fallbacks: regFallbacks }, gates: { werMax: WER_GATE, voicingContinuityMin: CONTINUITY_GATE }, consDuckSpans: consSpans.length, pronunciationSources: { ...sourceCounts }, @@ -1044,6 +1279,13 @@ async function singOne(slug) { console.log(`βœ“ ${mix}`); console.log(`βœ“ ${OUT}/${slug}.words.sung.json Β· ${sungWords.length} words`); console.log(`βœ“ ${OUT}/${slug}-sung-qa.json`); + for (const f of regFallbacks) { + report.push({ slug, line: f.line, word: "(register)", + info: `register fallback +${f.asked} β†’ +${f.used} (best WER ${f.wer}) β€” "${f.text}"` }); + } + if (!regFallbacks.length && specRegister !== 0) { + console.log(` register: every line held +${specRegister}`); + } return report; } diff --git a/spinging/README.md b/spinging/README.md index 18a2730e4..e2203993c 100644 --- a/spinging/README.md +++ b/spinging/README.md @@ -82,6 +82,44 @@ re-transcribed per line and the final mix once more β€” transcripts land verbatim in `out/-sung-qa.json`. QA is statistical AND machine-read, not ear-only. +Changelog β€” round 6 (2026-07, the REGISTER round + the SCALES singalong β€” +"I could be higher octave?"): +- **Register lift** β€” `plan.register` (semitones) applies AFTER the per-line + minimal-|shift| octave fit; sing-jingle's `--register` (default +12 this + round) drives it with a per-line fallback ladder (+12 β†’ +7 β†’ 0): a line + that misses the whisper gate re-renders lower and the most intelligible + take wins (ties prefer the higher register); fallbacks land in the QA + sidecar (`register.fallbacks`). Conformance is register-aware + (`vocal_shapes.conformance(register=…)`): the f0-linked bands (glide/ + drift/release/vib cents + hf_ratio) widen 35 %/octave and hf_ratio shifts + with the harmonic comb (Γ—2^(R/24)); duration/energy/click gates unchanged. +- **R6Β·2 final unstressed syllables stop starving** ("diminished" β†’ "deman"): + word-final unstressed vowels get a minimum-duration floor (0.14 s, borrowed + by anticipating the note into the preceding stressed vowel) and word-final + coda CLUSTERS (β‰₯2 phones) may articulate into the phrase gap (extension cap + 0.05 β†’ 0.18 s). Chords L4 now closes the word ("demand" β€” the coda exists; + the full /Κƒt/ remains whisper-hostile). +- **R6Β·3 phrase-boundary silence** ("keys. Sus" β†’ "kisses"): phrase-initial + FRICATIVE onsets get a real ~100 ms near-zero gap (verified in stats: + `gap_ms 100`, onset 115 ms) instead of the 22 ms glottal dip. +- **R6Β·4 phrase-medial onset prominence** ("control" β†’ "Troll"): raw plosive + composites in a phrase-medial word's onset ride an extra 1.25Γ— on top of + RAW_BOOST. +- **The scales reel** (pop/menuband/bin/sing-jingle.mjs `menuband-scales`) β€” + spoken lines placed verbatim at absolute times (level-matched to the sung + material, whisper word timings β†’ captions; all three spoken lines + transcribe at WER 0) around the sung notepat letter ladder: explicit + pitches C3..B4, `octave_opt` off, per-letter phrases via a notation-only + text (letter names + commas) so the ladder articulates instead of + smearing into one glissando; letter-name lyrics ("see dee ee …", curated + /eΙͺ/ "ay" + /aΙͺ/ "eye" in pronounce.mjs β€” CURATED wins over cache) and + letter-folded WER scoring ("C-D-E" == "see dee ee"). +- Result: every line CAN now ride an octave above round 5 (13/19 sung + campaign lines hold +12, 3 land +7, 3 fall to +0 β€” all reported); scales + descending ladder hits WER 0.154 (11/13 letters) with the ascending run + flagged honestly (whisper hears sustained sung letter-vowels as melody); + clicks 0 and conformance green on every shipped line. + Changelog β€” round 5 (2026-07, the DICTION round β€” consonant time-stretching the way trained choirs handle it; round 4's whisper gate was failing on swallowed consonants, not on the singing): diff --git a/spinging/cache/pronounce.json b/spinging/cache/pronounce.json index 784f00ff0..66d915300 100644 --- a/spinging/cache/pronounce.json +++ b/spinging/cache/pronounce.json @@ -383,5 +383,103 @@ "accent": "US", "v": 4, "fetchedAt": "2026-07-19T16:14:52.422Z" + }, + "see": { + "ipa": "/ˈsi/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.007Z" + }, + "dee": { + "ipa": "/ˈdiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.170Z" + }, + "ee": { + "ipa": "/ˈiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.438Z" + }, + "eff": { + "ipa": "/Ι›f/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:26.894Z" + }, + "gee": { + "ipa": "/dΝ‘Κ’iː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:27.120Z" + }, + "ay": { + "ipa": "/aΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:27.344Z" + }, + "bee": { + "ipa": "/ˈbiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.055Z" + }, + "aitch": { + "ipa": "/ˈeΙͺtΝ‘Κƒ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.250Z" + }, + "eye": { + "ipa": "/ˈaː/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.463Z" + }, + "jay": { + "ipa": "/ˈdΝ‘Κ’eΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.644Z" + }, + "kay": { + "ipa": "/keΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.806Z" + }, + "ell": { + "ipa": "/ΛˆΙ›l/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:35.346Z" + }, + "em": { + "ipa": "/ΛˆΙ›m/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:35.585Z" + }, + "en": { + "ipa": "/ΛˆΙ›n/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:36.065Z" } } \ No newline at end of file diff --git a/spinging/lib/pronounce.mjs b/spinging/lib/pronounce.mjs index c4133454e..afc4f82f3 100644 --- a/spinging/lib/pronounce.mjs +++ b/spinging/lib/pronounce.mjs @@ -264,6 +264,15 @@ const BUILTIN = { a: "/Ι™/", the: "/Γ°Ι™/", }; +// Round 6 β€” words where the dictionary's PRIMARY entry is the wrong +// homograph for singing (the scales letter ladder): "ay" resolves to the +// aye-vote /aΙͺ/ where the letter A is /eΙͺ/, and Wiktionary's "eye" line +// parses to /ˈaː/. CURATED wins over every source, the cache included. +const CURATED = { + ay: "/eΙͺ/", // the letter A + eye: "/aΙͺ/", // the letter I +}; + // ── cache + public API ───────────────────────────────────────────────────── function loadCache() { try { return JSON.parse(readFileSync(CACHE_FILE, "utf8")); } catch { return {}; } @@ -273,7 +282,7 @@ function saveCache(c) { writeFileSync(CACHE_FILE, JSON.stringify(c, null, 1)); } -export const sourceCounts = { wiktionary: 0, espeak: 0, builtin: 0, cache: 0 }; +export const sourceCounts = { wiktionary: 0, espeak: 0, builtin: 0, cache: 0, curated: 0 }; // Cache format version β€” bump to invalidate every cached word (round 4: the // per-line accent read + usSafe screen obsoleted every round-3 entry). @@ -289,6 +298,11 @@ async function resolveWord(word) { export async function pronounce(rawWord) { const word = rawWord.toLowerCase().replace(/[^a-z']/g, ""); + if (CURATED[word]) { + sourceCounts.curated++; + return { word, ipa: CURATED[word], source: "curated", + syllables: syllabify(tokenizeIPA(CURATED[word])) }; + } const cache = loadCache(); let entry = cache[word]; if (entry && entry.v === CACHE_V) sourceCounts.cache++; diff --git a/spinging/lib/sing_line_world.py b/spinging/lib/sing_line_world.py index 9ea966869..df1b08efb 100644 --- a/spinging/lib/sing_line_world.py +++ b/spinging/lib/sing_line_world.py @@ -1,8 +1,34 @@ #!/usr/bin/env python3 """ -sing_line_world.py β€” spinging's line-continuous singing engine (round 5). - -What round 5 adds β€” CONSONANT TIME-STRETCHING, the way trained choirs +sing_line_world.py β€” spinging's line-continuous singing engine (round 6). + +What round 6 adds β€” REGISTER + the diction starve fixes: + + R6Β·1 REGISTER β€” plan.register (semitones) lifts every target AFTER the + per-line minimal-|shift| octave fit: the fit still finds where the + spoken take naturally sits, then the whole line rides up (jeffrey: + "I could be higher octave?"). The formant envelope is untouched (no + kermit by construction) and the goalpost conformance becomes + register-aware (vocal_shapes.conformance widens/shifts the f0-linked + bands; duration/energy/click gates unchanged). The caller may retry + a line at lower registers if it sounds strained (fallback ladder). + R6Β·2 FINAL UNSTRESSED SYLLABLES STOP STARVING ("diminished" β†’ "deman"): + a word-final unstressed syllable's vowel gets a minimum-duration + floor (borrowed from the preceding stressed vowel by anticipating + the note), and word-final coda CLUSTERS (β‰₯2 phones) may articulate + into the phrase gap (extension cap 0.05 β†’ 0.18 s) so /Κƒt/-style + codas render at full value instead of vanishing. + R6Β·3 PHRASE-BOUNDARY SILENCE ("keys. Sus" β†’ "kisses"): a phrase-initial + fricative onset now gets a real ~100 ms silence carved before it + (near-zero floor, not the 22 ms glottal dip) so the /s/ can't weld + backward onto the previous phrase's coda. + R6Β·4 PHRASE-MEDIAL ONSET PROMINENCE ("control" β†’ "Troll"): raw plosive + composites inside a phrase-medial word's onset get an extra boost on + top of RAW_BOOST β€” the /k/ burst stays legible against the already- + leveled vowels around it. (Consonant-span bed ducking already covers + these frames via cons_mask.) + +Round 5 (kept) β€” CONSONANT TIME-STRETCHING, the way trained choirs handle diction (the whisper round-trip gate was failing on swallowed consonants, not on the singing): @@ -136,9 +162,14 @@ MAX_ONSET_OUT_S = 0.34 # stretched onset ceiling MAX_CODA_OUT_S = 0.42 # stretched coda ceiling PLOSIVE_GAP_S = 0.022 # pre-plosive silence (the choir's glottal set-up) GAP_FLOOR_AMP = 0.06 # the gap dips to this, never digital zero +PHRASE_FRIC_GAP_S = 0.10 # R6Β·3 real silence before a phrase-initial fricative +PHRASE_FRIC_GAP_AMP = 0.02 # …and it dips near-zero (a true phrase breath) CONS_GAIN_DB = 3.0 # consonant prominence on the WORLD path RAW_BOOST = 1.5 # raw composite boost (plosive bursts; R4 was 1.4) +ONSET_RAW_EXTRA = 1.25 # R6Β·4 extra boost on phrase-medial onset bursts CODA_EXTEND_S = 0.05 # word-final codas may run past hardEnd at phrase ends +CODA_CLUSTER_EXTEND_S = 0.18 # R6Β·2 …coda CLUSTERS may run this far into the gap +FINAL_UNSTRESSED_VOWEL_S = 0.14 # R6Β·2 word-final unstressed vowel floor MAX_BREATH_S = 0.45 BRIDGE_MAX_S = 0.45 # intra-phrase gaps up to this sustain legato BRIDGE_DIP_AMP = 0.72 # shallow energy dip at a bridged word boundary @@ -769,6 +800,13 @@ def main(): for s in segs: for sl in s["w"]["slots"]: sl["midi"] += line_transpose + # R6Β·1: the register lift rides ON TOP of the minimal-shift fit β€” the fit + # finds where the spoken take naturally sits, the register is the ask. + register = int(plan.get("register", 0)) + if register: + for s in segs: + for sl in s["w"]["slots"]: + sl["midi"] += register # ── output timeline ──────────────────────────────────────────────────── line_t0 = float(plan["line_t0"]) @@ -791,6 +829,7 @@ def main(): force_voiced = np.zeros(out_n, dtype=bool) # R4Β·2 sung voiced consonants sp_gain = np.ones(out_n) # R4Β·1 shallow bridge energy dips cons_gain = np.ones(out_n) # R5Β·4 consonant prominence (sp amp) + raw_gain = np.ones(out_n) # R6Β·4 extra boost on onset raw bursts cons_mask = np.zeros(out_n, dtype=bool) # R5 consonant frames (choir gate) gap_env = np.ones(out_n) # R5Β·2 pre-plosive glottal-set-up dips onset_marks = [] # (onset_f, vowel_f) voiced-onset QA @@ -817,13 +856,15 @@ def main(): cons_g = db(CONS_GAIN_DB) - def place_plan(o_end, plan, midi, med): + def place_plan(o_end, plan, midi, med, raw_extra=1.0): """R5Β·1: place a diction plan so its output ENDS at frame o_end. Raw runs map 1:1 (integer source steps β†’ the raw composite path picks them up with true transients); stretched runs map fractionally and render through WORLD (noise for unvoiced fricatives, the pitch path - for voiced sonorants). Returns the plan's output start frame.""" + for voiced sonorants). raw_extra (R6Β·4) rides on RAW_BOOST for the + raw runs β€” phrase-medial onset bursts sit prouder. Returns the + plan's output start frame.""" o = o_end - sum(p[2] for p in plan) o_start = o for ra, rb, out, m in plan: @@ -834,6 +875,7 @@ def main(): if m != "pitch" and out == rb - ra: src_pos[idx] = ra + (idx - o) # 1:1 β†’ raw composite natural[idx] = True + raw_gain[idx] = raw_extra else: pos = (np.linspace(ra, max(ra, rb - 1) + 1e-6, out) if out > 1 else np.array([float(ra)])) @@ -869,10 +911,17 @@ def main(): exp_on = s["exp"][0]["phonemes"]["onset"] if s["exp"] else None # the glottal set-up: before every plosive/affricate, and at every # phrase-initial consonant (a stretched phrase-opening /s/ otherwise - # welds onto the previous phrase's coda β€” "keys. Sus" β†’ "kisses") - gap_s = PLOSIVE_GAP_S if (exp_on and ( - exp_on[0]["cls"] in ("plosive", "affricate") - or (wi > 0 and s["w"].get("phraseStart")))) else 0.0 + # welds onto the previous phrase's coda β€” "keys. Sus" β†’ "kisses"). + # R6Β·3: a 22 ms dip wasn't enough for phrase-initial FRICATIVES β€” + # whisper still heard the /z s/ weld β€” so those now get a real + # ~100 ms near-silent phrase breath instead. + gap_s, gap_amp = 0.0, GAP_FLOOR_AMP + if exp_on: + phrase_medial_start = wi > 0 and s["w"].get("phraseStart") + if phrase_medial_start and exp_on[0]["cls"] == "fricative": + gap_s, gap_amp = PHRASE_FRIC_GAP_S, PHRASE_FRIC_GAP_AMP + elif exp_on[0]["cls"] in ("plosive", "affricate") or phrase_medial_start: + gap_s = PLOSIVE_GAP_S oplan, out_f = diction_plan( s["onset"][0], s["onset"][1], exp_on, cls, voiced, hf_ratio, stretch_scale, min(MAX_ONSET_OUT_S, max(0.04, avail - gap_s)), @@ -880,7 +929,7 @@ def main(): onset_plans.append({"plan": oplan, "out_f": out_f, "out_s": out_f * FRAME_S, "gap_f": int(round(gap_s / FRAME_S)), - "gap_s": gap_s}) + "gap_s": gap_s, "gap_amp": gap_amp}) stats_words = [] for wi, s in enumerate(segs): @@ -933,13 +982,15 @@ def main(): s["_v_delay"] = v_delay onset_start = o_end_on if op["plan"]: - onset_start = place_plan(o_end_on, op["plan"], slots[0]["midi"], med_log) + onset_start = place_plan(o_end_on, op["plan"], slots[0]["midi"], med_log, + raw_extra=ONSET_RAW_EXTRA if wi > 0 else 1.0) if s["onset_voiced"] and o_end_on > onset_start: onset_marks.append((max(0, onset_start), min(out_n, o_end_on))) if op["gap_f"] > 0: ga = max(0, onset_start - op["gap_f"]) if onset_start > ga: - gap_env[ga:onset_start] = GAP_FLOOR_AMP + gap_env[ga:onset_start] = np.minimum( + gap_env[ga:onset_start], op["gap_amp"]) s["_onset_start"] = onset_start # R5Β·3: coda diction plan β€” stretched to full value, stealing time @@ -947,14 +998,19 @@ def main(): # to CODA_EXTEND_S past hardEnd (bridged codas already ride the # bridge into the next word's onset). nxt_seg = segs[wi + 1] if wi + 1 < len(segs) else None + exp_coda_ph = s["exp"][-1]["phonemes"]["coda"] if s["exp"] else None + # R6Β·2: a coda CLUSTER (β‰₯2 phones β€” "diminished"'s /Κƒt/) needs real + # articulation room; let it run further into the phrase gap than a + # single-consonant coda would. + ext_cap = CODA_CLUSTER_EXTEND_S if exp_coda_ph and len(exp_coda_ph) >= 2 \ + else CODA_EXTEND_S ext_s = 0.0 if nxt_seg is None: - ext_s = CODA_EXTEND_S + ext_s = ext_cap elif nxt_seg["w"].get("phraseStart") and bridge_from is None: nxt_onset_t = nxt_seg["w"]["slots"][0]["t"] - \ (onset_plans[wi + 1]["out_s"] + onset_plans[wi + 1]["gap_s"]) - ext_s = max(0.0, min(CODA_EXTEND_S, nxt_onset_t - hard_end - 0.06)) - exp_coda_ph = s["exp"][-1]["phonemes"]["coda"] if s["exp"] else None + ext_s = max(0.0, min(ext_cap, nxt_onset_t - hard_end - 0.06)) coda_avail = (hard_end + ext_s) - (slots[-1]["t"] + 0.06) cp_plan, cp_out_f = diction_plan( s["coda"][0], s["coda"][1], exp_coda_ph, cls, voiced, hf_ratio, @@ -989,6 +1045,16 @@ def main(): else: v_end = hard_end - coda_len v_end = max(v_end, v_start + 0.03) + # R6Β·2: a word-final UNSTRESSED syllable must not starve β€” its + # vowel gets a minimum duration, borrowed by anticipating the + # note into the preceding (stressed) vowel's tail ("diminished"'s + # final -nished was 3 output frames + no coda room β†’ "deman"). + if k == n_slots - 1 and n_slots >= 2: + exp_k = s["exp"][k] if k < len(s["exp"]) else None + if exp_k is not None and not exp_k.get("stress") \ + and v_end - v_start < FINAL_UNSTRESSED_VOWEL_S: + v_start = max(slots[k - 1]["t"] + 0.10, + v_end - FINAL_UNSTRESSED_VOWEL_S) o_a, o_b = of(v_start), of(v_end) o_a = max(0, o_a) o_b = min(out_n, max(o_b, o_a + 2)) @@ -1297,7 +1363,8 @@ def main(): # R4/R5: articulate β€” the vowel leveling (arc conformance) # lifts sung vowels well above the raw spoken consonants; # boost the composite so plosive bursts stay legible. - seg = x[sa:sb] * RAW_BOOST + # R6Β·4: raw_gain adds onset-burst prominence on top. + seg = x[sa:sb] * RAW_BOOST * float(raw_gain[i]) L = ob - oa r = min(ramp, L // 2) if r > 1: @@ -1444,7 +1511,7 @@ def main(): notes_r = [(a, b) for a, b in (trimmed or notes_r) if a >= out_n or cons_mask[a:min(b, out_n)].mean() < 0.5] feats_r = [note_features(f0r, rms_r, hf_r, a, b) for a, b in notes_r] - conf = conformance(feats_r, bands) + conf = conformance(feats_r, bands, register=register) conf["_notes_measured"] = len(feats_r) clicks = click_scan(y.astype(np.float64), fs) # exonerate flux spikes that land on NATURAL consonant composites β€” those @@ -1481,6 +1548,7 @@ def main(): print(json.dumps({ "words": stats_words, "line_transpose": line_transpose, + "register": register, "beta": round(beta, 4), "harmony": harmony, "cons_stretch_scale": stretch_scale, "consonant_spans": cons_spans, diff --git a/spinging/lib/vocal_shapes.py b/spinging/lib/vocal_shapes.py index 3faa1034b..22741617a 100644 --- a/spinging/lib/vocal_shapes.py +++ b/spinging/lib/vocal_shapes.py @@ -161,18 +161,42 @@ def percentile_bands(all_feats, ps=(5, 10, 25, 50, 75, 90, 95)): return bands -def conformance(feats_list, bands, lo="p10", hi="p90"): - """Median-of-notes per feature vs reference band β†’ {feature: {…, pass}}.""" +# Round 6: bands whose reference values ride the SPEAKER'S F0 β€” the goalposts +# were measured on low-baritone acapellas, so a register lift (singing the +# same voice N semitones up) legitimately moves them: pitch-tracking noise +# scales with f0 (drift/glide/release cents), and the harmonic comb shifts +# energy above the 4 kHz line (hf_ratio). Duration / energy / click gates are +# register-independent and stay untouched. +F0_LINKED = ("onset_glide_cents", "onset_glide_ms", "plateau_drift_cents", + "release_cents", "vib_depth_cents", "hf_ratio") + + +def conformance(feats_list, bands, lo="p10", hi="p90", register=0): + """Median-of-notes per feature vs reference band β†’ {feature: {…, pass}}. + + register (semitones above the calibrated speaker) makes the f0-linked + bands register-aware: cents/ms bands widen by 35 % per octave of lift, + hf_ratio's band shifts up with the harmonic comb (Γ—2^(R/24)) and relaxes. + """ rep = {} ok_all = True + widen = 1.0 + 0.35 * abs(register) / 12.0 + hf_shift = 2.0 ** (register / 24.0) for k, band in bands.items(): vals = np.array([f[k] for f in feats_list if k in f]) if len(vals) == 0: rep[k] = {"value": None, "pass": None} # n/a this line continue v = float(np.median(vals)) - ok = bool(band[lo] <= v <= band[hi]) - rep[k] = {"value": round(v, 3), "lo": band[lo], "hi": band[hi], "pass": ok} + b_lo, b_hi = band[lo], band[hi] + if register and k in F0_LINKED: + if k == "hf_ratio": + b_lo, b_hi = b_lo * hf_shift / widen, b_hi * hf_shift * widen + else: + b_lo, b_hi = b_lo / widen, b_hi * widen + b_lo, b_hi = round(b_lo, 3), round(b_hi, 3) + ok = bool(b_lo <= v <= b_hi) + rep[k] = {"value": round(v, 3), "lo": b_lo, "hi": b_hi, "pass": ok} # duration + energy arcs are score/arrangement-driven (and vib delay's # estimator is weak on the references) β€” advisory only if k not in ("dur_s", "energy_release_ms", "vib_rate_hz", "vib_delay_ms") and not ok: -- 2.51.2