diff --git a/docs/video-scrub-rubric.md b/docs/video-scrub-rubric.md new file mode 100644 index 000000000..65d140e7d --- /dev/null +++ b/docs/video-scrub-rubric.md @@ -0,0 +1,89 @@ +# Video Scrub Rubric β€” synthtapes + the scrub instrument + +The tape player (`disks/video.mjs` + tape machinery in `bios.mjs`) is an +instrument: rate is the controlled quantity, position follows. This rubric +defines what "working" means and how it is checked. + +## Test surfaces + +- **`video scrub auto`** β€” the in-piece autopilot: scripted gesture segments + driven through the *same state the finger drives*, graded against + wall-clock ground truth. All output is console-greppable by `πŸ§ͺ`. + Variants compose: `video scrub break auto`, `video scrub 2 auto` (short + tape β†’ seam-crossing stress), etc. +- **Synthtapes** (`tape:play-synth` in bios) β€” deterministic tapes whose + frames render the soundtrack's own waveform on a scrolling ruler, so + scrub velocity, direction, and AV alignment are visible by eye. + Styles: `bed` (melodic), `break` (16th breakbeat), `house`, `dub`, + `sine` (legato sineline). All pitched voices are quantized to whole + cycles per loop β†’ the seam is phase-continuous by construction. +- **Headless driving** β€” wheel gestures via `dispatchEvent(new WheelEvent(...))`, + taps via CDP clicks; `window.__speaker_telemetry.runningCount` asserts the + audio source stayed alive. + +## The rubric (autopilot segments Γ— criteria) + +| Segment | Drives | Pass criteria | +|---|---|---| +| roll | nothing (normal play) | position advances | +| fast forward | rate 3Γ— held 90 ticks | effective rate within Β±25% of 3Γ— (motion-integral, wrap-proof) | +| release into inertia | release at speed | inertia decays and converges | +| reverse | rate βˆ’2Γ— held | effective βˆ’2Γ— Β±25%, seam wrap keeps direction | +| slow crawl | rate 0.5Γ— | effective 0.5Γ— Β±25% (slow-rate accuracy) | +| tap dip | single-tap dip | rate sags below 0.3Γ—, returns ≀1.05Γ—, negligible net motion | +| scratch | Β±3.2Γ— @ 2Hz rocking | both directions exceeded, net drift < 0.7 tape-seconds | +| fast scratch | Β±2.2Γ— @ 4Hz | same, amplitude-scaled | +| touch brake | hold from 1Γ— | rate sags to < 0.05Γ— | +| brake release spin-up | lift after brake | ramps back to pre-brake rate | + +Every scrub-driven segment also records: physics tick count, integrated +wall time (must β‰ˆ segment wall time), speed min/max, and the tape duration +used β€” so a failure names its layer. + +## Invariants (the things that regress) + +1. **Wall-clock honesty** β€” commanded rate means tape-seconds per + wall-second at any sim tick rate (measured `simDt`, burst-tolerant). +2. **One audio source** β€” tape audio is a single `loop: true` sample; the + worklet wraps the read in both directions (stample ring). Nothing + kills/restarts it during interaction; `runningCount` stays 1 through + seam crossings both ways. +3. **Absolute rate** β€” `tape:audio-rate` sets the worklet speed absolutely; + no relative-shift accumulation, no estimate drift. +4. **No handoffs** β€” every landing (park, wheel, dip, brake release, + friction) keeps the scrub drive running; driving at exactly 1.0Γ— *is* + playback, so there is never a restart jump near 1Γ—. +5. **Loop parity** β€” video duration = audio duration exactly + (`mediaRecorderDuration`, includes the last frame's display time), so + video and audio wrap at the same point at any rate. +6. **Scrub owns frames** β€” while scrubbing, the bios RAF clock must not + advance frames at 1Γ— between seeks. +7. **Musical loop grid** β€” at rest, loop periods converge to β‰ˆΒ±2ms of the + tape length (grid-anchored, logged as `πŸ•°οΈ`). + +## Gesture map under test + +tap = dip Β· hold = brake Β· grab+drag = displacement-is-rate scratch +(with an on-screen anchorβ†’finger vector showing direction and energy) Β· +flick = prize-wheel spin-down to pre-flick rate Β· two-finger scroll = +shuttle (Β±24Γ—) Β· top-right readout drag = steady-rate dial (friction-free +hold) Β· ←/β†’ = one-beat jump (audio relocated via `tape:audio-pos`) Β· +hold ↑ = quarter-beat chop repeat, hold ↓ = eighth-beat glitch chop Β· +release = seamless park + bearing friction home to 1Γ— Β· +spacebar = full reset/pause. A keys legend renders bottom-right. + +## Net-time unison + +Loop boundaries anchor to UTC multiples of the tape length (bios), and the +at-rest drive continuously lerps phase toward the AC network clock +(`clock.time()`, synced via `/api/clock`) with a Β±5% tempo lean β€” so every +player of the same tape converges into global unison without jumps. The +top-right `sync Β±Nms` readout is green when locked (<60ms). + +## Open items + +- Independent pitch-shift while playing (granular in the speaker worklet β€” + `targetDuration`/grain machinery exists in `lib/sound/synth.mjs`, needs a + live pitch param + a gesture axis). +- Seam-scratch segment (scratching across the loop point) in the autopilot. +- UTC/`net.time` phase alignment of the loop grid across devices. diff --git a/system/public/aesthetic.computer/bags.json b/system/public/aesthetic.computer/bags.json index 3baba9ee0..ceaf15772 100644 --- a/system/public/aesthetic.computer/bags.json +++ b/system/public/aesthetic.computer/bags.json @@ -26,6 +26,11 @@ "code": "video~scrub~dub", "name": "dub" }, + { + "type": "piece", + "code": "video~scrub~sine", + "name": "sineline" + }, { "type": "piece", "code": "video~scrub~break~2", diff --git a/system/public/aesthetic.computer/bios.mjs b/system/public/aesthetic.computer/bios.mjs index 0dee16701..90881d889 100644 --- a/system/public/aesthetic.computer/bios.mjs +++ b/system/public/aesthetic.computer/bios.mjs @@ -11546,12 +11546,12 @@ async function boot(parsed, bpm = 60, resolution, debug) { const bassRoots = [82.5, 65.5, 98, 73.5]; // Plucks quantized to 4Hz β€” whole cycles per quarter-beat note. const pent = [328, 392, 440, 492, 588]; - let sweepPhase = 0; // Phase-increment, not sin(TAU*f*t) β€” no drift on long tapes. let bassPhase = 0; let melPhase = 0; let ph1 = 0; // Generic per-style phase accumulators let ph2 = 0; let ph3 = 0; + let sineF = 164; // Sineline's gliding frequency (~E3 start) // πŸ₯ "break" style: one bar of 16th-note breakbeat, repeated β€” // scratch material with hard transients on a funk grid. const STEP = BEAT / 4; // 16ths at 120 BPM @@ -11600,6 +11600,20 @@ async function boot(parsed, bpm = 60, resolution, debug) { (Math.sin(ph1) + Math.sin(ph1 * 2) * 0.4) * Math.exp(-inHalf * 12) * 0.42; // Bass stab } + } else if (style === "sine") { + // 〰️ Sineline: one continuous legato sine gliding through a + // pentatonic line, with a soft sub octave β€” pure pitch + // material, lovely under a scrub. The glide keeps frequency + // continuous; a 15ms edge fade softens the loop seam. + const noteIdx = Math.floor(tt / BEAT); + const target = + pent[(noteIdx * 2 + Math.floor(noteIdx / 3)) % pent.length] / 2; + sineF += (target - sineF) * 0.0004; // ~50ms portamento + ph1 += (TAU * sineF) / sampleRate; + ph2 += (TAU * sineF) / 2 / sampleRate; + const sineEdge = Math.min(1, tt / 0.015, (duration - tt) / 0.015); + v += Math.sin(ph1) * 0.48 * sineEdge; + v += Math.sin(ph2) * 0.16 * sineEdge; } else if (style === "dub") { // 🌫️ Halftime dub: deep kicks on 1 & 3, rim on 3, a sub line // per half-bar, and offbeat skank chords. @@ -11650,12 +11664,8 @@ async function boot(parsed, bpm = 60, resolution, debug) { v += Math.sin(melPhase) * Math.exp(-inNote * 9) * 0.22; } - // Position sweep, quiet underneath, with a 50ms edge fade so - // the 880β†’220Hz jump at the loop seam can't click. - sweepPhase += (TAU * (220 + pos * 660)) / sampleRate; - const edge = Math.min(1, tt / 0.05, (duration - tt) / 0.05); - v += Math.sin(sweepPhase) * 0.07 * edge; - + // (The 220β†’880Hz position-sweep diagnostic is gone β€” the tapes + // are musical material now; position lives in the visuals.) pcm[s] = Math.tanh(v) * 32767; // Soft-clip the mix. } @@ -11930,11 +11940,16 @@ async function boot(parsed, bpm = 60, resolution, debug) { cachedCode: window.currentRecordingOptions?.cachedCode || null, }; - // Calculate total duration from frame timestamps - if (recordedFrames.length > 1) { + // Total duration must include the LAST frame's display time β€” + // mediaRecorderDuration has it. The timestamp span alone runs one + // frame short, which made the piece's wrap point land ~a frame + // before the audio loop's and drift by that much every lap. + if (mediaRecorderDuration > 0) { + info.totalDuration = mediaRecorderDuration / 1000; + } else if (recordedFrames.length > 1) { const firstTimestamp = recordedFrames[0][0]; const lastTimestamp = recordedFrames[recordedFrames.length - 1][0]; - info.totalDuration = (lastTimestamp - firstTimestamp) / 1000; // Convert to seconds + info.totalDuration = (lastTimestamp - firstTimestamp) / 1000; } // Reply back to the disk @@ -12308,6 +12323,32 @@ async function boot(parsed, bpm = 60, resolution, debug) { return; } + if (type === "tape:audio-pos") { + // Absolute read-head position (0..1) for the tape audio β€” used by + // beat jumps and chop repeats to relocate the free-running loop. + const tapeAudioId = Object.keys(sfxPlaying).find((id) => + id.startsWith("tape:audio_"), + ); + if (tapeAudioId && sfxPlaying[tapeAudioId]) { + const p = typeof content === "number" ? Math.max(0, Math.min(1, content)) : 0; + sfxPlaying[tapeAudioId].update({ samplePosition: p }); + } + return; + } + + if (type === "tape:audio-rate") { + // Absolute playback rate for the tape audio β€” drift-proof, unlike + // accumulating relative shifts against a piece-side estimate. + const tapeAudioId = Object.keys(sfxPlaying).find((id) => + id.startsWith("tape:audio_"), + ); + if (tapeAudioId && sfxPlaying[tapeAudioId]) { + const rate = typeof content === "number" ? content : 1; + sfxPlaying[tapeAudioId].update({ sampleSpeed: rate }); + } + return; + } + // πŸ“Ό Load and parse tape ZIP from URL (for replay piece) if (type === "tape:load-zip") { console.log("πŸ“Ό Loading tape ZIP from:", content); @@ -15321,10 +15362,6 @@ async function boot(parsed, bpm = 60, resolution, debug) { let tapeSoundId; - // πŸ•°οΈ Absolute grid for musical loop timing β€” see the loop branch. - let tapeLoopGridEpoch = null; - let tapeLoopGridCount = 0; - stopTapePlayback = () => { continuePlaying = false; stopped = true; @@ -15419,9 +15456,6 @@ async function boot(parsed, bpm = 60, resolution, debug) { // Adjust playback start time to match new position const currentTime = performance.now(); playbackStart = currentTime - (progress * playbackDurationMs); - // A seek breaks the musical loop grid β€” re-anchor at next loop. - tapeLoopGridEpoch = null; - tapeLoopGridCount = 0; // Update display with new frame if (recordedFrames[f]) { @@ -15617,7 +15651,11 @@ async function boot(parsed, bpm = 60, resolution, debug) { } // Advance frames while playback has progressed past the current frame's time - if (f >= recordedFrames.length - 1) { + if (isScrubbing && !render) { + // While the piece is speed-scrubbing, its seeks own the frame + // index β€” the RAF clock must not advance frames at 1Γ— between + // seeks (at slow rates that read as jitter and inaccuracy). + } else if (f >= recordedFrames.length - 1) { // For video export, don't loop - complete when all frames are processed if (doneCb && render) { console.log(`🎬 πŸ“Ή Video export reaching completion - final frame processed`); @@ -15627,25 +15665,12 @@ async function boot(parsed, bpm = 60, resolution, debug) { // For normal playback, loop f = 0; - // πŸ•°οΈ Musical loop grid: anchor each loop start to an absolute - // grid so the period can't accumulate RAF drift (measured - // ~-31ms/loop before). The next loop simply begins at - // epoch + N Γ— duration; if the frames arrive early the grid - // holds tempo, if late they catch up. + // πŸ•°οΈ Net-time phase lock: every loop boundary anchors to UTC + // multiples of the tape length, so any two players of the + // same tape converge into phase β€” and the local period can't + // accumulate RAF drift either. const gridNow = performance.now(); - if ( - tapeLoopGridEpoch === null || - Math.abs(gridNow - (tapeLoopGridEpoch + (tapeLoopGridCount + 1) * mediaRecorderDuration)) > - mediaRecorderDuration / 2 - ) { - // First loop or a broken grid (seek/suspend) β€” re-anchor. - tapeLoopGridEpoch = gridNow; - tapeLoopGridCount = 0; - playbackStart = gridNow; - } else { - tapeLoopGridCount += 1; - playbackStart = tapeLoopGridEpoch + tapeLoopGridCount * mediaRecorderDuration; - } + playbackStart = gridNow - (Date.now() % mediaRecorderDuration); playbackProgress = 0; // The next update's f===0 branch would reset playbackStart to // "now", clobbering the grid β€” isResuming makes it skip once. @@ -15670,7 +15695,7 @@ async function boot(parsed, bpm = 60, resolution, debug) { window.__lastTapeLoopAt = gridNow; if (loopPeriod) { console.log( - `πŸ•°οΈ Video loop period ${loopPeriod.toFixed(1)}ms vs tape ${mediaRecorderDuration.toFixed(1)}ms (drift ${(loopPeriod - mediaRecorderDuration).toFixed(1)}ms, grid loop #${tapeLoopGridCount})`, + `πŸ•°οΈ Video loop period ${loopPeriod.toFixed(1)}ms vs tape ${mediaRecorderDuration.toFixed(1)}ms (drift ${(loopPeriod - mediaRecorderDuration).toFixed(1)}ms, utc phase ${(Date.now() % mediaRecorderDuration).toFixed(0)}ms)`, ); } } diff --git a/system/public/aesthetic.computer/disks/video.mjs b/system/public/aesthetic.computer/disks/video.mjs index 9846c227f..43e6e0586 100644 --- a/system/public/aesthetic.computer/disks/video.mjs +++ b/system/public/aesthetic.computer/disks/video.mjs @@ -136,6 +136,26 @@ let flickVel = 0; // Low-passed drag velocity, for telling flicks from lets-go const FLICK_THRESHOLD = 5; // px/event of recent drag velocity const FLICK_KICK = 0.12; // Extra rate per px of flick velocity +// πŸ“Ÿ Steady-rate dial: dragging the top-right rate readout vertically sets +// a held steady rate β€” friction leaves it alone until it's brought back to +// 1Γ— (or spacebar resets). +let rateBtn = null; +let steadyHold = false; +let rateDragStartY = 0; +let rateDragStartRate = 1; + +// 🧭 Gesture vector debug: the grab anchor and current finger, so the +// stretch reads as a drawn vector (direction + energy) on screen. +let elasticAnchorY = null; +let penX = null; +let penY = null; + +// πŸŽ›οΈ Deck keys (Pioneer-style): ← β†’ beat-jump; holding ↑/↓ chop-repeats a +// beat fraction. chopActive holds the slice length in beats (0 = off). +let chopActive = 0; +let chopStart = 0; +const BEAT_SEC = 0.5; // 120 BPM + // Scrub physics runs on measured wall time, not an assumed tick rate β€” // sim ticks at ~120Hz here, and a fixed 1/60 dt made every commanded // speed land at roughly half its real value. @@ -207,10 +227,11 @@ function ensureScrubStripButton(ui, screen, enabled) { } function nudgeTapeAudioSpeed(send, targetSpeed) { - const clampedTarget = Math.max(-16, Math.min(16, targetSpeed)); - const delta = clampedTarget - scrubAudioSpeed; - if (Math.abs(delta) < 0.0005) return; - send({ type: "tape:audio-shift", content: delta }); + // Absolute rate: the worklet is set to exactly this speed, so repeated + // gestures can never accumulate drift the way relative shifts did. + const clampedTarget = Math.max(-24, Math.min(24, targetSpeed)); + if (Math.abs(clampedTarget - scrubAudioSpeed) < 0.0005) return; + send({ type: "tape:audio-rate", content: clampedTarget }); scrubAudioSpeed = clampedTarget; } @@ -313,6 +334,9 @@ function boot({ wipe, rec, gizmo, jump, notice, store, params, send, hud }) { lastScrollAt = 0; wheelActive = false; sustained = false; + steadyHold = false; + rateBtn = null; + chopActive = 0; flickVel = 0; resumeTarget = 1; dipBase = 1; @@ -444,7 +468,9 @@ function boot({ wipe, rec, gizmo, jump, notice, store, params, send, hud }) { ? "house" : rest.includes("dub") ? "dub" - : "bed"; + : rest.includes("sine") || rest.includes("line") + ? "sine" + : "bed"; const nums = rest.map(parseFloat).filter((n) => Number.isFinite(n)); let duration = nums[0] || 8; // Four bars by default, break included if (rest.includes("bar")) duration = 2; @@ -485,6 +511,7 @@ function paint({ sound, send, num, + clock, }) { if (typeof needsPaint === "function") { requestPaint = needsPaint; @@ -848,11 +875,21 @@ function paint({ sustained || tapDipTime >= 0; const liveRate = scrubDriven ? scrubSpeed : playing ? 1 : 0; - ink(255, 255, 0).write(`${liveRate.toFixed(2)}x`, { - x: screen.width - 6, - y: 6, - right: true, - }); + ink(steadyHold ? [0, 255, 180] : [255, 255, 0]).write( + `${liveRate.toFixed(2)}x`, + { x: screen.width - 6, y: 6, right: true }, + ); + + // πŸ“Ÿ The readout is also a dial β€” keep its hit area in place. + if (!rateBtn) { + rateBtn = new ui.Button(screen.width - 64, 0, 64, 24); + rateBtn.stickyScrubbing = true; + rateBtn.noRolloverActivation = true; + } + rateBtn.box.x = screen.width - 64; + rateBtn.box.y = 0; + rateBtn.box.w = 64; + rateBtn.box.h = 24; // πŸ”΄ The red marker rides the bottom edge at the actual playback // position β€” drawn from live state so it never lies. @@ -865,6 +902,56 @@ function paint({ 6, 4, ); + + // πŸ•°οΈ Net-time unison readout: phase offset vs the AC network clock's + // grid β€” green when locked, amber while converging. + if (tapeInfo?.totalDuration) { + const durMs = tapeInfo.totalDuration * 1000; + const nowMs = clock?.time?.()?.getTime?.() ?? Date.now(); + let phaseErr = (nowMs % durMs) / durMs - livePos; + if (phaseErr > 0.5) phaseErr -= 1; + else if (phaseErr < -0.5) phaseErr += 1; + const errMs = Math.round(phaseErr * durMs); + const locked = Math.abs(errMs) < 60; + ink(locked ? [0, 255, 120] : [255, 170, 0]).write( + `sync ${errMs >= 0 ? "+" : ""}${errMs}ms`, + { x: screen.width - 6, y: 18, right: true }, + ); + } + + // 🧭 Gesture vector: anchor β†’ finger, with the horizontal component + // (the part that drives the rate) emphasized. + if (isScrubbing && elasticAnchorX !== null && penX !== null) { + const ay = elasticAnchorY ?? penY ?? 0; + ink(255, 255, 255, 70).line(elasticAnchorX, ay, penX, penY ?? ay); + const fwd = penX >= elasticAnchorX; + ink(fwd ? [0, 255, 120, 200] : [255, 80, 80, 200]).box( + Math.min(elasticAnchorX, penX), + (ay) - 1, + Math.max(1, Math.abs(penX - elasticAnchorX)), + 3, + ); + ink(255, 255, 255, 220).box(elasticAnchorX - 2, ay - 2, 5, 5); + ink(255, 255, 255).write(`${scrubSpeed.toFixed(2)}x`, { + x: penX + 8, + y: (penY ?? ay) - 4, + }); + } + + // ⌨️ Deck keys legend, bottom-right. + const keyLines = [ + "<- -> beat jump", + "hold ^ 1/4 chop", + "hold v 1/8 chop", + "space reset", + ]; + keyLines.forEach((l, i) => { + ink(255, 255, 255, 110).write(l, { + x: screen.width - 6, + y: screen.height - 10 - (keyLines.length - i) * 10, + right: true, + }); + }); } // Scrub overlay (STAMPLE-style speed-based) @@ -950,7 +1037,7 @@ function paint({ return true; // Always keep painting } -function sim({ needsPaint, rec, send }) { +function sim({ needsPaint, rec, send, clock }) { ellipsisTicker?.sim(); frameCount++; // Increment frame counter for animations @@ -1033,18 +1120,10 @@ function sim({ needsPaint, rec, send }) { const phase = Math.min(1, tapDipTime / TAP_DIP_DURATION); scrubSpeed = dipBase * (1 - TAP_DIP_DEPTH * Math.sin(Math.PI * phase)); if (phase >= 1) { + // Lands as a seamless park at the departed rate β€” no handoff jump. tapDipTime = -1; - if (Math.abs(dipBase - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = dipBase; - sustained = true; - } + scrubSpeed = dipBase; + sustained = true; } } @@ -1091,21 +1170,14 @@ function sim({ needsPaint, rec, send }) { if (scrubSpeed < 0.02) scrubSpeed = 0; } - // πŸ–±οΈ Scroll release: the wheel went quiet β€” park at the landed rate. + // πŸ–±οΈ Scroll release: the wheel went quiet β€” a seamless park at the + // landed rate (pinned to exactly 1 when close). if (scrollScrubbing && isScrubbing && performance.now() - lastScrollAt > 150) { scrollScrubbing = false; tapDipTime = -1; isScrubbing = false; - if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - sustained = true; - } + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) scrubSpeed = 1; + sustained = true; } // 🎰 Wheel spin-down: after a flick the platter runs free, then eases @@ -1114,33 +1186,50 @@ function sim({ needsPaint, rec, send }) { scrubSpeed = resumeTarget + (scrubSpeed - resumeTarget) * Math.pow(0.975, rate); if (Math.abs(scrubSpeed - resumeTarget) < 0.04) { + // Lands as a seamless park β€” the drive keeps running, no handoff. wheelActive = false; - if (Math.abs(resumeTarget - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = resumeTarget; - sustained = true; - } + scrubSpeed = resumeTarget; + sustained = true; } } // πŸŒ€ Friction: even a parked rate isn't forever β€” it glides home to // 1Γ— over a few seconds, like a wheel that always feels the bearing. - if (sustained && !isScrubbing) { + // It never hands off: the drive just converges to exactly 1.0 and + // keeps driving β€” seamless by construction. + if (sustained && !steadyHold && !isScrubbing && !chopActive) { scrubSpeed += (1 - scrubSpeed) * (1 - Math.pow(0.9965, rate)); - if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - sustained = false; - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); + if (Math.abs(scrubSpeed - 1) < 0.005) scrubSpeed = 1; + + // πŸ•°οΈ Net-time phase pull: at rest the loop always lerps toward the + // AC network clock's phase grid (clock.mjs / /api/clock synced), so + // every player of this tape converges into unison β€” a gentle Β±5% + // tempo lean, never a jump. + if (Math.abs(scrubSpeed - 1) < 0.01) { + const durMs = totalDuration * 1000; + const nowMs = clock?.time?.()?.getTime?.() ?? Date.now(); + const target = (nowMs % durMs) / durMs; + let phaseErr = target - scrubCurrentProgress; + if (phaseErr > 0.5) phaseErr -= 1; + else if (phaseErr < -0.5) phaseErr += 1; + scrubSpeed = 1 + Math.max(-0.05, Math.min(0.05, phaseErr * 0.15)); + } + } + + // πŸŒ€ Chop repeat: while ↑/↓ is held, loop a beat-fraction slice β€” + // glitch stutter, Pioneer-style. + if (chopActive && rec?.presenting) { + const chopLen = (chopActive * BEAT_SEC) / totalDuration; + let rel = scrubCurrentProgress - chopStart; + if (rel < -0.5) rel += 1; + else if (rel > 0.5) rel -= 1; + if (rel >= chopLen || rel < 0) { + scrubCurrentProgress = chopStart; send({ type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, + content: { progress: chopStart, speedScrub: true }, }); + send({ type: "tape:audio-pos", content: chopStart }); } } @@ -1148,18 +1237,10 @@ function sim({ needsPaint, rec, send }) { if (brakeResume && !isScrubbing) { scrubSpeed += (resumeTarget - scrubSpeed) * (1 - Math.pow(0.88, rate)); if (Math.abs(scrubSpeed - resumeTarget) < 0.05) { + // Lands as a seamless park β€” the drive keeps running, no handoff. brakeResume = false; - if (Math.abs(resumeTarget - 1) < PARK_SNAP) { - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); - } else { - scrubSpeed = resumeTarget; - sustained = true; - } + scrubSpeed = resumeTarget; + sustained = true; } } @@ -2263,6 +2344,8 @@ function act({ sustained = false; scrollScrubbing = false; tapDipTime = -1; + steadyHold = false; + chopActive = 0; scrubSpeed = 0; nudgeTapeAudioSpeed(send, 1); if (rec.playing) rec.pause(); @@ -2271,6 +2354,41 @@ function act({ return; } + // πŸŽ›οΈ Deck keys, Pioneer-style: ← β†’ beat-jump the tape (audio included); + // holding ↑ chop-repeats a quarter beat, ↓ an eighth β€” release to run on. + if (rec.presenting && tapeInfo?.totalDuration) { + const dur = tapeInfo.totalDuration; + const beatJump = (beats) => { + ensureDriven(rec, send); + let p = scrubCurrentProgress + (beats * BEAT_SEC) / dur; + p = ((p % 1) + 1) % 1; + scrubCurrentProgress = p; + send({ + type: "recorder:present:seek", + content: { progress: p, speedScrub: true }, + }); + send({ type: "tape:audio-pos", content: p }); + triggerRender(); + }; + if (e.is("keyboard:down:arrowleft")) beatJump(-1); + if (e.is("keyboard:down:arrowright")) beatJump(1); + if (e.is("keyboard:down:arrowup") && !chopActive) { + ensureDriven(rec, send); + chopActive = 0.25; + chopStart = scrubCurrentProgress; + send({ type: "tape:audio-pos", content: chopStart }); + } + if (e.is("keyboard:down:arrowdown") && !chopActive) { + ensureDriven(rec, send); + chopActive = 0.125; + chopStart = scrubCurrentProgress; + send({ type: "tape:audio-pos", content: chopStart }); + } + if (e.is("keyboard:up:arrowup") || e.is("keyboard:up:arrowdown")) { + chopActive = 0; + } + } + // πŸ–±οΈ Two-finger scroll scrubs directly β€” no tap-drag needed. Wheel // deltas bend the rate; when the gesture stops it springs back to 1Γ—. if (e.is("scroll") && rec.presenting) { @@ -2297,15 +2415,61 @@ function act({ } scrubMoved = true; // Expressive range: two-finger scroll can push way past the drag's - // reach β€” up to Β±12Γ—. Negated so scroll direction matches drag + // reach β€” up to Β±24Γ—. Negated so scroll direction matches drag // direction (natural scrolling inverts the wheel deltas). - scrubSpeed = Math.max(-12, Math.min(12, scrubSpeed * 0.6 - d * 0.35)); + scrubSpeed = Math.max(-24, Math.min(24, scrubSpeed * 0.6 - d * 0.35)); lastScrollAt = performance.now(); nudgeTapeAudioSpeed(send, scrubSpeed); triggerRender(); } } + // πŸ“Ÿ Steady-rate dial: vertical drag on the top-right readout sets a + // held rate the friction won't touch. + if (rateBtn && rec.presenting && !isPrinting && !isPostingTape) { + rateBtn.act(e, { + down: () => { + rateDragStartY = e.y ?? 0; + rateDragStartRate = + isScrubbing || inertiaActive || brakeResume || wheelActive || sustained || tapDipTime >= 0 + ? scrubSpeed + : rec.playing + ? 1 + : 0; + }, + scrub: () => { + if (e.y === undefined) return; + if (!sustained && !isScrubbing) { + // Engage the drive so the dialed rate actually plays. + scrubCurrentProgress = rec.presentProgress || scrubCurrentProgress || 0; + if (!rec.playing) rec.play(); + send({ + type: "recorder:present:seek", + content: { progress: scrubCurrentProgress, speedScrub: true, scrubbing: true }, + }); + } + inertiaActive = false; + brakeResume = false; + wheelActive = false; + tapDipTime = -1; + sustained = true; + steadyHold = true; + const dyRate = (rateDragStartY - e.y) * 0.03; + scrubSpeed = Math.max(-24, Math.min(24, rateDragStartRate + dyRate)); + nudgeTapeAudioSpeed(send, scrubSpeed); + triggerRender(); + }, + up: () => { + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { + scrubSpeed = 1; + steadyHold = false; // Back at play speed β€” friction may hold it + } + triggerRender(); + }, + }); + if (rateBtn.down) return; // The dial owns this gesture + } + if (!anyButtonDown && !isPrinting && !isPostingTape && rec.presenting) { ensureScrubStripButton(ui, screen, true); scrubStripBtn?.act(e, { @@ -2319,6 +2483,9 @@ function act({ holdTime = 0; flickVel = 0; elasticAnchorX = e.x ?? null; + elasticAnchorY = e.y ?? null; + penX = e.x ?? null; + penY = e.y ?? null; // πŸ…ΏοΈ A parked, spinning, or ramping rate survives a new touch: // the gesture starts FROM it β€” so you can grab a fast wheel and // drag it slower β€” and returns TO it. @@ -2362,6 +2529,8 @@ function act({ // grab point IS the rate β€” hold still and it holds still, a third // of the screen right drags at 3Γ—, left drags in reverse. const dx = e.x - elasticAnchorX; + penX = e.x; + penY = e.y ?? penY; if (Math.abs(dx) > 2) scrubMoved = true; flickVel = flickVel * 0.6 + (e.delta?.x || 0) * 0.4; const K = 9 / screen.width; @@ -2382,22 +2551,18 @@ function act({ // 🎰 Flick: the platter runs free with the throw's momentum, // then eases down like a prize wheel to the pre-flick rate. scrubSpeed = Math.max( - -12, - Math.min(12, scrubSpeed + flickVel * FLICK_KICK), + -24, + Math.min(24, scrubSpeed + flickVel * FLICK_KICK), ); nudgeTapeAudioSpeed(send, scrubSpeed); wheelActive = true; - } else if (Math.abs(scrubSpeed - 1) < PARK_SNAP) { - // Close enough to play speed β€” hand back to normal playback. - scrubSpeed = 0; - nudgeTapeAudioSpeed(send, 1); - send({ - type: "recorder:present:seek", - content: { progress: scrubCurrentProgress, scrubEnd: true }, - }); } else { - // πŸ…ΏοΈ Gentle release: the rate parks where you left it. + // πŸ…ΏοΈ Release parks where you left it. Near 1Γ— it pins to + // exactly 1 β€” the scrub drive at 1.0 IS normal playback, so + // there's no handoff and no jump, ever. + if (Math.abs(scrubSpeed - 1) < PARK_SNAP) scrubSpeed = 1; sustained = true; + nudgeTapeAudioSpeed(send, scrubSpeed); } triggerRender(); return true; @@ -3048,6 +3213,7 @@ function leave({ send }) { tapDipTime = -1; wheelActive = false; sustained = false; + steadyHold = false; flickVel = 0; scrubSpeed = 0; scrubCurrentProgress = 0; @@ -3069,6 +3235,29 @@ export { boot, paint, sim, act, signal, receive, leave }; // πŸ“š Library (Useful functions used throughout the piece) +// πŸŽ›οΈ Make sure the scrub drive owns playback (used by deck keys) β€” engages +// a sustained 1Γ— drive if nothing else is driving. +function ensureDriven(rec, send) { + if ( + isScrubbing || + sustained || + wheelActive || + brakeResume || + inertiaActive || + tapDipTime >= 0 + ) { + return; + } + scrubCurrentProgress = rec.presentProgress || 0; + scrubSpeed = 1; + sustained = true; + if (!rec.playing) rec.play(); + send({ + type: "recorder:present:seek", + content: { progress: scrubCurrentProgress, speedScrub: true, scrubbing: true }, + }); +} + // πŸ§ͺ Synthtape autopilot β€” a scripted stand-in for the finger. Each segment // either lets the tape roll, holds a scrub speed, or releases into inertia, // then the observed progress delta is graded against the physics. All output @@ -3092,6 +3281,8 @@ function autopilot(rec, send, simDt) { { name: "tap dip", mode: "dip" }, { name: "scratch", mode: "scratch", frames: 150 }, { name: "scratch release", mode: "coast" }, + { name: "fast scratch", mode: "scratch", frames: 120, hz: 4, amp: 2.2 }, + { name: "fast scratch release", mode: "coast" }, { name: "touch brake", mode: "brake", frames: 90 }, { name: "brake release spin-up", mode: "brakeRelease" }, ]; @@ -3200,8 +3391,11 @@ function autopilot(rec, send, simDt) { autoTimer += 1; if (autoSeg.mode === "scratch") { - // 🎚️ Musical scratch: the rate rocks Β±3.2Γ— at ~2Hz like a hand on the reel. - scrubSpeed = 3.2 * Math.sin((autoTimer / 60) * Math.PI * 2); + // 🎚️ Musical scratch: the rate rocks like a hand on the reel β€” + // amplitude and tempo per segment (default Β±3.2Γ— at 2Hz). + const amp = autoSeg.amp || 3.2; + const hz = autoSeg.hz || 2; + scrubSpeed = amp * Math.sin((autoTimer / 120) * Math.PI * 2 * hz); } if (autoSeg.mode === "scrub") { scrubSpeed = autoSeg.speed; // Hold the drag @@ -3268,8 +3462,10 @@ function autopilot(rec, send, simDt) { } } else if (autoSeg.mode === "scratch") { const totalDuration = tapeInfo?.totalDuration || 10; + const amp = autoSeg.amp || 3.2; const driftSecs = Math.abs(autoMotionSum * totalDuration); - pass = autoSpeedMax > 1.5 && autoSpeedMin < -1.5 && driftSecs < 0.7; + pass = + autoSpeedMax > amp * 0.45 && autoSpeedMin < -amp * 0.45 && driftSecs < 0.7; detail += ` (rate swung ${autoSpeedMin.toFixed(1)}..${autoSpeedMax.toFixed(1)}Γ—, net drift ${driftSecs.toFixed(2)}s)`; } else if (autoSeg.mode === "dip") { const totalDuration = tapeInfo?.totalDuration || 10; -- 2.51.2 From 75e477d5b61370e02f555993a411a96ca4033bb1 Mon Sep 17 00:00:00 2001 From: "prompt.ac/@jeffrey" Date: Sun, 19 Jul 2026 14:28:42 -0700 Subject: [PATCH 2/2] =?UTF-8?q?spinging:=20round-6=20=E2=80=94=20scales=20?= =?UTF-8?q?singalong=20reel,=20+12=20register=20lift=20with=20fallback=20l?= =?UTF-8?q?adder?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- pop/menuband/bin/reel-lib.mjs | 9 +- pop/menuband/bin/render-jingles.mjs | 98 ++++++++- pop/menuband/bin/sim-scales.mjs | 200 +++++++++++++++++ pop/menuband/bin/sing-jingle.mjs | 318 ++++++++++++++++++++++++---- spinging/README.md | 38 ++++ spinging/cache/pronounce.json | 98 +++++++++ spinging/lib/pronounce.mjs | 16 +- spinging/lib/sing_line_world.py | 102 +++++++-- spinging/lib/vocal_shapes.py | 32 ++- 9 files changed, 842 insertions(+), 69 deletions(-) create mode 100644 pop/menuband/bin/sim-scales.mjs diff --git a/pop/menuband/bin/reel-lib.mjs b/pop/menuband/bin/reel-lib.mjs index 1c77099a2..3eab92cbd 100644 --- a/pop/menuband/bin/reel-lib.mjs +++ b/pop/menuband/bin/reel-lib.mjs @@ -165,9 +165,12 @@ export function makeParticles(ctx) { } // ── the strip rig β€” real captured strip, re-lightable per note set ───────── -/// Midis the cache holds single-note captures for (the waltz's white keys, -/// G4..B5). Any melody is folded onto these before lighting. -export const STRIP_MIDIS = [67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; +/// Midis the cache holds single-note captures for: the waltz's white keys +/// (G4..B5) + the low C4..F4 the scales ladder added (round 6 β€” captured via +/// `MenuBand --render-menubar --notes --light`, same args as sim.mjs). +/// Together that's every white key on the strip, C4..B5. Any melody is +/// folded onto these before lighting. +export const STRIP_MIDIS = [60, 62, 64, 65, 67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; const PC_TO_STRIP = new Map([[0, 72], [2, 74], [4, 76], [5, 77], [7, 79], [9, 69], [11, 71]]); /// Fold any (white-key) midi onto a strip key: same pitch class, the cached diff --git a/pop/menuband/bin/render-jingles.mjs b/pop/menuband/bin/render-jingles.mjs index 88b131d1f..c207a4ae6 100644 --- a/pop/menuband/bin/render-jingles.mjs +++ b/pop/menuband/bin/render-jingles.mjs @@ -1,10 +1,13 @@ #!/usr/bin/env node -// render-jingles.mjs β€” the three campaign jingles for the Menu Band promo -// reels (announce / features / chords), built on the same /pop lullaby engine -// as the launch waltz (render-waltz.mjs). Each jingle writes an mp3 + a -// notes.json its sim choreographs to; the chords jingle also writes a -// segment score (menuband-chords.score.json) so the audio and the on-screen -// modifier keycaps agree to the frame. +// render-jingles.mjs β€” the four campaign jingles for the Menu Band promo +// reels (announce / features / chords / scales), built on the same /pop +// lullaby engine as the launch waltz (render-waltz.mjs). Each jingle writes +// an mp3 + a notes.json its sim choreographs to; the chords jingle also +// writes a segment score (menuband-chords.score.json) so the audio and the +// on-screen modifier keycaps agree to the frame; the scales jingle writes +// menuband-scales.score.json (spoken cue times + the letter ladder) that +// sing-jingle.mjs and sim-scales.mjs both read, so the voice, the bed and +// the visuals share ONE timeline. // // EVERY lead note is a WHITE KEY: the strip rig re-lights the real captured // menu-bar piano from the single-note captures sim.mjs cached (G4..B5, @@ -201,3 +204,86 @@ function master(name, events, opts) { JSON.stringify({ durationSec: +durationSec.toFixed(4), segs }, null, 2)); console.log(` ${segs.length} chord segments`); } + +// ════════════════════════════════════════════════════════════════════════ +// 4 Β· SCALES β€” the teaching singalong (round 6). Spoken frame + the notepat +// two-octave letter ladder (c d e f g a b Β· h i j k l m n = the second +// octave, straight from notepat's NOTE_TO_KEYBOARD_KEY) sung up then +// down at 108 BPM over a soft drone bed. The `lead` lane carries the +// ladder at the STRIP's midis (C4..B5 β€” every one a real cached key); +// jeffrey sings it an octave below (C3..B4, bright top, unstrained). +// The bed stays light β€” a pedal drone, gentle kalimba, a quiet +// vibraphone doubling the ladder β€” never burying the voice. +// ════════════════════════════════════════════════════════════════════════ +{ + const BPM = 108, BEAT = 60 / BPM, BAR = 4 * BEAT; + const { events, push } = makeTrack(); + + // ── the shared timeline (spoken cues measured off the cached TTS takes) ── + const INTRO_T = 0.8; // "Here's how to type out the C scale." (~2.1s) + const RUN_T = 3.5; // "C, D, E, F, G, A, B. Now sing it!" (~2.9s) + const SING0 = 6.95; // the ladder's first beat + const LETTERS_ASC = ["c", "d", "e", "f", "g", "a", "b", + "h", "i", "j", "k", "l", "m", "n"]; + const STRIP_ASC = [60, 62, 64, 65, 67, 69, 71, 72, 74, 76, 77, 79, 81, 83]; + const HOLD_TOP = 2.0, HOLD_BOTTOM = 2.5; // beats: top n rings, home c lands + const ladder = []; + LETTERS_ASC.forEach((letter, i) => { + const top = i === LETTERS_ASC.length - 1; + ladder.push({ letter, t: +(SING0 + i * BEAT).toFixed(4), + dur: +((top ? HOLD_TOP : 0.92) * BEAT).toFixed(4), + strip: STRIP_ASC[i], vocal: STRIP_ASC[i] - 12, dir: "up" }); + }); + // descending starts on m after the top-n hold (2 beats of ring) + const DESC0 = SING0 + (LETTERS_ASC.length + 1) * BEAT; + for (let j = 0; j < 13; j++) { + const i = 12 - j; // m l k j i h b a g f e d c + const bottom = j === 12; + ladder.push({ letter: LETTERS_ASC[i], t: +(DESC0 + j * BEAT).toFixed(4), + dur: +((bottom ? HOLD_BOTTOM : 0.92) * BEAT).toFixed(4), + strip: STRIP_ASC[i], vocal: STRIP_ASC[i] - 12, dir: "down" }); + } + const SING1 = DESC0 + 12 * BEAT + HOLD_BOTTOM * BEAT; // ladder ends + const OUTRO_T = +(SING1 + 0.9).toFixed(3); // "Wanna type it yourself? …" + + // ── the bed ────────────────────────────────────────────────────────────── + // pedal drone: soft C2/G2 bass, one note per bar, entering under the intro + const LAST_BAR = Math.ceil((OUTRO_T + 3.2) / BAR); + for (let bar = 0; bar < LAST_BAR; bar++) { + const t = bar * BAR; + push("bass", "bass", t, bar % 4 === 3 ? "G2" : "C2", 3.6 * BEAT, 0.20, 0); + } + // gentle kalimba broken chord, quiet, from the spoken run onward + const ARP = ["C4", "E4", "G4", "A3"]; + for (let bar = 1; bar < LAST_BAR - 1; bar++) { + for (let s = 0; s < 4; s++) { + push("harmony", "kalimba", bar * BAR + s * BEAT, ARP[s % 4], + 0.9 * BEAT, 0.085, s % 2 ? 0.2 : -0.2, 1.0); + } + } + // the ladder itself: lead lane at STRIP midis β€” lights the real keys and + // doubles the voice an octave up, quietly + for (const n of ladder) { + push("lead", "vibraphone", n.t, n.strip, n.dur * 1.15, 0.22, 0.05, 1.25); + } + // soft woodblock ticks keep the singalong honest through the ladder bars + for (let t = SING0; t < SING1 - BEAT; t += 2 * BEAT) { + push("perc", "woodblock", +t.toFixed(4), "E5", 0.12, 0.06, 0.12); + } + // home-chord sparkle under the outro CTA + push("harmony", "glockenspiel", OUTRO_T + 0.3, m("E6"), 2.0, 0.08, 0.28, 1.5); + push("harmony", "glockenspiel", OUTRO_T + 0.9, m("C6"), 2.4, 0.09, -0.2, 1.5); + + const durationSec = master("menuband-scales", events, { title: "Menu Band C Scale Singalong" }); + writeScore("menuband-scales", events, {}, { + bpm: BPM, beatSec: +BEAT.toFixed(6), barSec: +BAR.toFixed(6), + durationSec: +durationSec.toFixed(4), + }); + writeFileSync(resolve(OUT_DIR, "menuband-scales.score.json"), JSON.stringify({ + bpm: BPM, beatSec: +BEAT.toFixed(6), durationSec: +durationSec.toFixed(4), + spoken: { intro: { t: INTRO_T }, run: { t: RUN_T }, outro: { t: OUTRO_T } }, + sing: { t0: +SING0.toFixed(4), t1: +SING1.toFixed(4) }, + ladder, + }, null, 2)); + console.log(` ladder ${ladder.length} letters Β· sing ${SING0.toFixed(2)}–${SING1.toFixed(2)}s Β· outro @ ${OUTRO_T}s`); +} diff --git a/pop/menuband/bin/sim-scales.mjs b/pop/menuband/bin/sim-scales.mjs new file mode 100644 index 000000000..93dc98566 --- /dev/null +++ b/pop/menuband/bin/sim-scales.mjs @@ -0,0 +1,200 @@ +#!/usr/bin/env node +// sim-scales.mjs β€” the Menu Band SCALES teaching-singalong reel base (v6). +// +// The reel that literally demonstrates the spinging speakβ†’sing transform: +// a spoken frame ("Here's how to type out the C scale." … "Now sing it!") +// around the notepat two-octave letter ladder β€” c d e f g a b h i j k l m n +// up and back down (h..n = notepat's second octave) β€” each letter sung on +// its own scale degree while the REAL captured menu-bar strip lights that +// key (strip rig, now covering every white key C4..B5) and a BIG keycap +// shows the letter karaoke-chart style. Spoken lines get normal captions. +// +// Timeline comes from out/menuband-scales.score.json (render-jingles.mjs) β€” +// the same sidecar sing-jingle.mjs sings from, so audio, voice and visuals +// share ONE clock. Word timings: out/menuband-scales.words.sung.json. +// +// This reel only exists sung β€” run with --sung (the suffix keeps the +// naming pattern of the other campaign reels): +// node pop/menuband/bin/sing-jingle.mjs menuband-scales +// node pop/menuband/bin/sim-scales.mjs --sung +// node pop/menuband/bin/chrome-reel.mjs menuband-scales-sung + +import { readFileSync } from "node:fs"; +import { + W, H, FPS, OUT, INK, INK_RGB, easeOut, clamp01, rgb, + makeStage, roundRect, text, drawDesktop, vignette, drawIcon, + makeParticles, loadStripRig, drawStrip, stripKeyX, stripKeyColor, + loadScore, leadOf, litAt, makeOnsets, + renderVideo, writeMeta, makeScenes, sungMode, loadSungWords, makeKaraoke, +} from "./reel-lib.mjs"; + +const SLUG = "menuband-scales"; +const { sung: SUNG, suffix: VAR } = sungMode(); +if (!SUNG) { + console.error("βœ— the scales reel only exists sung β€” run with --sung"); + process.exit(1); +} +const score = loadScore(SLUG); +const sc = JSON.parse(readFileSync(`${OUT}/${SLUG}.score.json`, "utf8")); +const TOTAL = score.durationSec; +const lead = leadOf(score); +const onsetsBetween = makeOnsets(lead); + +// spoken words β†’ normal bottom captions; sung letters drive the big keycap +const allWords = loadSungWords(SLUG); +const karaoke = makeKaraoke(allWords.filter((w) => w.spoken), { y: H * 0.925 }); + +const { canvas, ctx } = makeStage(); +const rig = await loadStripRig(); +const particles = makeParticles(ctx); + +const SING0 = sc.sing.t0, SING1 = sc.sing.t1; +const { scenes: SCENES, sceneAt } = makeScenes([ + { name: "intro", from: 0, to: sc.spoken.run.t / TOTAL, tint: [97, 158, 255] }, + { name: "speak", from: sc.spoken.run.t / TOTAL, to: SING0 / TOTAL, tint: [255, 214, 56] }, + { name: "sing-up", from: SING0 / TOTAL, to: (SING0 + (SING1 - SING0) * 0.52) / TOTAL, tint: [51, 209, 179] }, + { name: "sing-down", from: (SING0 + (SING1 - SING0) * 0.52) / TOTAL, to: (SING1 + 0.8) / TOTAL, tint: [167, 139, 250] }, + { name: "end", from: (SING1 + 0.8) / TOTAL, to: 1.0, tint: [255, 77, 107] }, +], TOTAL); + +// ── the strip: parked upper area, playing the ladder ─────────────────────── +const HERO_W = W * 0.96, HERO_X = (W - HERO_W) / 2; +function heroRect(t) { + const h = HERO_W / rig.aspect; + const enter = easeOut(clamp01(t / 1.1)); + const bob = Math.sin(t * 1.5) * 7 * enter; + const rest = H * 0.165 - h / 2; + return { x: HERO_X, y: (-h - 40) + (rest - (-h - 40)) * enter + bob, w: HERO_W, h }; +} + +// ── the ladder clock: which letter is active at t ────────────────────────── +const LADDER = sc.ladder; // [{ letter, t, dur, strip, vocal, dir }] +function activeNote(t) { + let cur = null; + for (const n of LADDER) if (t >= n.t) cur = n; else break; + if (!cur) return null; + if (t > cur.t + Math.max(cur.dur, 0.5) + 0.35) return null; // ladder done + return cur; +} +const LETTERS = [...new Set(LADDER.map((n) => n.letter))]; // c..n in order + +// ── the BIG teaching keycap β€” the letter you type, huge, in the key's own +// strip color, pressing on every onset ───────────────────────────────────── +function drawKeycap(t) { + const n = activeNote(t); + if (!n) return; + const local = clamp01((t - n.t) / 0.14); + const press = 1 - easeOut(local); // 1 at strike β†’ 0 settled + const inA = easeOut(clamp01((t - SING0 + 0.4) / 0.5)) * + (1 - easeOut(clamp01((t - (SING1 + 0.35)) / 0.5))); + if (inA <= 0) return; + const size = 430; + const x = W / 2 - size / 2, y = H * 0.335 - size / 2 + press * 14; + const col = stripKeyColor(rig, n.strip); + ctx.save(); + ctx.globalAlpha = inA; + ctx.shadowColor = "rgba(0,0,0,0.32)"; + ctx.shadowBlur = 34 * (1 - press * 0.7); ctx.shadowOffsetY = 16 * (1 - press * 0.7); + roundRect(ctx, x, y, size, size, 64); + ctx.fillStyle = "rgb(250,249,253)"; ctx.fill(); + ctx.shadowColor = "transparent"; + // the key's strip color washes the cap as it presses + ctx.globalAlpha = inA * (0.16 + 0.22 * press); + roundRect(ctx, x, y, size, size, 64); + ctx.fillStyle = rgb(col); ctx.fill(); + ctx.globalAlpha = inA; + ctx.lineWidth = 3; ctx.strokeStyle = "rgba(20,18,28,0.22)"; + roundRect(ctx, x, y, size, size, 64); ctx.stroke(); + const pop = 1 + 0.09 * press; + ctx.translate(W / 2, y + size * 0.47); ctx.scale(pop, pop); + text(ctx, n.letter.toUpperCase(), 0, 0, 250, INK, 800); + ctx.scale(1 / pop, 1 / pop); + text(ctx, n.dir === "up" ? "↑" : "↓", 0, size * 0.36, 54, rgb(col), 800); + ctx.restore(); +} + +// ── the ladder chart β€” all 14 letters, the singalong teaching row ────────── +function drawChart(t) { + const a = easeOut(clamp01((t - SING0 + 0.6) / 0.5)) * + (1 - easeOut(clamp01((t - (SING1 + 0.5)) / 0.5))); + if (a <= 0) return; + const n = activeNote(t); + const activeIdx = n ? LETTERS.indexOf(n.letter) : -1; + const goingUp = !n || n.dir === "up"; + const cellW = 68, y = H * 0.585; + const x0 = W / 2 - (LETTERS.length * cellW) / 2; + ctx.save(); + ctx.globalAlpha = a; + for (let i = 0; i < LETTERS.length; i++) { + const cx = x0 + i * cellW + cellW / 2; + const isActive = i === activeIdx; + const sungAlready = goingUp ? i < activeIdx : i > activeIdx; + if (isActive) { + const col = stripKeyColor(rig, LADDER[i].strip); + roundRect(ctx, cx - 28, y - 34, 56, 68, 16); + ctx.fillStyle = rgb(col); ctx.fill(); + text(ctx, LETTERS[i], cx, y + 1, 44, "rgba(255,255,255,0.98)", 800); + } else { + text(ctx, LETTERS[i], cx, y + 1, 40, + sungAlready ? INK : "rgba(20,18,28,0.32)", 700); + } + } + text(ctx, "c major Β· two octaves Β· h through n = the next c to b", + W / 2, y + 74, 34, "rgba(60,50,80,0.85)", 600); + ctx.restore(); +} + +// ── the end card β€” app icon + menuband.app ───────────────────────────────── +function drawEndCard(t) { + const e = easeOut(clamp01((t - (SING1 + 1.0)) / 0.7)); + if (e <= 0) return; + const cw = W * 0.78, chh = H * 0.40; + const cx = (W - cw) / 2, cy = H * 0.44 - chh / 2 + (1 - e) * H * 0.35; + ctx.save(); ctx.globalAlpha = e; + ctx.shadowColor = "rgba(0,0,0,0.45)"; ctx.shadowBlur = 60; ctx.shadowOffsetY = 24; + roundRect(ctx, cx, cy, cw, chh, 40); ctx.fillStyle = "rgba(250,249,253,0.99)"; ctx.fill(); + ctx.shadowColor = "transparent"; + const litIcon = new Set(litAt(lead, t).map((mm) => ((mm % 12) + 12) % 12 % 5)); + const ipx = 330; + drawIcon(ctx, W / 2 - ipx / 2, cy + 40, ipx, litIcon); + text(ctx, "menuband.app", W / 2, cy + 450, 88, INK, 800); + text(ctx, "free on the Mac App Store", W / 2, cy + 540, 44, "rgba(60,50,80,0.9)", 700); + text(ctx, "type it yourself", W / 2, cy + 610, 40, "rgba(60,50,80,0.75)", 600); + ctx.restore(); +} + +function drawFrame(t) { + drawDesktop(ctx); + const dt = 1 / FPS; + + const hero = heroRect(t); + const hRect = drawStrip(ctx, rig, litAt(lead, t, 0.3), hero.x, hero.y, hero.w); + for (const n of onsetsBetween(t - dt, t)) { + particles.spawnNote(stripKeyX(rig, n.midi, hRect), hRect.y + hRect.h + 6, + stripKeyColor(rig, n.midi), true); + } + + // kicker under the strip β€” it has its say during the spoken frame, then + // clears the stage for the big teaching keycap + const ka = easeOut(clamp01((t - 0.9) / 0.6)) * + (1 - easeOut(clamp01((t - (SING0 - 0.5)) / 0.5))); + if (ka > 0) { + ctx.save(); ctx.globalAlpha = ka; + text(ctx, "your keyboard knows the C scale", W / 2, H * 0.245, 52, INK, 800); + ctx.restore(); + } + + drawKeycap(t); + drawChart(t); + drawEndCard(t); + + particles.stepAndDraw(dt); + vignette(ctx); + karaoke.draw(ctx, t); +} + +await renderVideo({ + canvas, audioPath: `${OUT}/${SLUG}${VAR}.mp3`, outPath: `${OUT}/base-${SLUG}${VAR}.mp4`, + total: TOTAL, drawFrame, label: `menuband scales sim${VAR}`, +}); +writeMeta(`${SLUG}${VAR}`, TOTAL, SCENES); diff --git a/pop/menuband/bin/sing-jingle.mjs b/pop/menuband/bin/sing-jingle.mjs index f5e36e666..da086b0b4 100644 --- a/pop/menuband/bin/sing-jingle.mjs +++ b/pop/menuband/bin/sing-jingle.mjs @@ -1,6 +1,30 @@ #!/usr/bin/env node // sing-jingle.mjs β€” jeffrey SINGS the Menu Band campaign jingles. // +// v6 β€” the REGISTER round + the SCALES teaching singalong: +// Β· REGISTER β€” `--register ` (default +12) lifts every line +// that many semitones ABOVE the engine's minimal-|shift| octave fit +// (jeffrey: "I could be higher octave?"). The formant envelope is never +// touched (no kermit by construction) and the goalpost conformance is +// register-aware (f0-linked bands widen/shift with the lift; duration/ +// energy/click gates unchanged). Per-line FALLBACK LADDER: if a line +// fails the WER gate at the asked register it re-renders at +7 then 0 +// and the most intelligible take wins (ties prefer the higher register); +// fallbacks are reported per line. +// Β· SCALES (menuband-scales) β€” a spoken frame around a sung two-octave +// notepat letter ladder (c d e f g a b h i j k l m n up, back down; +// h..n = notepat's second octave). Spoken lines are placed verbatim +// (natural TTS, unpitched, level-matched to the sung lines) with +// whisper word timings for captions; the ladder is explicit-mode with +// absolute pitches C3..B4 (register 0, octave_opt off β€” the ladder IS +// the register) and letter-name lyrics ("see dee ee …") for curated +// IPA. WER scoring letter-folds both sides so "C-D-E" == "see dee ee". +// Β· Engine fixes this round (spinging/lib/sing_line_world.py R6Β·2-4): +// final unstressed syllables stop starving ("diminished" β†’ "deman"), +// real ~100 ms phrase-boundary silence before phrase-initial fricatives +// ("keys. Sus" β†’ "kisses"), phrase-medial onset bursts sit prouder +// ("control" β†’ "Troll"). +// // v5 β€” the DICTION round: consonant time-stretching the way trained choirs // handle it (round 4's whisper gate was failing on swallowed consonants): // Β· STRETCHED CONSONANTS β€” the engine (spinging/lib/sing_line_world.py) @@ -102,6 +126,12 @@ const SR = 48_000; // Round 3 defaults to a ~0.875 lock β€” "drift more into the perfect harmony". const hIdx = process.argv.indexOf("--harmony"); const HARMONY = hIdx > 0 ? parseFloat(process.argv[hIdx + 1]) : 0.875; +// --register: semitones ABOVE the minimal-shift octave fit (v6 default +12 β€” +// "I could be higher octave?"). Lines that fail the WER gate up there fall +// back down the ladder (+7, then 0) and the best take wins. +const rIdx = process.argv.indexOf("--register"); +const REGISTER = rIdx > 0 ? parseInt(process.argv[rIdx + 1], 10) : 12; +const REGISTER_FALLBACKS = [7, 0]; const QA_PASSES = 3; // re-render budget per line (percentile gate) const CLARITY_PASSES = 2; // extra re-renders if the whisper WER gate fails const BRIDGE_MAX_S = 0.45; // keep in sync with sing_line_world.BRIDGE_MAX_S @@ -183,7 +213,28 @@ const LYRICS = { ["keys", [[18.8, 1.6, 48]]]] }, ], }, + // v6 β€” the SCALES teaching singalong. A spoken frame around the notepat + // two-octave letter ladder; timings + pitches come from the shared + // menuband-scales.score.json sidecar (render-jingles.mjs), so buildLines + // assembles the words there. `register: 0` β€” the ladder places its own + // absolute pitches (C3..B4, bright top, unstrained); octave_opt off. + "menuband-scales": { + mode: "scales", register: 0, + intro: "Here's how to type out the C scale.", + run: "C, D, E, F, G, A, B. Now sing it!", + asc: "C. D. E. F. G. A. B. H. I. J. K. L. M. N.", + desc: "M, L, K, J, I, H, B, A, G, F, E, D, C!", + outro: "Wanna type it yourself? Try Menu Band.", + }, +}; + +// notepat letter β†’ letter-NAME word (the lyric the ladder is sung on β€” +// real dictionary words so pronounce.mjs serves curated GenAm IPA). +const LETTER_WORDS = { + a: "ay", b: "bee", c: "see", d: "dee", e: "ee", f: "eff", g: "gee", + h: "aitch", i: "eye", j: "jay", k: "kay", l: "ell", m: "em", n: "en", }; +const LETTER_NAME_SET = new Set(Object.values(LETTER_WORDS)); // ── helpers ──────────────────────────────────────────────────────────────── // Percentile-gate feedback: nudge the engine's tweak knobs toward the @@ -357,6 +408,9 @@ const HOMOPHONES = { won: "one", banned: "band", cord: "chord", cords: "chords", suss: "sus", knew: "new", oar: "or", ore: "or", write: "right", rite: "right", }; +// casual contractions whisper expands ("wanna" β†’ "want to") β€” canonicalize +// BOTH sides to the expansion so the outro CTA scores verbatim +const CONTRACTIONS = { wanna: ["want", "to"], gonna: ["going", "to"], gotta: ["got", "to"] }; function normTokens(text) { const rough = String(text).toLowerCase() .replace(/[β€™β€˜]/g, "'") @@ -366,10 +420,25 @@ function normTokens(text) { const out = []; for (const w of rough) { if (/^\d+$/.test(w)) out.push(...numberToWords(parseInt(w, 10))); + else if (CONTRACTIONS[w]) out.push(...CONTRACTIONS[w]); else out.push(w.replace(/'/g, "")); } return out.map((w) => HOMOPHONES[w] || w); } + +// v6 letter folding (the scales ladder): "C-D-E" and "see dee ee" must score +// as the same thing. Single tokens a–n become their letter names; welded +// letter runs ("cde") split first. Only applied on lettered lines. +function letterFoldTokens(toks) { + const out = []; + for (const t of toks) { + if (/^[a-n]$/.test(t)) out.push(LETTER_WORDS[t]); + else if (/^[a-n]{2,}$/.test(t) && !LETTER_NAME_SET.has(t) && !HOMOPHONES[t]) + out.push(...t.split("").map((c) => LETTER_WORDS[c])); + else out.push(t); + } + return out; +} const tokEq = (a, b) => a === b || (a.length >= 3 && b.length >= 3 && editDist(a, b) <= 1); // merge hyp tokens that SPLIT one ref token ("full screen" vs "fullscreen", @@ -425,9 +494,11 @@ function werScore(refToks, hypToks) { } return m ? prev[n] / m : 0; } -function evalWER(refText, hypText) { - const ref = normTokens(refText); - const hyp = dewedge(ref, rewedge(ref, normTokens(hypText))); +function evalWER(refText, hypText, { lettered = false } = {}) { + let ref = normTokens(refText); + let hypRaw = normTokens(hypText); + if (lettered) { ref = letterFoldTokens(ref); hypRaw = letterFoldTokens(hypRaw); } + const hyp = dewedge(ref, rewedge(ref, hypRaw)); const wer = +werScore(ref, hyp).toFixed(3); const missing = ref.filter((w) => w.length >= 3 && !STOPWORDS.has(w) && !hyp.some((h) => tokEq(w, h))); @@ -504,6 +575,30 @@ function presplitHeard(scoreWords, heard) { return out; } +// v6 (scales): before alignment, fold whisper's heard letters onto the +// spelled letter-name score words β€” "C" β†’ "see", and welded runs ("CDE", +// or dashes normed away: "c-d-e" β†’ "cde") split char-proportionally into +// one window per letter. Only for lettered lines. +function letterizeHeard(heard) { + const out = []; + for (const h of heard) { + const t = norm(h.text); + if (/^[a-n]$/.test(t)) { out.push({ ...h, text: LETTER_WORDS[t] }); continue; } + if (/^[a-n]{2,}$/.test(t) && !LETTER_NAME_SET.has(t)) { + const span = h.toMs - h.fromMs; + let t0 = h.fromMs; + for (let c = 0; c < t.length; c++) { + const w = span / t.length; + out.push({ text: LETTER_WORDS[t[c]], fromMs: Math.round(t0), toMs: Math.round(t0 + w) }); + t0 += w; + } + continue; + } + out.push(h); + } + return out; +} + // Whisper stamps the words at a clip's hard end zero-width and buries their // audio inside the PREVIOUS window ("Type" 110-1020 actually holds "type to // play"). Clamp every window into the audio, then repair each RUN of @@ -651,6 +746,30 @@ function buildLines(slug) { }); lines.push({ tts: line.tts, words }); } + } else if (spec.mode === "scales") { + // spoken frame + the sung letter ladder, all timed by the shared + // score sidecar (render-jingles.mjs writes it; sim-scales reads it too) + const sc = JSON.parse(readFileSync(`${OUT}/${slug}.score.json`, "utf8")); + const asc = sc.ladder.filter((n) => n.dir === "up"); + const desc = sc.ladder.filter((n) => n.dir === "down"); + // notationText: the spelled letter names WITH commas β€” notation.mjs + // matches its tokens to the lyric words, so every letter becomes its own + // phrase and the engine ARTICULATES the ladder (tiny onset gaps, no + // melisma blur) instead of welding 14 letters into one glissando. + const sungLine = (tts, notes) => ({ + tts, lettered: true, + notationText: notes.map((n) => LETTER_WORDS[n.letter]).join(", ") + ".", + words: notes.map((n) => ({ + w: LETTER_WORDS[n.letter], display: n.letter, + slots: [{ t: n.t, dur: n.dur, midi: n.vocal }], + })), + }); + lines.push({ tts: spec.intro, spoken: true, t: sc.spoken.intro.t, words: [] }); + // lettered: the run's welded "C-D-E-F-G-A-B" caption splits into letters + lines.push({ tts: spec.run, spoken: true, lettered: true, t: sc.spoken.run.t, words: [] }); + lines.push(sungLine(spec.asc, asc)); + lines.push(sungLine(spec.desc, desc)); + lines.push({ tts: spec.outro, spoken: true, t: sc.spoken.outro.t, words: [] }); } else { for (const line of spec.lines) { lines.push({ @@ -659,13 +778,19 @@ function buildLines(slug) { }); } } - return { lines, durationSec: score.durationSec }; + return { lines, durationSec: score.durationSec, spec }; } // ── main ─────────────────────────────────────────────────────────────────── async function singOne(slug) { console.log(`\nβ–Έ ${slug} β€” jeffrey sings (line-continuous WORLD)`); - const { lines, durationSec } = buildLines(slug); + const { lines, durationSec, spec } = buildLines(slug); + // v6: register ladder β€” the asked lift first, then the kermit/strain + // fallbacks; a line stops falling as soon as it passes the WER gate. + const specRegister = spec.register ?? REGISTER; + const regLadder = [specRegister, ...REGISTER_FALLBACKS.filter((r) => r < specRegister)]; + const OCT_OPT = spec.mode !== "scales"; // the ladder places its own pitches + console.log(` register +${specRegister} (fallback ladder ${regLadder.join(" β†’ ")})`); const dir = `${OUT}/sung/${slug}`; mkdirSync(`${dir}/words`, { recursive: true }); if (!existsSync(WHISPER_MODEL)) throw new Error(`whisper model missing: ${WHISPER_MODEL}`); @@ -677,6 +802,8 @@ async function singOne(slug) { const qaLines = []; const lineSpans = []; // absolute spans for the per-line stem re-transcribe const consSpans = []; // v5: absolute consonant spans β†’ extra bed duck + const spokenPlacements = []; // v6: spoken lines placed after level match + const regFallbacks = []; // v6: per-line register fallbacks (reported) if (!existsSync(GOALPOSTS)) { throw new Error(`goalposts missing: ${GOALPOSTS} β€” build with spinging goalposts`); } @@ -691,6 +818,63 @@ async function singOne(slug) { const mp3 = `${dir}/line-${li}-${hash}.mp3`; await ttsLine(line.tts, mp3); + // ── v6 spoken lines (the scales frame): natural TTS placed verbatim at + // its absolute time β€” no engine, no pitch; whisper word timings become + // the caption windows and the transcript must be essentially verbatim ── + if (line.spoken) { + const w16s = mp3.replace(/\.mp3$/, "-16k.wav"); + if (!existsSync(w16s)) { + sh("ffmpeg", ["-y", "-v", "error", "-i", mp3, "-ac", "1", "-ar", "16000", w16s]); + } + const wjs = mp3.replace(/\.mp3$/, "-words"); + if (!existsSync(`${wjs}.json`)) { + sh("whisper-cli", ["-m", WHISPER_MODEL, "-f", w16s, "-ml", "1", "-oj", "-ojf", "-of", wjs], + { stdio: ["ignore", "ignore", "pipe"] }); + } + const heard = wordsFromWhisper(`${wjs}.json`); + const { audio: spokenAudio } = decodeAudioMono(mp3, SR); + spokenPlacements.push({ audio: spokenAudio, at: Math.floor(line.t * SR) }); + // caption words: strip trailing commas/periods; a lettered spoken line + // (the "C, D, E…" run) splits welded letter runs into single capitals + let capWords = heard; + if (line.lettered) { + capWords = []; + for (const h of heard) { + const tN = norm(h.text); + if (/^[a-n]{2,}$/.test(tN) && !LETTER_NAME_SET.has(tN)) { + const span = (h.toMs - h.fromMs) / tN.length; + let t0 = h.fromMs; + for (const c of tN) { + capWords.push({ text: c.toUpperCase(), fromMs: Math.round(t0), toMs: Math.round(t0 + span) }); + t0 += span; + } + } else if (/^[a-n]$/.test(tN)) capWords.push({ ...h, text: tN.toUpperCase() }); + else capWords.push(h); + } + } + for (const h of capWords) { + const text = h.text.replace(/[.,;:]+$/, ""); + if (!text) continue; + sungWords.push({ + text, fromMs: Math.round(line.t * 1000 + h.fromMs), + toMs: Math.round(line.t * 1000 + h.toMs), line: li, spoken: true, + }); + } + const tx = whisperTranscribe(mp3); + const wr = evalWER(line.tts, tx, { lettered: !!line.lettered }); + console.log(` line ${li} (spoken): "${line.tts}" β†’ heard "${tx}" Β· WER ${wr.wer} ${wr.pass ? "βœ“" : "βœ—"}`); + qaLines.push({ + line: li, text: line.tts, spoken: true, + whisper: { transcript: tx, wer: wr.wer, missing: wr.missing, pass: wr.pass }, + }); + lineSpans.push({ + li, text: line.tts, lettered: !!line.lettered, spoken: true, + t0: Math.max(0, line.t - 0.1), + t1: Math.min(durationSec, line.t + spokenAudio.length / SR + 0.15), + }); + continue; + } + // 16k mono for whisper (unpadded β€” trailing silence makes whisper smear // word timestamps into it; repairWindows handles the zero-width final // word it stamps at a hard file end), 48k mono for the WORLD engine @@ -710,8 +894,10 @@ async function singOne(slug) { const { audio: lineAudio } = decodeAudioMono(mp3, SR); const lineLen = lineAudio.length / SR; const mapWords = line.words.map((w) => w.w); + let heardRaw = expandDigitWords(wordsFromWhisper(`${wj}.json`)); + if (line.lettered) heardRaw = letterizeHeard(heardRaw); // v6 scales const heard = presplitHeard(mapWords, - rescaleHeard(expandDigitWords(wordsFromWhisper(`${wj}.json`)), lineAudio, lineLen * 1000)); + rescaleHeard(heardRaw, lineAudio, lineLen * 1000)); const windows = repairWindows(alignWords(mapWords, heard), lineLen * 1000, lineAudio); console.log(` line ${li}: "${line.tts}" Β· whisper heard "${heard.map((h) => h.text).join(" ")}"`); @@ -720,7 +906,7 @@ async function singOne(slug) { // punctuation + melody rests β‰₯ 0.4 s; the engine bridges legato inside a // phrase and breathes only at its edges). Phonemes from curated US IPA. const score = await buildLineScore({ - text: line.tts, + text: line.notationText ?? line.tts, // v6: scales phrases per letter words: line.words.map((w) => ({ w: w.w, slots: w.slots })), }); const scorePath = `${dir}/words/line-${li}-score.json`; @@ -763,7 +949,8 @@ async function singOne(slug) { slots, hardEnd: +tEnd.toFixed(4), phraseStart: phraseStartOf[wi], }); sungWords.push({ - text: word.w, fromMs: Math.round(tStart * 1000), toMs: Math.round(tEnd * 1000), line: li, + text: word.display ?? word.w, // v6: the scales karaoke shows LETTERS + fromMs: Math.round(tStart * 1000), toMs: Math.round(tEnd * 1000), line: li, }); } @@ -789,7 +976,7 @@ async function singOne(slug) { // the reference plateau-drift band on pass 1 const tweaks = { drift_scale: 1.6, glide_scale: 1, vib_depth_scale: 1, beta_scale: 1, air_scale: 1, cons_stretch_scale: 1 }; // v5: consonant diction stretch (engine caps at 2.5Γ—) - const renderPlan = () => { + const renderPlan = (register) => { const plan = { line_wav: w48, out_wav: outWav, @@ -801,7 +988,8 @@ async function singOne(slug) { f0_floor: 60, f0_ceil: 300, // jeffrey's real range β€” de-kermit // (the octave optimizer weighs words by voiced evidence instead of // narrowing this ceiling β€” the TTS exclamations genuinely peak high) - octave_opt: true, choir: true, + octave_opt: OCT_OPT, choir: true, + register, // v6: applied AFTER the octave fit tweaks, words: planWords, }; @@ -823,7 +1011,7 @@ async function singOne(slug) { let lastIsBest = false; const consider = (st) => { const tx = whisperTranscribe(outWav); - const wr = evalWER(line.tts, tx); + const wr = evalWER(line.tts, tx, { lettered: !!line.lettered }); const cf = confOf(st); const wins = best.stats === null || (wr.wer < best.res.wer && (cf || !best.conf)) @@ -838,35 +1026,49 @@ async function singOne(slug) { }; let stats = {}; let passes = 0; - for (let pass = 1; pass <= QA_PASSES; pass++) { - passes = pass; - stats = renderPlan(); + let clarityPasses = 0; + const registersTried = []; + // ── v6: register ladder β€” full QA + clarity budget per register; a line + // stops falling as soon as the whisper gate passes. consider() keeps the + // best take across ALL registers (ties prefer the earlier = higher one). + for (const reg of regLadder) { + registersTried.push(reg); + Object.assign(tweaks, { drift_scale: 1.6, glide_scale: 1, vib_depth_scale: 1, + beta_scale: 1, air_scale: 1, cons_stretch_scale: 1 }); + for (let pass = 1; pass <= QA_PASSES; pass++) { + passes++; + stats = renderPlan(reg); + if (stats.error) break; + const clean = stats.clicks && stats.clicks.clicks === 0 && stats.clicks.flux_spikes === 0; + if ((!stats.conformance || stats.conformance._pass) && clean) { consider(stats); break; } + consider(stats); + if (pass === QA_PASSES) break; + adjustTweaks(tweaks, stats.conformance); + console.log(` ↻ pass ${pass} (reg +${reg}): out of band β€” retweak ` + + Object.entries(tweaks).map(([k, v]) => `${k}=${v.toFixed(2)}`).join(" ")); + } if (stats.error) break; - const clean = stats.clicks && stats.clicks.clicks === 0 && stats.clicks.flux_spikes === 0; - if ((!stats.conformance || stats.conformance._pass) && clean) { consider(stats); break; } - consider(stats); - if (pass === QA_PASSES) break; - adjustTweaks(tweaks, stats.conformance); - console.log(` ↻ pass ${pass}: out of band β€” retweak ` + - Object.entries(tweaks).map(([k, v]) => `${k}=${v.toFixed(2)}`).join(" ")); + for (let cp = 1; cp <= CLARITY_PASSES && !best.res.pass; cp++) { + clarityPasses++; + tweaks.air_scale *= 0.6; + tweaks.vib_depth_scale *= 0.7; + // v5: more diction, not just less air β€” lean the stretch in harder + tweaks.cons_stretch_scale = Math.min(1.3, tweaks.cons_stretch_scale * 1.15); + console.log(` ↻ clarity pass ${cp} (reg +${reg}): WER ${best.res.wer} ` + + `(heard "${best.transcript}") β€” re-render with less air/vibrato, more stretch`); + const st = renderPlan(reg); + if (st.error) break; + consider(st); + } + if (best.res?.pass) break; + if (reg !== regLadder[regLadder.length - 1]) { + console.log(` ↧ register +${reg} missed the WER gate (${best.res?.wer}) β€” falling back`); + } } if (stats.error) { report.push({ slug, line: li, word: "(line)", note: stats.error }); continue; } - let clarityPasses = 0; - for (let cp = 1; cp <= CLARITY_PASSES && !best.res.pass; cp++) { - clarityPasses = cp; - tweaks.air_scale *= 0.6; - tweaks.vib_depth_scale *= 0.7; - // v5: more diction, not just less air β€” lean the stretch in harder - tweaks.cons_stretch_scale = Math.min(1.3, tweaks.cons_stretch_scale * 1.15); - console.log(` ↻ clarity pass ${cp}: WER ${best.res.wer} ` + - `(heard "${best.transcript}") β€” re-render with less air/vibrato, more stretch`); - const st = renderPlan(); - if (st.error) break; - consider(st); - } if (!lastIsBest) { // the final re-render didn't win β€” restore the best-scoring take copyFileSync(bestWav, outWav); @@ -875,10 +1077,18 @@ async function singOne(slug) { let werRes = best.res; let transcript = best.transcript; stats = best.stats; + // v6: which register actually won β€” a drop below the asked one is a + // reported fallback (kermit/strain safety valve) + const finalReg = best.stats?.register ?? regLadder[0]; + if (finalReg !== specRegister) { + regFallbacks.push({ slug, line: li, text: line.tts, + asked: specRegister, used: finalReg, wer: best.res.wer }); + console.log(` ‡ line ${li} register fallback: +${specRegister} β†’ +${finalReg}`); + } // v5 diagnostic: transcribe the choir-less LEAD stem too β€” separates // diction gains from choir masking in the QA sidecar const leadTranscript = whisperTranscribe(leadWav); - const leadWer = evalWER(line.tts, leadTranscript); + const leadWer = evalWER(line.tts, leadTranscript, { lettered: !!line.lettered }); // v5: absolute consonant spans drive the bed's extra diction duck for (const [a, b] of stats.consonant_spans || []) { consSpans.push([lineT0 + a, lineT0 + b]); @@ -914,6 +1124,8 @@ async function singOne(slug) { console.log(` lead-only diagnostic: WER ${leadWer.wer} β€” heard "${leadTranscript}"`); qaLines.push({ line: li, text: line.tts, passes, clarityPasses, tweaks, + register: finalReg, registersTried, + registerFallback: finalReg !== specRegister, lineTranspose: stats.line_transpose, beta: stats.beta, harmony: HARMONY, consStretchScale: stats.cons_stretch_scale, f0JumpMaxCents: stats.f0_jump_max_cents, f0JumpP95Cents: stats.f0_jump_p95_cents, @@ -925,7 +1137,7 @@ async function singOne(slug) { conformance: stats.conformance, clicks: stats.clicks, }); lineSpans.push({ - li, text: line.tts, + li, text: line.tts, lettered: !!line.lettered, t0: Math.max(0, planWords[0].slots[0].t - 0.15), t1: Math.min(durationSec, planWords[planWords.length - 1].hardEnd + 0.3), }); @@ -937,6 +1149,28 @@ async function singOne(slug) { for (let i = 0; i < sung.length && at + i < master.length; i++) master[at + i] += sung[i]; } + // v6: place the spoken lines now, level-matched to the sung material β€” + // active-frame RMS of the assembled sung vocal vs each spoken take, spoken + // sitting just under the singing so the frame never shouts over the song + if (spokenPlacements.length) { + const rmsActive = (buf) => { + let e = 0, n = 0; + for (let i = 0; i < buf.length; i++) { + const v = buf[i]; + if (Math.abs(v) > 1e-3) { e += v * v; n++; } + } + return n ? Math.sqrt(e / n) : 0; + }; + const sungRms = rmsActive(master); + for (const p of spokenPlacements) { + const own = rmsActive(p.audio); + const g = sungRms > 0 && own > 0 ? Math.min(1.6, (sungRms * 0.9) / own) : 1; + for (let i = 0; i < p.audio.length && p.at + i < master.length; i++) { + master[p.at + i] += p.audio[i] * g; + } + } + } + // normalize the vocal, write it, then MASTER the mix let peak = 0; for (let i = 0; i < master.length; i++) peak = Math.max(peak, Math.abs(master[i])); @@ -1020,7 +1254,7 @@ async function singOne(slug) { console.log(` whisper end-to-end on the vocal stem …`); const stemLines = lineSpans.map((s) => { const tx = whisperTranscribeSpan(vocalWav, s.t0, s.t1, `${dir}/words/stem-line-${s.li}`); - const w = evalWER(s.text, tx); + const w = evalWER(s.text, tx, { lettered: !!s.lettered }); const mark = w.pass ? "βœ“" : "βœ—"; console.log(` ${mark} L${s.li} WER ${w.wer} "${s.text}" β†’ heard "${tx}"`); return { line: s.li, text: s.text, transcript: tx, wer: w.wer, missing: w.missing, pass: w.pass }; @@ -1030,8 +1264,9 @@ async function singOne(slug) { writeFileSync(`${OUT}/${slug}.words.sung.json`, JSON.stringify(sungWords, null, 2)); writeFileSync(`${OUT}/${slug}-sung-qa.json`, JSON.stringify({ - slug, harmony: HARMONY, engine: "spinging/lib/sing_line_world.py (round 5)", + slug, harmony: HARMONY, engine: "spinging/lib/sing_line_world.py (round 6)", goalposts: GOALPOSTS, + register: { asked: specRegister, ladder: regLadder, fallbacks: regFallbacks }, gates: { werMax: WER_GATE, voicingContinuityMin: CONTINUITY_GATE }, consDuckSpans: consSpans.length, pronunciationSources: { ...sourceCounts }, @@ -1044,6 +1279,13 @@ async function singOne(slug) { console.log(`βœ“ ${mix}`); console.log(`βœ“ ${OUT}/${slug}.words.sung.json Β· ${sungWords.length} words`); console.log(`βœ“ ${OUT}/${slug}-sung-qa.json`); + for (const f of regFallbacks) { + report.push({ slug, line: f.line, word: "(register)", + info: `register fallback +${f.asked} β†’ +${f.used} (best WER ${f.wer}) β€” "${f.text}"` }); + } + if (!regFallbacks.length && specRegister !== 0) { + console.log(` register: every line held +${specRegister}`); + } return report; } diff --git a/spinging/README.md b/spinging/README.md index 18a2730e4..e2203993c 100644 --- a/spinging/README.md +++ b/spinging/README.md @@ -82,6 +82,44 @@ re-transcribed per line and the final mix once more β€” transcripts land verbatim in `out/-sung-qa.json`. QA is statistical AND machine-read, not ear-only. +Changelog β€” round 6 (2026-07, the REGISTER round + the SCALES singalong β€” +"I could be higher octave?"): +- **Register lift** β€” `plan.register` (semitones) applies AFTER the per-line + minimal-|shift| octave fit; sing-jingle's `--register` (default +12 this + round) drives it with a per-line fallback ladder (+12 β†’ +7 β†’ 0): a line + that misses the whisper gate re-renders lower and the most intelligible + take wins (ties prefer the higher register); fallbacks land in the QA + sidecar (`register.fallbacks`). Conformance is register-aware + (`vocal_shapes.conformance(register=…)`): the f0-linked bands (glide/ + drift/release/vib cents + hf_ratio) widen 35 %/octave and hf_ratio shifts + with the harmonic comb (Γ—2^(R/24)); duration/energy/click gates unchanged. +- **R6Β·2 final unstressed syllables stop starving** ("diminished" β†’ "deman"): + word-final unstressed vowels get a minimum-duration floor (0.14 s, borrowed + by anticipating the note into the preceding stressed vowel) and word-final + coda CLUSTERS (β‰₯2 phones) may articulate into the phrase gap (extension cap + 0.05 β†’ 0.18 s). Chords L4 now closes the word ("demand" β€” the coda exists; + the full /Κƒt/ remains whisper-hostile). +- **R6Β·3 phrase-boundary silence** ("keys. Sus" β†’ "kisses"): phrase-initial + FRICATIVE onsets get a real ~100 ms near-zero gap (verified in stats: + `gap_ms 100`, onset 115 ms) instead of the 22 ms glottal dip. +- **R6Β·4 phrase-medial onset prominence** ("control" β†’ "Troll"): raw plosive + composites in a phrase-medial word's onset ride an extra 1.25Γ— on top of + RAW_BOOST. +- **The scales reel** (pop/menuband/bin/sing-jingle.mjs `menuband-scales`) β€” + spoken lines placed verbatim at absolute times (level-matched to the sung + material, whisper word timings β†’ captions; all three spoken lines + transcribe at WER 0) around the sung notepat letter ladder: explicit + pitches C3..B4, `octave_opt` off, per-letter phrases via a notation-only + text (letter names + commas) so the ladder articulates instead of + smearing into one glissando; letter-name lyrics ("see dee ee …", curated + /eΙͺ/ "ay" + /aΙͺ/ "eye" in pronounce.mjs β€” CURATED wins over cache) and + letter-folded WER scoring ("C-D-E" == "see dee ee"). +- Result: every line CAN now ride an octave above round 5 (13/19 sung + campaign lines hold +12, 3 land +7, 3 fall to +0 β€” all reported); scales + descending ladder hits WER 0.154 (11/13 letters) with the ascending run + flagged honestly (whisper hears sustained sung letter-vowels as melody); + clicks 0 and conformance green on every shipped line. + Changelog β€” round 5 (2026-07, the DICTION round β€” consonant time-stretching the way trained choirs handle it; round 4's whisper gate was failing on swallowed consonants, not on the singing): diff --git a/spinging/cache/pronounce.json b/spinging/cache/pronounce.json index 784f00ff0..66d915300 100644 --- a/spinging/cache/pronounce.json +++ b/spinging/cache/pronounce.json @@ -383,5 +383,103 @@ "accent": "US", "v": 4, "fetchedAt": "2026-07-19T16:14:52.422Z" + }, + "see": { + "ipa": "/ˈsi/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.007Z" + }, + "dee": { + "ipa": "/ˈdiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.170Z" + }, + "ee": { + "ipa": "/ˈiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:25.438Z" + }, + "eff": { + "ipa": "/Ι›f/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:26.894Z" + }, + "gee": { + "ipa": "/dΝ‘Κ’iː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:27.120Z" + }, + "ay": { + "ipa": "/aΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:27.344Z" + }, + "bee": { + "ipa": "/ˈbiː/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.055Z" + }, + "aitch": { + "ipa": "/ˈeΙͺtΝ‘Κƒ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.250Z" + }, + "eye": { + "ipa": "/ˈaː/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.463Z" + }, + "jay": { + "ipa": "/ˈdΝ‘Κ’eΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.644Z" + }, + "kay": { + "ipa": "/keΙͺ/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:33.806Z" + }, + "ell": { + "ipa": "/ΛˆΙ›l/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:35.346Z" + }, + "em": { + "ipa": "/ΛˆΙ›m/", + "source": "wiktionary", + "accent": "untagged-usSafe", + "v": 4, + "fetchedAt": "2026-07-19T20:06:35.585Z" + }, + "en": { + "ipa": "/ΛˆΙ›n/", + "source": "wiktionary", + "accent": "US", + "v": 4, + "fetchedAt": "2026-07-19T20:06:36.065Z" } } \ No newline at end of file diff --git a/spinging/lib/pronounce.mjs b/spinging/lib/pronounce.mjs index c4133454e..afc4f82f3 100644 --- a/spinging/lib/pronounce.mjs +++ b/spinging/lib/pronounce.mjs @@ -264,6 +264,15 @@ const BUILTIN = { a: "/Ι™/", the: "/Γ°Ι™/", }; +// Round 6 β€” words where the dictionary's PRIMARY entry is the wrong +// homograph for singing (the scales letter ladder): "ay" resolves to the +// aye-vote /aΙͺ/ where the letter A is /eΙͺ/, and Wiktionary's "eye" line +// parses to /ˈaː/. CURATED wins over every source, the cache included. +const CURATED = { + ay: "/eΙͺ/", // the letter A + eye: "/aΙͺ/", // the letter I +}; + // ── cache + public API ───────────────────────────────────────────────────── function loadCache() { try { return JSON.parse(readFileSync(CACHE_FILE, "utf8")); } catch { return {}; } @@ -273,7 +282,7 @@ function saveCache(c) { writeFileSync(CACHE_FILE, JSON.stringify(c, null, 1)); } -export const sourceCounts = { wiktionary: 0, espeak: 0, builtin: 0, cache: 0 }; +export const sourceCounts = { wiktionary: 0, espeak: 0, builtin: 0, cache: 0, curated: 0 }; // Cache format version β€” bump to invalidate every cached word (round 4: the // per-line accent read + usSafe screen obsoleted every round-3 entry). @@ -289,6 +298,11 @@ async function resolveWord(word) { export async function pronounce(rawWord) { const word = rawWord.toLowerCase().replace(/[^a-z']/g, ""); + if (CURATED[word]) { + sourceCounts.curated++; + return { word, ipa: CURATED[word], source: "curated", + syllables: syllabify(tokenizeIPA(CURATED[word])) }; + } const cache = loadCache(); let entry = cache[word]; if (entry && entry.v === CACHE_V) sourceCounts.cache++; diff --git a/spinging/lib/sing_line_world.py b/spinging/lib/sing_line_world.py index 9ea966869..df1b08efb 100644 --- a/spinging/lib/sing_line_world.py +++ b/spinging/lib/sing_line_world.py @@ -1,8 +1,34 @@ #!/usr/bin/env python3 """ -sing_line_world.py β€” spinging's line-continuous singing engine (round 5). - -What round 5 adds β€” CONSONANT TIME-STRETCHING, the way trained choirs +sing_line_world.py β€” spinging's line-continuous singing engine (round 6). + +What round 6 adds β€” REGISTER + the diction starve fixes: + + R6Β·1 REGISTER β€” plan.register (semitones) lifts every target AFTER the + per-line minimal-|shift| octave fit: the fit still finds where the + spoken take naturally sits, then the whole line rides up (jeffrey: + "I could be higher octave?"). The formant envelope is untouched (no + kermit by construction) and the goalpost conformance becomes + register-aware (vocal_shapes.conformance widens/shifts the f0-linked + bands; duration/energy/click gates unchanged). The caller may retry + a line at lower registers if it sounds strained (fallback ladder). + R6Β·2 FINAL UNSTRESSED SYLLABLES STOP STARVING ("diminished" β†’ "deman"): + a word-final unstressed syllable's vowel gets a minimum-duration + floor (borrowed from the preceding stressed vowel by anticipating + the note), and word-final coda CLUSTERS (β‰₯2 phones) may articulate + into the phrase gap (extension cap 0.05 β†’ 0.18 s) so /Κƒt/-style + codas render at full value instead of vanishing. + R6Β·3 PHRASE-BOUNDARY SILENCE ("keys. Sus" β†’ "kisses"): a phrase-initial + fricative onset now gets a real ~100 ms silence carved before it + (near-zero floor, not the 22 ms glottal dip) so the /s/ can't weld + backward onto the previous phrase's coda. + R6Β·4 PHRASE-MEDIAL ONSET PROMINENCE ("control" β†’ "Troll"): raw plosive + composites inside a phrase-medial word's onset get an extra boost on + top of RAW_BOOST β€” the /k/ burst stays legible against the already- + leveled vowels around it. (Consonant-span bed ducking already covers + these frames via cons_mask.) + +Round 5 (kept) β€” CONSONANT TIME-STRETCHING, the way trained choirs handle diction (the whisper round-trip gate was failing on swallowed consonants, not on the singing): @@ -136,9 +162,14 @@ MAX_ONSET_OUT_S = 0.34 # stretched onset ceiling MAX_CODA_OUT_S = 0.42 # stretched coda ceiling PLOSIVE_GAP_S = 0.022 # pre-plosive silence (the choir's glottal set-up) GAP_FLOOR_AMP = 0.06 # the gap dips to this, never digital zero +PHRASE_FRIC_GAP_S = 0.10 # R6Β·3 real silence before a phrase-initial fricative +PHRASE_FRIC_GAP_AMP = 0.02 # …and it dips near-zero (a true phrase breath) CONS_GAIN_DB = 3.0 # consonant prominence on the WORLD path RAW_BOOST = 1.5 # raw composite boost (plosive bursts; R4 was 1.4) +ONSET_RAW_EXTRA = 1.25 # R6Β·4 extra boost on phrase-medial onset bursts CODA_EXTEND_S = 0.05 # word-final codas may run past hardEnd at phrase ends +CODA_CLUSTER_EXTEND_S = 0.18 # R6Β·2 …coda CLUSTERS may run this far into the gap +FINAL_UNSTRESSED_VOWEL_S = 0.14 # R6Β·2 word-final unstressed vowel floor MAX_BREATH_S = 0.45 BRIDGE_MAX_S = 0.45 # intra-phrase gaps up to this sustain legato BRIDGE_DIP_AMP = 0.72 # shallow energy dip at a bridged word boundary @@ -769,6 +800,13 @@ def main(): for s in segs: for sl in s["w"]["slots"]: sl["midi"] += line_transpose + # R6Β·1: the register lift rides ON TOP of the minimal-shift fit β€” the fit + # finds where the spoken take naturally sits, the register is the ask. + register = int(plan.get("register", 0)) + if register: + for s in segs: + for sl in s["w"]["slots"]: + sl["midi"] += register # ── output timeline ──────────────────────────────────────────────────── line_t0 = float(plan["line_t0"]) @@ -791,6 +829,7 @@ def main(): force_voiced = np.zeros(out_n, dtype=bool) # R4Β·2 sung voiced consonants sp_gain = np.ones(out_n) # R4Β·1 shallow bridge energy dips cons_gain = np.ones(out_n) # R5Β·4 consonant prominence (sp amp) + raw_gain = np.ones(out_n) # R6Β·4 extra boost on onset raw bursts cons_mask = np.zeros(out_n, dtype=bool) # R5 consonant frames (choir gate) gap_env = np.ones(out_n) # R5Β·2 pre-plosive glottal-set-up dips onset_marks = [] # (onset_f, vowel_f) voiced-onset QA @@ -817,13 +856,15 @@ def main(): cons_g = db(CONS_GAIN_DB) - def place_plan(o_end, plan, midi, med): + def place_plan(o_end, plan, midi, med, raw_extra=1.0): """R5Β·1: place a diction plan so its output ENDS at frame o_end. Raw runs map 1:1 (integer source steps β†’ the raw composite path picks them up with true transients); stretched runs map fractionally and render through WORLD (noise for unvoiced fricatives, the pitch path - for voiced sonorants). Returns the plan's output start frame.""" + for voiced sonorants). raw_extra (R6Β·4) rides on RAW_BOOST for the + raw runs β€” phrase-medial onset bursts sit prouder. Returns the + plan's output start frame.""" o = o_end - sum(p[2] for p in plan) o_start = o for ra, rb, out, m in plan: @@ -834,6 +875,7 @@ def main(): if m != "pitch" and out == rb - ra: src_pos[idx] = ra + (idx - o) # 1:1 β†’ raw composite natural[idx] = True + raw_gain[idx] = raw_extra else: pos = (np.linspace(ra, max(ra, rb - 1) + 1e-6, out) if out > 1 else np.array([float(ra)])) @@ -869,10 +911,17 @@ def main(): exp_on = s["exp"][0]["phonemes"]["onset"] if s["exp"] else None # the glottal set-up: before every plosive/affricate, and at every # phrase-initial consonant (a stretched phrase-opening /s/ otherwise - # welds onto the previous phrase's coda β€” "keys. Sus" β†’ "kisses") - gap_s = PLOSIVE_GAP_S if (exp_on and ( - exp_on[0]["cls"] in ("plosive", "affricate") - or (wi > 0 and s["w"].get("phraseStart")))) else 0.0 + # welds onto the previous phrase's coda β€” "keys. Sus" β†’ "kisses"). + # R6Β·3: a 22 ms dip wasn't enough for phrase-initial FRICATIVES β€” + # whisper still heard the /z s/ weld β€” so those now get a real + # ~100 ms near-silent phrase breath instead. + gap_s, gap_amp = 0.0, GAP_FLOOR_AMP + if exp_on: + phrase_medial_start = wi > 0 and s["w"].get("phraseStart") + if phrase_medial_start and exp_on[0]["cls"] == "fricative": + gap_s, gap_amp = PHRASE_FRIC_GAP_S, PHRASE_FRIC_GAP_AMP + elif exp_on[0]["cls"] in ("plosive", "affricate") or phrase_medial_start: + gap_s = PLOSIVE_GAP_S oplan, out_f = diction_plan( s["onset"][0], s["onset"][1], exp_on, cls, voiced, hf_ratio, stretch_scale, min(MAX_ONSET_OUT_S, max(0.04, avail - gap_s)), @@ -880,7 +929,7 @@ def main(): onset_plans.append({"plan": oplan, "out_f": out_f, "out_s": out_f * FRAME_S, "gap_f": int(round(gap_s / FRAME_S)), - "gap_s": gap_s}) + "gap_s": gap_s, "gap_amp": gap_amp}) stats_words = [] for wi, s in enumerate(segs): @@ -933,13 +982,15 @@ def main(): s["_v_delay"] = v_delay onset_start = o_end_on if op["plan"]: - onset_start = place_plan(o_end_on, op["plan"], slots[0]["midi"], med_log) + onset_start = place_plan(o_end_on, op["plan"], slots[0]["midi"], med_log, + raw_extra=ONSET_RAW_EXTRA if wi > 0 else 1.0) if s["onset_voiced"] and o_end_on > onset_start: onset_marks.append((max(0, onset_start), min(out_n, o_end_on))) if op["gap_f"] > 0: ga = max(0, onset_start - op["gap_f"]) if onset_start > ga: - gap_env[ga:onset_start] = GAP_FLOOR_AMP + gap_env[ga:onset_start] = np.minimum( + gap_env[ga:onset_start], op["gap_amp"]) s["_onset_start"] = onset_start # R5Β·3: coda diction plan β€” stretched to full value, stealing time @@ -947,14 +998,19 @@ def main(): # to CODA_EXTEND_S past hardEnd (bridged codas already ride the # bridge into the next word's onset). nxt_seg = segs[wi + 1] if wi + 1 < len(segs) else None + exp_coda_ph = s["exp"][-1]["phonemes"]["coda"] if s["exp"] else None + # R6Β·2: a coda CLUSTER (β‰₯2 phones β€” "diminished"'s /Κƒt/) needs real + # articulation room; let it run further into the phrase gap than a + # single-consonant coda would. + ext_cap = CODA_CLUSTER_EXTEND_S if exp_coda_ph and len(exp_coda_ph) >= 2 \ + else CODA_EXTEND_S ext_s = 0.0 if nxt_seg is None: - ext_s = CODA_EXTEND_S + ext_s = ext_cap elif nxt_seg["w"].get("phraseStart") and bridge_from is None: nxt_onset_t = nxt_seg["w"]["slots"][0]["t"] - \ (onset_plans[wi + 1]["out_s"] + onset_plans[wi + 1]["gap_s"]) - ext_s = max(0.0, min(CODA_EXTEND_S, nxt_onset_t - hard_end - 0.06)) - exp_coda_ph = s["exp"][-1]["phonemes"]["coda"] if s["exp"] else None + ext_s = max(0.0, min(ext_cap, nxt_onset_t - hard_end - 0.06)) coda_avail = (hard_end + ext_s) - (slots[-1]["t"] + 0.06) cp_plan, cp_out_f = diction_plan( s["coda"][0], s["coda"][1], exp_coda_ph, cls, voiced, hf_ratio, @@ -989,6 +1045,16 @@ def main(): else: v_end = hard_end - coda_len v_end = max(v_end, v_start + 0.03) + # R6Β·2: a word-final UNSTRESSED syllable must not starve β€” its + # vowel gets a minimum duration, borrowed by anticipating the + # note into the preceding (stressed) vowel's tail ("diminished"'s + # final -nished was 3 output frames + no coda room β†’ "deman"). + if k == n_slots - 1 and n_slots >= 2: + exp_k = s["exp"][k] if k < len(s["exp"]) else None + if exp_k is not None and not exp_k.get("stress") \ + and v_end - v_start < FINAL_UNSTRESSED_VOWEL_S: + v_start = max(slots[k - 1]["t"] + 0.10, + v_end - FINAL_UNSTRESSED_VOWEL_S) o_a, o_b = of(v_start), of(v_end) o_a = max(0, o_a) o_b = min(out_n, max(o_b, o_a + 2)) @@ -1297,7 +1363,8 @@ def main(): # R4/R5: articulate β€” the vowel leveling (arc conformance) # lifts sung vowels well above the raw spoken consonants; # boost the composite so plosive bursts stay legible. - seg = x[sa:sb] * RAW_BOOST + # R6Β·4: raw_gain adds onset-burst prominence on top. + seg = x[sa:sb] * RAW_BOOST * float(raw_gain[i]) L = ob - oa r = min(ramp, L // 2) if r > 1: @@ -1444,7 +1511,7 @@ def main(): notes_r = [(a, b) for a, b in (trimmed or notes_r) if a >= out_n or cons_mask[a:min(b, out_n)].mean() < 0.5] feats_r = [note_features(f0r, rms_r, hf_r, a, b) for a, b in notes_r] - conf = conformance(feats_r, bands) + conf = conformance(feats_r, bands, register=register) conf["_notes_measured"] = len(feats_r) clicks = click_scan(y.astype(np.float64), fs) # exonerate flux spikes that land on NATURAL consonant composites β€” those @@ -1481,6 +1548,7 @@ def main(): print(json.dumps({ "words": stats_words, "line_transpose": line_transpose, + "register": register, "beta": round(beta, 4), "harmony": harmony, "cons_stretch_scale": stretch_scale, "consonant_spans": cons_spans, diff --git a/spinging/lib/vocal_shapes.py b/spinging/lib/vocal_shapes.py index 3faa1034b..22741617a 100644 --- a/spinging/lib/vocal_shapes.py +++ b/spinging/lib/vocal_shapes.py @@ -161,18 +161,42 @@ def percentile_bands(all_feats, ps=(5, 10, 25, 50, 75, 90, 95)): return bands -def conformance(feats_list, bands, lo="p10", hi="p90"): - """Median-of-notes per feature vs reference band β†’ {feature: {…, pass}}.""" +# Round 6: bands whose reference values ride the SPEAKER'S F0 β€” the goalposts +# were measured on low-baritone acapellas, so a register lift (singing the +# same voice N semitones up) legitimately moves them: pitch-tracking noise +# scales with f0 (drift/glide/release cents), and the harmonic comb shifts +# energy above the 4 kHz line (hf_ratio). Duration / energy / click gates are +# register-independent and stay untouched. +F0_LINKED = ("onset_glide_cents", "onset_glide_ms", "plateau_drift_cents", + "release_cents", "vib_depth_cents", "hf_ratio") + + +def conformance(feats_list, bands, lo="p10", hi="p90", register=0): + """Median-of-notes per feature vs reference band β†’ {feature: {…, pass}}. + + register (semitones above the calibrated speaker) makes the f0-linked + bands register-aware: cents/ms bands widen by 35 % per octave of lift, + hf_ratio's band shifts up with the harmonic comb (Γ—2^(R/24)) and relaxes. + """ rep = {} ok_all = True + widen = 1.0 + 0.35 * abs(register) / 12.0 + hf_shift = 2.0 ** (register / 24.0) for k, band in bands.items(): vals = np.array([f[k] for f in feats_list if k in f]) if len(vals) == 0: rep[k] = {"value": None, "pass": None} # n/a this line continue v = float(np.median(vals)) - ok = bool(band[lo] <= v <= band[hi]) - rep[k] = {"value": round(v, 3), "lo": band[lo], "hi": band[hi], "pass": ok} + b_lo, b_hi = band[lo], band[hi] + if register and k in F0_LINKED: + if k == "hf_ratio": + b_lo, b_hi = b_lo * hf_shift / widen, b_hi * hf_shift * widen + else: + b_lo, b_hi = b_lo / widen, b_hi * widen + b_lo, b_hi = round(b_lo, 3), round(b_hi, 3) + ok = bool(b_lo <= v <= b_hi) + rep[k] = {"value": round(v, 3), "lo": b_lo, "hi": b_hi, "pass": ok} # duration + energy arcs are score/arrangement-driven (and vib delay's # estimator is weak on the references) β€” advisory only if k not in ("dur_s", "energy_release_ms", "vib_rate_hz", "vib_delay_ms") and not ok: