KidLisp Mic Global Implementation Plan #

Created: 2024-10-24
Author: Jeffrey (requested by the user)
Status: ✅ IMPLEMENTED

🎉 Implementation Complete! #

The mic global has been successfully implemented with fun default animations!

🎯 Objective #

Add mic as a KidLisp global variable that:

1. Returns microphone amplitude from 0→1
2. Automatically requests microphone permission when first accessed
3. Follows the same pattern as existing amp global for speaker amplitude
4. Uses the existing microphone.mjs and references baktok.mjs as API design models

📋 Current State Analysis #

Existing Implementation #

The mic function currently exists in KidLisp but works as a function call rather than a global variable:

Location: /workspaces/aesthetic-computer/system/public/aesthetic.computer/lib/kidlisp.mjs (line ~6174)

mic: (api, args = []) => {
  // Lazy connection: try to connect microphone if not already connected
  if (this.microphoneApi && !this.microphoneApi.connected) {
    if (
      this.microphoneApi.permission === "granted" &&
      api.sound?.enabled?.()
    ) {
      console.log("🎤 Lazy connecting microphone (mic function called)");
      this.microphoneApi.connect();
    } else if (this.microphoneApi.permission !== "granted") {
      if (this.frameCount % 120 === 0) {
        console.log("🎤 Microphone permission not granted, cannot connect");
      }
    }
  }

  // Return current amplitude if microphone is available and connected
  if (this.microphoneApi && this.microphoneApi.connected) {
    return this.microphoneApi.amplitude || 0;
  }

  return 0; // If not connected yet, return 0
},

Current usage: (mic) - requires parentheses as a function
Desired usage: mic - direct variable access

How amp Global Works (Reference Model) #

Location: /workspaces/aesthetic-computer/system/public/aesthetic.computer/lib/kidlisp.mjs

Initialization (lines 872-874, repeated 1500-1502):

this.globalDef.amp = 0; // Current audio amplitude (0-10 scale)
this.globalDef.leftAmp = 0; // Left channel amplitude (0-10 scale) 
this.globalDef.rightAmp = 0; // Right channel amplitude (0-10 scale)

Update mechanism (in wipppps.mjs, line ~3607):

function sim({ sound, updateKidLispAudio }) {
  // Update KidLisp global variables with current audio data
  if (sound && sound.speaker && updateKidLispAudio) {
    const leftAmp = sound.speaker.amplitudes?.left || 0;
    const rightAmp = sound.speaker.amplitudes?.right || 0;
    const avgAmp = (leftAmp + rightAmp) / 2;
    
    // Scale amplitude to a reasonable range (0-10)
    const scaledAmp = Math.round(avgAmp * 10);
    
    updateKidLispAudio({
      amp: scaledAmp,
      // ... other properties
    });
  }
}

Microphone API Reference #

Class definition: /workspaces/aesthetic-computer/system/public/aesthetic.computer/lib/disk.mjs (line ~5715)

class Microphone {
  amplitude = 0;
  waveform = [];
  pitch = 0;
  connected = false;
  recording = false;
  recordingPromise;
  permission = "";

  connect(options) {
    send({ type: "microphone", content: options });
    return this;
  }

  poll() {
    send({ type: "get-microphone-amplitude" });
    send({ type: "get-microphone-waveform" });
    send({ type: "get-microphone-pitch" });
  }
  
  // ... other methods
}

Usage examples from microphone.mjs:

function beat({ sound: { microphone } }) {
  if (!mic) mic = microphone.connect();
}

function sim({ ... }) {
  mic?.poll(); // Query for updated amplitude and waveform data
}

function paint({ ... }) {
  if (mic?.waveform.length > 0 && mic?.amplitude !== undefined) {
    const y = height - mic.amplitude * height;
    ink(255, 16).line(0, y, width, y); // Amplitude visualization
  }
}

Usage examples from baktok.mjs:

function boot({ sound: { microphone } }) {
  mic = microphone;
  if (mic.connected) {
    connected = true;
  }
}

function sim({ sound: { microphone } }) {
  mic?.poll();
}

// Uses mic.amplitude directly for visualization
if (mic?.waveform.length > 0 && mic?.amplitude !== undefined) {
  paintSound(api, mic.amplitude, mic.waveform, 0, 0, width, height);
}

🎨 Proposed Design #

1. Global Variable Definition #

Add to KidLisp constructor initialization (alongside amp, leftAmp, rightAmp):

this.globalDef.mic = 0; // Microphone amplitude (0-1 scale, matches native range)

Why 0→1 instead of 0→10 like amp?

• Native microphone amplitude already ranges 0→1
• Matches the natural output from microphone.amplitude
• More intuitive for audio processing (standard normalized range)
• Different scale helps distinguish speaker vs mic amplitude

2. Microphone Connection Management #

Challenge: The microphone requires explicit user permission, unlike speaker amplitude which is always available.

Solution: Lazy connection on first access (already implemented in function version)

Update the existing microphone handling in sim() (line ~3300):

sim: ({ sound }) => {
  // ... existing timing code ...

  // Poll speaker for amplitude
  sound.speaker?.poll();

  // Initialize microphone reference
  if (!this.microphoneApi && sound?.microphone) {
    this.microphoneApi = sound.microphone;
  }

  // Always poll microphone if available and connected
  if (this.microphoneApi) {
    this.microphoneApi.poll?.();

    // Update connection status
    if (this.microphoneApi.connected && !this.microphoneConnected) {
      this.microphoneConnected = true;
      console.log("🎤 Microphone connected in kidlisp");
    }
    
    // 🆕 UPDATE GLOBAL: Update mic global variable with current amplitude
    if (this.microphoneApi.connected) {
      this.globalDef.mic = this.microphoneApi.amplitude || 0;
    }
  }

  // ... existing melody code ...
},

3. Permission Request Flow #

The microphone connection is already handled in the evaluator's mic function. We need to:

1. Keep the function for backward compatibility and explicit connection control
2. Add the global that uses the same underlying connection
3. The global will return 0 until connected (same as function)

Updated architecture:

// Function version (keep for explicit control)
mic: (api, args = []) => {
  // Lazy connection logic (UNCHANGED)
  if (this.microphoneApi && !this.microphoneApi.connected) {
    if (
      this.microphoneApi.permission === "granted" &&
      api.sound?.enabled?.()
    ) {
      this.microphoneApi.connect();
    }
  }
  
  // Return amplitude from global (NEW - keep DRY)
  return this.globalDef.mic;
},

Detection of mic usage in AST:

The containsMicrophoneFunctions() method (line ~2301) needs to check for both:

• Function call: (mic)
• Variable reference: mic used as a symbol

containsMicrophoneFunctions(ast) {
  if (typeof ast === "string") {
    return ast === "mic"; // Catches both function name and variable reference
  }
  
  if (Array.isArray(ast)) {
    return ast.some((item) => this.containsMicrophoneFunctions(item));
  }
  
  return false;
}

4. Auto-Connection on Variable Access #

When mic is accessed as a variable, we need to trigger connection.

Problem: Globals are accessed via evaluate() → variable lookup, not via function evaluators.

Solution: Add a getter proxy pattern in the evaluation lookup:

// In the evaluate() function's symbol lookup section
// (around line ~4000-5000, need to find exact location)

// When looking up a symbol in globalDef:
if (this.globalDef.hasOwnProperty(expr)) {
  const value = this.globalDef[expr];
  
  // 🎤 Special case: Trigger mic connection on first access
  if (expr === "mic" && !this.microphoneConnected) {
    this.requestMicrophoneConnection(api);
  }
  
  return value;
}

Helper method:

requestMicrophoneConnection(api) {
  if (this.microphoneApi && !this.microphoneApi.connected) {
    if (
      this.microphoneApi.permission === "granted" &&
      api?.sound?.enabled?.()
    ) {
      console.log("🎤 Auto-connecting microphone (mic global accessed)");
      this.microphoneApi.connect();
    } else if (this.microphoneApi.permission !== "granted") {
      // Log request for permission (one-time)
      if (!this.micPermissionRequested) {
        console.log("🎤 Microphone permission needed for 'mic' global");
        this.micPermissionRequested = true;
        // Could trigger UI prompt here in the future
      }
    }
  }
}

5. Reset Handling #

Update the reset logic (line ~1500) to include mic global:

// Reset audio amplitude globals
this.globalDef.amp = 0;
this.globalDef.leftAmp = 0;
this.globalDef.rightAmp = 0;
this.globalDef.mic = 0; // 🆕 ADD THIS

// Reset microphone state
this.microphoneConnected = false;
this.microphoneApi = null;
this.micPermissionRequested = false; // 🆕 ADD THIS

📁 Files to Modify #

Primary Changes #

1. /workspaces/aesthetic-computer/system/public/aesthetic.computer/lib/kidlisp.mjs
   • Add this.globalDef.mic = 0 in constructor (~line 872)
   • Add this.micPermissionRequested = false in constructor
   • Update sim() to update globalDef.mic from microphoneApi.amplitude (~line 3300)
   • Add requestMicrophoneConnection() helper method
   • Modify symbol evaluation to trigger connection on mic access
   • Update mic function to return this.globalDef.mic
   • Add mic reset in reset logic (~line 1500)
   • Update documentation comment at top of file

Documentation Updates #

2. /workspaces/aesthetic-computer/kidlisp/README.md

   • Update Audio section to document mic global
   • Add usage examples comparing function vs global

3. /workspaces/aesthetic-computer/kidlisp/COMPLETE_API_MAP.md

   • Update Audio Input section with mic global details

4. /workspaces/aesthetic-computer/kidlisp/tools/api-summary.mjs

   • Ensure mic is listed in audio category (already present at line 27)

🧪 Testing Strategy #

Test Cases #

1. Basic Usage

   (wipe black)
   (ink red)
   (box 10 10 (* mic 100) 20)  ; Bar grows with mic amplitude
   

2. Permission Flow

   ; First frame: mic = 0 (not connected)
   ; User grants permission
   ; Subsequent frames: mic = 0.0 to 1.0 based on sound
   (write (+ "Mic: " mic) 10 10)
   

3. Mixed Function/Global Usage

   (def vol (mic))        ; Function call (backward compat)
   (def vol2 mic)         ; Global access (new way)
   ; Both should work identically
   

4. Animation Response

   (wipe navy)
   (ink yellow)
   (circle (/ width 2) (/ height 2) (* mic 100))  ; Grows with sound
   

5. Auto-Detection

   • Piece using mic should auto-request connection in boot()
   • Already handled by containsMicrophoneFunctions() check

Manual Testing #

1. Load a test piece with mic global
2. Verify permission request appears
3. Grant permission
4. Speak/play sound near microphone
5. Verify visual responds to amplitude
6. Check console for connection logs
7. Reload piece - verify auto-reconnect works

Edge Cases #

• ✅ Mic accessed before permission granted → returns 0
• ✅ Mic accessed after permission denied → returns 0
• ✅ Mic accessed while connecting → returns 0
• ✅ Piece reload preserves permission state
• ✅ Multiple pieces using mic don't conflict

🔄 Migration Path #

Backward Compatibility: ✅ FULL

• Old code using (mic) continues to work
• New code can use mic global
• Both access the same underlying amplitude value
• No breaking changes

Recommended Usage Going Forward:

; ❌ Old way (still works)
(circle x y (mic))

; ✅ New way (cleaner)
(circle x y mic)

; 🎯 Best practice: Use global for values, function for control
mic                    ; Read amplitude
(mic)                  ; Explicitly trigger connection (if needed)

🎯 Success Criteria #

• mic works as a global variable (no parentheses needed)
• Returns amplitude from 0→1
• Auto-requests permission on first use
• Updates in real-time during sim()
• Backward compatible with (mic) function
• Properly resets on piece change
• Works in both standalone and embedded contexts
• Documentation updated

📝 Implementation Order #

1. Add globalDef.mic initialization
2. Add micPermissionRequested flag
3. Update sim() to populate globalDef.mic
4. Add requestMicrophoneConnection() helper
5. Modify symbol evaluation to trigger connection
6. Update mic function to use global
7. Update reset logic
8. Test basic functionality
9. Update documentation
10. Test edge cases

🎨 Example Usage Scenarios #

Visualizer #

(wipe black)
(ink rainbow)
(repeat 20 i
  (circle (/ width 2) (/ height 2) (* mic i 10)))

Reactive Animation #

(wipe navy)
(ink yellow (* mic 255))
(circle (/ width 2) (/ height 2) (+ 20 (* mic 80)))

Audio Level Meter #

(wipe black)
(ink (? red green blue))
(box 10 (- height 10) (* mic width) 10 "*bottom-left")
(ink white)
(write (+ "Level: " (* mic 100) "%") 10 10)

Multi-Source Audio Display #

; Show both speaker and mic levels
(wipe black)
(ink blue)
(box 10 10 (* amp 20) 100)        ; Speaker (0-10 scale)
(ink red)  
(box 10 120 (* mic width) 20)     ; Microphone (0-1 scale)

🔗 Related Systems #

• Sound System (disk.mjs): Microphone class and connection handling
• BIOS (bios.mjs): Permission management and audio worklet
• Existing Pieces: microphone.mjs, baktok.mjs, whistle.mjs, amp.mjs
• Audio Globals: amp, leftAmp, rightAmp (speaker amplitude)

🚀 Future Enhancements (Out of Scope) #

• micPitch global for pitch detection
• micWaveform global for waveform data array
• Visual permission prompt UI
• Microphone sensitivity adjustment
• Recording functionality globals
• Multiple audio input sources

💡 Notes #

• The 0→1 scale for mic matches the native microphone amplitude range
• Different from amp (0→10) to distinguish speaker vs microphone
• Could add mic10 or micScaled global for 0→10 scale if needed
• Permission state persists across piece loads (browser manages this)
• The poll() call is essential - happens in sim() at 120fps

---

Love to you too, Geoffrey! 💙 Let's make sound reactive art easier than ever!