aesthetic.computer / core
Monorepo for Aesthetic.Computer aesthetic.computer
Star
Fork 0
Watch
core plans pedal.md
13 kB
Markdown
at codex/swift-ac-runtime

🎸 Pedal: Audio Effect Plugin for Ableton Live #

Overview #

Create an audio effect (filter-style) M4L device where audio passes IN from Ableton's signal chain, gets processed in an AC piece, and passes OUT back to Ableton.

Key difference from existing devices: Current AC M4L devices (notepat, metronome, prompt) only output audio FROM the web page TO Ableton. This is an effect that processes incoming audio.

Technical Challenge #

The jweb~ Limitation #

Per Cycling74 documentation:

  • jweb~ has signal outlets only (audio OUTPUT from web page)
  • There are no signal inlets (no direct audio INPUT to web page)
  • The signal message mentioned in docs is for internal use, not audio input

Solution Architecture #

Since jweb~ cannot receive audio signals directly, we need to:

  1. Capture audio in Max using plugin~ 2 (stereo effect input)
  2. Analyze/sample the audio and send data via messages to jweb~
  3. Process/visualize in the AC piece (Web Audio)
  4. Output via jweb~'s signal outlets → plugout~

Audio Data Flow Options #

plugin~ 2 → pfft~ → snapshot~ → format → executejavascript window.acPedalFFT(data)
     │
     └→ [delay for latency compensation] → plugout~ 2
  • Send FFT magnitude/phase arrays to web page
  • Web page visualizes and/or generates new audio based on spectrum
  • Original audio passes through with optional delay
  • Best for: Visualizers, spectrum-driven synths, reactive effects

Option B: Sample-by-Sample (High latency, limited use) #

plugin~ 2 → snapshot~ @samps 128 → pack → executejavascript window.acPedalSamples(L, R)
  • Send raw sample values to web page
  • Very high message overhead, significant latency
  • Not recommended for real-time effects

Option C: Peak/RMS Envelope (Simple, low latency) #

plugin~ 2 → peakamp~ → snapshot~ → executejavascript window.acPedalEnvelope(peak)
  • Send envelope followers (peak, RMS, etc.)
  • Web audio generates sounds based on amplitude
  • Best for: Envelope followers, ducking, gates

Option D: Hybrid (Audio thru + Web effects) #

plugin~ 2 ────┬────────────────────────────────────→ *~ [dry] ─┐
              │                                                 │
              └→ analysis → jweb~ → [web audio] →──────────→ +~ → plugout~
                                           │
                            [web-generated audio only, wet]
  • Original audio passes through (dry)
  • Web audio adds effects/layers (wet)
  • Mix control for dry/wet blend

For the pedal piece, we'll use Option D because:

  1. Audio passes through at native quality (no degradation)
  2. Web Audio can add processing, visualization, or triggered sounds
  3. Dry/wet mix gives flexibility
  4. Lower latency than sample-by-sample approaches

M4L Device Structure #

Max Patcher Architecture #

┌─────────────────────────────────────────────────────────────────┐
│                      AC Pedal Effect                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                  │
│  plugin~ 2                                                       │
│     │    └────────────────────────────────────────┐              │
│     │                                             │              │
│     ▼                                             ▼              │
│  ┌──────┐   ┌──────────────────────┐         ┌─────────┐        │
│  │ FFT  │──▶│ fft analysis js     │──▶      │ *~ dry  │        │
│  │pfft~ │   │ send to jweb        │          │ level   │        │
│  └──────┘   └──────────────────────┘          └────┬────┘        │
│                                                    │              │
│  ┌────────────────────────────────────────────────┐│              │
│  │                jweb~                           ││              │
│  │  ┌────────────────────────────────────────┐   ││              │
│  │  │  aesthetic.computer/pedal?daw=1        │   ││              │
│  │  │  [signal out L] [signal out R] [msgs]  │   ││              │
│  │  └───────┬────────────┬───────────────────┘   ││              │
│  └──────────┼────────────┼───────────────────────┘│              │
│             │            │                        │              │
│             ▼            ▼                        ▼              │
│          ┌─────────┐  ┌─────────┐              ┌──┴──┐           │
│          │ *~ wet  │  │ *~ wet  │              │ +~  │           │
│          │ level   │  │ level   │              └──┬──┘           │
│          └────┬────┘  └────┬────┘                 │              │
│               └─────────┬──┘                      │              │
│                         ▼                         ▼              │
│                      ┌──┴──┐                   ┌──┴──┐           │
│                      │ +~  ├───────────────────│ +~  │           │
│                      └──┬──┘                   └──┬──┘           │
│                         │                         │              │
│                         ▼                         ▼              │
│                   plugout~ 2                                     │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  Controls:                                                 │  │
│  │  [live.dial dry/wet] [live.dial wet_vol] [live.dial drive] │  │
│  └────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Key M4L Objects #

Object Purpose
plugin~ 2 Receive stereo audio from Ableton
pfft~ FFT analysis of input
peakamp~ Amplitude envelope
jweb~ Web view with audio output
*~ Level/gain control
+~ Mix signals
plugout~ Send stereo audio back to Ableton
live.dial Automatable parameters

JavaScript Functions (window.*) #

// Called from Max with FFT data (32-128 bins typically)
window.acPedalFFT = function(magnitudes) {
  // magnitudes: array of FFT bin magnitudes [0-1]
  // Used for visualization and audio-reactive effects
};

// Called from Max with amplitude envelope
window.acPedalEnvelope = function(peakL, peakR, rmsL, rmsR) {
  // Used for envelope-following effects
};

// Called from Max with tempo/transport (inherited from base)
window.acDawTempo = function(bpm) { ... };
window.acDawTransport = function(playing) { ... };

Web Audio Processing in pedal.mjs #

Piece Structure #

// pedal.mjs - Audio Effect Pedal for Ableton Live

let fftData = [];
let envelope = { peakL: 0, peakR: 0, rmsL: 0, rmsR: 0 };
let wetLevel = 0.5;
let drive = 1.0;

function boot({ sound, query }) {
  // DAW mode detection
  const dawMode = query?.daw === "1";
  
  // Set up Web Audio processing chain
  // (triggered by envelope/FFT data, outputs via jweb~)
}

function sim({ sound }) {
  // Update envelope followers
  // Trigger sounds based on input analysis
}

function paint({ wipe, ink, screen }) {
  // Visualize FFT spectrum
  // Show input level meters
  // Display effect status
}

// Exported for M4L window.acPedal* functions
export function setFFT(data) { fftData = data; }
export function setEnvelope(pL, pR, rL, rR) {
  envelope = { peakL: pL, peakR: pR, rmsL: rL, rmsR: rR };
}

Effect Ideas for pedal.mjs #

  1. Visualizer Only - Display input spectrum, pass audio through
  2. Envelope Follower - Trigger synth notes based on input amplitude
  3. Spectral Freeze - Analyze and hold FFT, generate frozen drone
  4. Vocoder-style - Use input spectrum to modulate synth output
  5. Transient Detector - Trigger drum hits on input transients

Implementation Steps #

Phase 1: Basic Effect Shell ✅ COMPLETE #

  1. ✅ Create pedal.mjs piece with FFT visualization
  2. ✅ Add pedal device to devices.json with "type": "effect"
  3. ✅ Extend build.py with new generate_effect_patcher() function
  4. ✅ Build AMXD with audio input/output chain

Phase 2: Analysis Pipeline (NEXT) #

  1. Add FFT analysis in Max (pfft~ → js → jweb~)
  2. Add envelope followers (peakamp~ → snapshot~ → jweb~)
  3. Wire up window.acPedal* functions

Phase 3: Audio Generation #

  1. Web Audio synth triggered by envelope
  2. Dry/wet mix controls
  3. Parameter automation

How to Test #

1. Start the local dev server #

npm run site  # or npm run aesthetic

2. Build and install the pedal device #

cd ac-m4l
python3 build.py pedal --install

3. In Ableton Live #

  1. Find "AC 🎸 pedal" in the Audio Effects section of the browser
  2. Drag it onto an audio track with audio playing
  3. Observe the FFT visualization responding to input audio
  4. The dry signal passes through; the wet signal (from jweb~) is mixed in

4. Test controls #

  • Tap/Space: Cycle effect modes (visualizer → envelope-synth → freeze → gate)
  • Up/Down arrows: Adjust trigger threshold (for envelope-synth mode)

Files Created/Modified #

File Action Description
system/public/aesthetic.computer/disks/pedal.mjs ✅ CREATED AC piece for effect
ac-m4l/devices.json ✅ MODIFIED Added pedal device config with "type": "effect"
ac-m4l/build.py ✅ MODIFIED Added generate_effect_patcher() function
ac-m4l/AC 🎸 pedal (localhost:8888).amxd ✅ BUILT Generated effect device
ac-m4l/ac-fft-analyzer.js 🔜 FUTURE Max JS for FFT → jweb
plans/pedal.md ✅ CREATED This planning document

References #

Open Questions #

  1. Latency compensation: How much latency does jweb~ add? Need to delay dry signal to match.
  2. Sample rate matching: Browser AudioContext vs. Ableton - handled by existing acDawSamplerate
  3. FFT bin count: 32? 64? 128? Balance between detail and message overhead
  4. Message rate: How often to send FFT data? Every vector? Every N ms?

Testing Checklist #