From efd4548e8f4a82eb5630fc918fc35ef9eb27ace1 Mon Sep 17 00:00:00 2001 From: "prompt.ac/@jeffrey" Date: Tue, 31 Mar 2026 17:40:46 -0700 Subject: [PATCH] =?UTF-8?q?plan:=20paper=20outline=20=E2=80=94=20Sampling?= =?UTF-8?q?=20in=20AC=20(every=20sound=20is=20a=20painting)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Outlines a paper tracing sampling from notepat microphone capture through stample pixel-sample encoding to cross-platform painting-based storage. Connects to Attali's composition era and existing platter papers. Co-Authored-By: Claude Opus 4.6 (1M context) --- plans/paper-sampling-in-ac.md | 187 ++++++++++++++++++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 plans/paper-sampling-in-ac.md diff --git a/plans/paper-sampling-in-ac.md b/plans/paper-sampling-in-ac.md new file mode 100644 index 0000000000..4736d2d262 --- /dev/null +++ b/plans/paper-sampling-in-ac.md @@ -0,0 +1,187 @@ +# Paper: Sampling in Aesthetic Computer + +## Working Title +**"Every Sound is a Painting: Sampling as Visual-Auditory Practice in Aesthetic Computer"** + +## Abstract Sketch + +Aesthetic Computer treats audio samples as first-class visual objects by encoding +them as RGB pixel data in shareable "paintings." This paper traces the evolution +of sampling in AC — from the initial microphone capture in `notepat.mjs` through +`stample.mjs`'s pixel-sample encoding to a cross-platform architecture where bare- +metal devices and web browsers share samples through the same painting infrastructure. +We argue that collapsing the boundary between visual and auditory media enables new +creative workflows: a recorded sound becomes an image you can see, share as a short +code, and play back on any AC device. The system extends Attali's concept of +"composition" (where everyone creates) to the sample itself — every user's sound +becomes a painting in a shared library. + +--- + +## Paper Structure + +### 1. Introduction: The Sample Problem + +- Digital audio workstations treat samples as opaque binary blobs (WAV, AIFF) +- Sharing samples requires file transfer, format negotiation, storage accounts +- Disconnect between visual creative tools and audio creative tools +- AC's premise: what if a sample was just another painting? + +### 2. Background and Related Work + +- **Attali's "Noise"** (already cited in PLORK paper): recording → repetition → + composition. AC pushes into composition where everyone records and shares. +- **Laptop orchestra tradition** (PLORK paper context): PLOrk, SLOrk, CLOrk. + Sample sharing in ensemble contexts. +- **Visual-auditory encoding precedents**: spectrograms as visual representation, + Jerobeam Fenderson's oscilloscope art, bytebeat/pixel-to-audio experiments +- **Painting as data format**: the tradition of steganography, data art, + Manfred Mohr's algorithmic marks containing their own instructions + +### 3. Sampling in AC: Current Architecture + +#### 3.1 notepat.mjs — The Instrument + +- 7 waveform types including "sample" mode +- Home key: record from microphone +- End key: arm per-key sample recording (different sample per note) +- Sample bank: `sampleBank[key]` with individual float32 buffers +- Auto-save to `/mnt/ac-sample.raw` on native boot media +- Auto-load on next boot — the instrument remembers its voice + +Reference: `fedac/native/pieces/notepat.mjs` lines 16-28, 385-430, 590-600 + +#### 3.2 stample.mjs — The Sampler as Painting + +- `encodeSampleToBitmap(data, width)`: float32 audio → 8-bit RGB pixels + - 3 samples per pixel (R, G, B channels) + - -1.0..+1.0 mapped to 0..255 + - 256px wide, height varies with duration + - 5 seconds @ 48kHz = 240,000 samples = 80,000 pixels = 256×313 PNG ~100KB +- `decodeBitmapToSample(bitmap, meta)`: reverse process +- `loadPaintingAsAudio(source, opts)`: load ANY painting as audio + - Painting code (`#k3d`) → resolve via API → download PNG → decode → play + - KidLisp source (`$roz`) → render to bitmap → decode → play + - System painting → current canvas → decode → play + +Reference: `system/public/aesthetic.computer/lib/pixel-sample.mjs` + +#### 3.3 The Painting Infrastructure + +- Upload: `track-media.mjs` → presigned URL → DO Spaces CDN → MongoDB record +- Short codes: every painting gets a code like `#k3d` +- URL addressable: `aesthetic.computer/painting/#k3d` +- Owned by @handle: `@jeffrey/painting/slug` +- Social: shareable, embeddable, browsable +- Already handles images of any size and format + +### 4. The Bridge: Samples as Paintings Across Platforms + +#### 4.1 The Problem of Two Worlds + +- Native (ac-native): QuickJS, ALSA, direct hardware, no DOM, no Canvas +- Web (browser): Web Audio API, Canvas, full network stack +- Same pieces need to work on both platforms +- Samples recorded on bare metal should be playable in browsers and vice versa + +#### 4.2 The Solution: Paint the Sound + +- Phase 1: Add pixel-sample encoding to native (pure JS, no DOM needed) +- Phase 2: Native uploads encoded bitmaps as paintings via existing API +- Phase 3: Native downloads paintings by code and decodes to audio +- Phase 4: Unified `samples.mjs` piece on both platforms +- Phase 5: notepat gains cloud sample support on both platforms + +#### 4.3 Data Flow + +``` +Record sound → float32 array → encode as RGB pixels → upload as painting + → get short code (#abc) → share + ↓ +Download painting → decode RGB pixels → float32 array → play as audio +``` + +### 5. Design Philosophy + +#### 5.1 Why Paintings, Not WAV Files + +1. **Infrastructure reuse**: no new storage backend, API, or auth flow +2. **Visibility**: you can SEE your sample as an image +3. **Compactness**: 8-bit RGB is 4× smaller than float32 +4. **Universality**: PNG renders everywhere, WAV needs specialized tools +5. **Social integration**: paintings are already AC's shareable unit +6. **Artistic potential**: the visual appearance of a sample IS its visual identity + +#### 5.2 The Encoding as Aesthetic Choice + +The RGB encoding is lossy (32-bit float → 8-bit per channel) but this is +a feature, not a bug. The quantization introduces a subtle lo-fi character +that is consistent and reversible-within-tolerance. Like vinyl's warmth or +cassette's hiss, the encoding IS the medium. + +The visual appearance of an encoded sample reveals its structure: +- Sine waves produce smooth color gradients +- Percussion creates sharp color boundaries +- Noise produces visual static +- Speech shows rhythmic color patterns + +A user browsing a gallery of sample-paintings can develop visual intuition +for how sounds look — and vice versa. + +#### 5.3 Every User's Sound Becomes a Painting + +Following Attali's trajectory: in the "composition" era, the distinction +between producer and consumer dissolves. In AC, every user who records a +sample creates a painting. That painting enters the shared library with a +short code. Other users can load it, play it, modify it, re-record it. + +The sample is not a file in a folder. It is a painting in a gallery. + +### 6. Implementation Status and Future Work + +#### 6.1 What Exists Today +- stample.mjs (web): full pixel-sample pipeline +- notepat.mjs (native): microphone recording, per-key sample bank +- samples.mjs (native): local sample library browser +- pixel-sample.mjs: encode/decode library +- Painting upload/download/code infrastructure + +#### 6.2 What's In Progress +- Native pixel-sample encoding (pure JS) +- Native painting upload from device +- Cross-platform samples.mjs +- Cloud sample sync (local-first, cloud optional) + +#### 6.3 Future Directions +- **Collaborative sampling**: multiple devices record simultaneously, merge into one painting +- **Live sample streaming**: one device records, others play in real time via UDP +- **Generative samples**: KidLisp programs that generate sample-paintings algorithmically +- **Sample DNA**: track lineage when samples are re-recorded/modified (like git for sound) +- **Physical prints**: print sample-paintings on paper, scan them back as audio + +### 7. Conclusion + +By treating samples as paintings, AC removes an artificial boundary between +visual and auditory creative practice. A sound has a color. A painting has +a voice. The infrastructure is the same. The creative act is unified. + +--- + +## References to Include + +- Attali, Jacques. "Noise: The Political Economy of Music." 1985. +- Blackwell et al. "Live Coding: A User's Manual." 2022. +- Turchet, Luca. "Elk Audio OS." 2021. +- Existing AC papers: arxiv-ac, arxiv-plork, arxiv-notepat +- Collins, Nick. "Introduction to Computer Music." (on laptop orchestras) +- Roads, Curtis. "Microsound." (on granular/sample-based synthesis) +- Wishart, Trevor. "On Sonic Art." (on the materiality of sound) + +## Placement + +This paper fits in the platter as a companion to the notepat paper, +expanding from "the instrument" to "the sample as creative object." +It connects to the PLORK paper's discussion of distributed music-making +and to the main AC paper's architecture sections. + +Directory: `papers/arxiv-sampling/` -- 2.51.2