diff --git a/papers/arxiv-granularity/granularity.tex b/papers/arxiv-granularity/granularity.tex --- a/papers/arxiv-granularity/granularity.tex +++ b/papers/arxiv-granularity/granularity.tex @@ -187,7 +187,7 @@ The speed dropdown is a machine for maximizing articulatory distance. What you want is a rate --- a smooth, one-dimensional quantity, the same kind of quantity your hand produces when it moves faster or slower. What the interface gives you is a symbolic act: open a menu, read a list of tokens, choose the token \texttt{1.5$\times$}, dismiss the menu. You are not moving the rate. You are naming one of five rates from a controlled vocabulary. \citet{norman1988everyday} would call the space between "I want it a little faster" and "select the fourth item" a \emph{gulf of execution} --- and the dropdown makes you cross it symbolically every time, with a controller (a discrete list) whose form has nothing in common with the continuous thing it sets. Engagement is the second casualty. \citet{shneiderman1983direct}, who coined \emph{direct manipulation}, built it on a \emph{continuous representation of the object of interest}, acted on by \emph{physical actions} rather than command syntax, through \emph{rapid, incremental, reversible} operations whose effect is immediately visible. Dragging a record satisfies every clause: the object is continuously present under your hand, the action is incremental and reversible in the same gesture, and the effect --- the sound --- is immediate. A speed menu satisfies none. The object of interest disappears behind a chrome popover; the action is a discrete jump; reversal means reopening the menu; and the effect is deferred until after you commit. -There is a hard number under "immediate." \citet{nielsen1993usability} fixes 0.1 seconds as the threshold below which a response feels instantaneous --- the point at which the user perceives the system as reacting rather than replying. Direct manipulation lives entirely inside that window; you drag and the world moves with your hand. The turntablist's standard is stricter still. Scratching is built from percussive gestures timed to the tens of milliseconds --- the crossfader cut several times a second, pitch events on the order of tens of ms~\citep{hansen2010scratching}. Latency on that same scale, sitting between the hand and the sound, would smear the very attack the technique is made of; the demand for tight, low-latency, closed-loop coupling between gesture and result is what \citet{wessel2002intimate} name \emph{control intimacy}. \citet{omodhrain2000playing} shows that loop is not only auditory but haptic --- the instrument pushing back against the hand is itself part of the control. The speed dropdown does not even try to enter that budget. +There is a hard number under "immediate." \citet{nielsen1993usability} fixes 0.1 seconds as the threshold below which a response feels instantaneous --- the point at which the user perceives the system as reacting rather than replying. Direct manipulation lives entirely inside that window; you drag and the world moves with your hand. And the tolerance is not folklore --- it is measured: \citet{annett2014low} find that for continuous inking, the lag between the pen and the mark it leaves becomes perceptible in the low tens of milliseconds, an order below Nielsen's already-tight bound. The turntablist's standard is stricter still. Scratching is built from percussive gestures timed to the tens of milliseconds --- the crossfader cut several times a second, pitch events on the order of tens of ms~\citep{hansen2010scratching}. Latency on that same scale, sitting between the hand and the sound, would smear the very attack the technique is made of; the demand for tight, low-latency, closed-loop coupling between gesture and result is what \citet{wessel2002intimate} name \emph{control intimacy}. \citet{omodhrain2000playing} shows that loop is not only auditory but haptic --- the instrument pushing back against the hand is itself part of the control. The speed dropdown does not even try to enter that budget. The last piece is Buxton's. The richness of an interaction, he argues, tracks the number of \emph{continuous degrees of freedom} the input affords~\citep{buxton2007multitouch,buxton2018gesture}. A menu selection is zero continuous degrees of freedom --- a choice among labels. A slider is one, sampled coarsely. A finger dragging across a groove is a continuous position \emph{and} a velocity \emph{and}, on a good surface, pressure --- several coupled dimensions streaming at the frame rate of the hand. \citet{buxton1986chunking} calls the difference \emph{chunking}: a good gesture lets you perform an entire phrase --- speed up here, hold, snap back --- as one motor act, where the menu forces you to decompose the same intention into a sequence of discrete selections. HCI has a name for the axis, too. Card, Mackinlay and Robertson's design space of input devices treats a control's \emph{resolution} --- its granularity --- as a first-class property, and a playback rate is natively a continuous, high-resolution dimension that the menu resamples down to four points~\citep{card1991morphological}. Buxton had put it more bluntly still: conventional interfaces throw away most of the hand's continuous motor bandwidth before the user ever gets to spend it~\citep{buxton1986more}. Consumer playback UIs have spent fifteen years converging on the zero-degree end of that scale. @@ -218,7 +218,7 @@ \section{The Expressive Ceiling We Already Cleared} The coarseness would be forgivable if fine, real-time control of recorded audio were hard, exotic, or unsolved. It is none of those things. It is a mature craft with a research literature, a professional practice, and cheap, shipping algorithms. -Turntablism is the existence proof at the human end. \citet{hansen2010scratching}, in a full doctoral thesis on the acoustics and performance of scratching, treats the DJ's hand as a high-resolution musical controller and models the mapping from gesture to sound in detail. \citet{hansen2006mapping} lay out the parameter mappings --- position to playback phase, velocity to pitch and rate, the crossfader as an amplitude gate --- that let a performer improvise emotion into a record in real time. The technique has been formalized enough to build instruments around: the \emph{Skipproof} virtual turntable lets non-experts execute complex scratch gestures through simplified high-level controls~\citep{hansen2010skipproof}, and Lopes explores alternative multitouch interfaces for DJing on a phone~\citep{lopes2010mtdjing}. Its cultural history is documented in as much depth as its acoustics: \citet{katz2012groove} traces scratching from a mid-1970s Bronx accident into a codified, competitive, globally taught art form. And designing the expressive gesture-to-sound mappings underneath it is a live research practice of its own~\citep{fiebrink2009meta}. This is not folk knowledge. It is a studied, teachable, decades-deep discipline of expressive playback --- and its entire premise is the continuous, gestural, low-latency control that consumer apps refuse. +Turntablism is the existence proof at the human end. \citet{hansen2010scratching}, in a full doctoral thesis on the acoustics and performance of scratching, treats the DJ's hand as a high-resolution musical controller and models the mapping from gesture to sound in detail. \citet{hansen2006mapping} lay out the parameter mappings --- position to playback phase, velocity to pitch and rate, the crossfader as an amplitude gate --- that let a performer improvise emotion into a record in real time. The technique has been formalized enough to build instruments around: the \emph{Skipproof} virtual turntable lets non-experts execute complex scratch gestures through simplified high-level controls~\citep{hansen2010skipproof}, and Lopes explores alternative multitouch interfaces for DJing on a phone~\citep{lopes2010mtdjing}. Its cultural history is documented in as much depth as its acoustics: \citet{katz2012groove} traces scratching from a mid-1970s Bronx accident into a codified, competitive, globally taught art form, and \citet{farrugia2012beyond} follows the same instruments --- vinyl, mp3, DJ software --- into the question of who is invited to command them. And designing the expressive gesture-to-sound mappings underneath it is a live research practice of its own~\citep{fiebrink2009meta}. This is not folk knowledge. It is a studied, teachable, decades-deep discipline of expressive playback --- and its entire premise is the continuous, gestural, low-latency control that consumer apps refuse. Elastic audio is the existence proof at the tool end. Every serious production environment already treats a recording as a stretchable material. Ableton Live's Warp modes use granular resynthesis to decouple time from pitch, so you can slow a passage without dropping its key or drag its tempo around a grid in real time~\citep{ableton2024warp}. Pro Tools' Elastic Audio does real-time time-stretch and pitch-shift on any clip~\citep{avid2024elastic}. Serato Sample offers real-time waveform scrubbing and needle-drop, letting you stretch or contract individual hits while preserving pitch~\citep{serato2017sample}. The hard algorithmic problem underneath all of this --- high-quality time-stretching that preserves pitch --- is so thoroughly solved that most tools license the same off-the-shelf engine and move on. The pitch-preserving stretch that would let YouTube offer a smooth, gestural speed control without the chipmunk effect is a commodity. It runs on a phone. diff --git a/papers/arxiv-granularity/references.bib b/papers/arxiv-granularity/references.bib --- a/papers/arxiv-granularity/references.bib +++ b/papers/arxiv-granularity/references.bib @@ -48,6 +48,16 @@ publisher={Morgan Kaufmann}, note={The 0.1s response-time threshold for the feeling of direct manipulation} } +@inproceedings{annett2014low, + title={How Low Should We Go? Understanding the Perception of Latency While Inking}, + author={Annett, Michelle and Ng, Albert and Dietz, Paul and Bischof, Walter F. and Gupta, Anoop}, + booktitle={Proceedings of Graphics Interface 2014 (GI '14)}, + pages={167--174}, + year={2014}, + note={Measures when the lag between a continuous gesture and its rendered + result becomes perceptible --- on the order of tens of milliseconds} +} + % === Input, Degrees of Freedom, Gesture === @incollection{buxton1986chunking, @@ -271,6 +281,14 @@ title={Groove Music: The Art and Culture of the Hip-Hop DJ}, author={Katz, Mark}, publisher={Oxford University Press}, address={New York}, + year={2012} +} + +@book{farrugia2012beyond, + title={Beyond the Dance Floor: Female DJs, Technology and Electronic Dance Music Culture}, + author={Farrugia, Rebekah}, + publisher={Intellect}, + address={Bristol, UK}, year={2012} } diff --git a/papers/cli.mjs b/papers/cli.mjs --- a/papers/cli.mjs +++ b/papers/cli.mjs @@ -1005,7 +1005,7 @@ // Paper detail descriptions and short previews, keyed by siteName. const PAPER_COPY = { "gaps-in-granularity-26-arxiv": { - detail: "Expressive Control and the Impoverishment of Consumer Playback · arXiv 6pp", + detail: "Expressive Control and the Impoverishment of Consumer Playback · arXiv 9pp", abstract: "A record scratch gives you continuous, gestural, real-time control over a recording; YouTube and Spotify give you a dropdown of round speeds, and some give nothing at all. This paper names that gap 'granularity,' grounds it in the theory of directness and the mature craft of turntablism and elastic audio, and shows that Aesthetic Computer already ships the alternative across its players. The real opening is not that the instrument is impossible, but that it lives everywhere except the default transport of the apps you actually listen and watch through.", }, diff --git a/system/public/papers.aesthetic.computer/index.html b/system/public/papers.aesthetic.computer/index.html --- a/system/public/papers.aesthetic.computer/index.html +++ b/system/public/papers.aesthetic.computer/index.html @@ -1066,7 +1066,7 @@
Gaps in Granularity
-
Expressive Control and the Impoverishment of Consumer Playback · arXiv 6pp
+
Expressive Control and the Impoverishment of Consumer Playback · arXiv 9pp
A record scratch gives you continuous, gestural, real-time control over a recording; YouTube and Spotify give you a dropdown of round speeds, and some give nothing at all. This paper names that gap 'granularity,' grounds it in the theory of directness and the mature craft of turntablism and elastic audio, and shows that Aesthetic Computer already ships the alternative across its players. The real opening is not that the instrument is impossible, but that it lives everywhere except the default transport of the apps you actually listen and watch through.
@jeffrey07/14revision 1Jul 14 17:53