What (experimental media) design could do for humanising algorithmic listening

Reflections on workshop 1

What (experimental media) design could do for humanising algorithmic listening

In search for a common field of problems

By Shintaro Miyazaki

Disclaimer: Please read the following keeping in mind that I am not trained in signal processing, data mining nor have an engineering background. The concepts I use need complementation by more knowledgeable persons.

I decided to write up my thoughts right after the great first workshop (which I really enjoyed) and before they fade away. Hopefully I can clarify some of my arguments here; perhaps we can also begin to sketch some common fields of problems for all network participants. As a media archaeologist turned non-affirmative design researcher with a background in improvised laptop music I might care at least for some of the overlapping fields. My remarks resonate mostly with the aspect of „Unpacking black boxes through listening“ Alice sketched out in her introductory blog post.

stethoscope Standardized Patient Program examining the abdomen, Photo by Steve Perrin CC BY 2.0

Historically considered, listening has been a crucial method of unpacking boxes or enclosed bodies even before literally unpacking it, but by trying to listen to the sounds coming from inside the body both passively with a listening device or even actively by gently knocking it. You can do this with a variety of artefacts, products and found things such as stones, water melons, cheese, earth grounds, human bodies, cars and of course with all sorts of percussive instruments. In all these examples there is a direct epistemic relationship (tight coupling) between the sound created and the characteristics and structure of the body under inquiry. Since the dawn of transducers such as the telephone in the late 1870s, we can listen into electrical signals such as those of muscles or nerves, and at the same time transform electrical signals into sound. And with the dawn of the vacuum tube amplification in the 1910s, we can amplify tiny signals, such as those of brain neurons.

Following the links Alice posted and through some of the more informal discussions I had during the workshop I guess that filtering is a key aspect of machine or algorithmic listening. Filtering here is not only meant in a more metaphorical meaning as a way to structure incoming data and material (thus as a common element to all data analysis) but literally, thus more media archaeologically, as procedures in signal processing. The digitisation and quantification of analog electroacoustic signals into digital data for example is some sort of filtering, also the hand-crafted filtering of essential structures in audio feature extraction, which is, as I learned during the workshop, is often the first step of algorithm-based audio analysis (aka machine listening) before filtering them further with classification, regression and other algorithms as done in data mining and machine learning. This „hand-crafted“ and skilled practice for me is of particular interest: Humanising Algorithmic Listening might mean working through practices of filtering with a combination of ethnographic/STS, historical, critical and practice-based methods (both engineering and design/arts).

Following questions are asked to cause some response: Who decides how, when and why about the value of a certain feature and what kind of media and cultural techniques (visualization, drawing, singing, computing) are involved in the practices of hand-crafted audio feature extraction? Are there aesthetic and designerly decisions? What are the protocols within such a field of practice? Unboxing and unpacking machine or algorithmic listening means for me to critically inquire each step, each of its module in the pipe line of feature extraction, classification, segmentation and visualization. Sonification, audification and visualization of the filtering steps, processes and results are then also helpful in unboxing the more advanced data-driven feature extractors using neural networks as done by Keunwoo Choi currently a PhD Student at Queen Mary University of London.

Is there space for design-based experimentation for example in varying the modes of visualization or for auralisation/ audification? Can we listen to the algorithms filtering the data as sonification, not of the data but of the processes that generate data? Can we design human interfaces for the modules in the data analysis pipeline? Would that be helpful for getting a more comprehensive and critical understanding of machinic listening? Can we diffract algorithmic listening? Who are the data workers and researchers involved in algorithmic listening? What is the „sayable and the visible“ (Foucault) of algorithmic listening and how could we transform it by design? What kind of effects would historical contextualization have on that process? The history of resistor–capacitor circuits and filters is for example strongly connected to telegraphy and might be an early example of mathematical modelling in signal processing. The context of underwater warfare during WW2 and later might be another field of historical contextualization: Operating on different audio filters was a crucial skill for sonar operators.

Hopefully these questions and my short reflections caused some response. Please write some of your thoughts in the comments section.

Dialogue & Discussion