Non-Verbal Imitations as a Sketching Tool for Sound Design

Lemaître, Guillaume; Susini, Patrick; Rocchesso, Davide; Lambourg, Christophe; Boussard, Patrick

doi:10.1007/978-3-319-12976-1_34

Cited by 6 publications

(8 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Whereas musicians, sounds engineers, or acousticians rely on a specialized vocabulary [1], most lay persons struggle to describe their auditory experience [2, 3]. Consider for example this excerpt of a conversation from a game-like experiment: a first person was describing a sound he had just heard (the “referent sound”) to a second person who, in turn, had to retrieve the referent sound from a list of distractors [4]:…”

Section: Introductionmentioning

confidence: 99%

“…We replicated this example with several persons and confirmed that human speakers spontaneously rely on vocalizations and gestures to describe sounds whenever they run out of words. Following our previous work [4–7], the goal of the present study was to further characterize what human vocal imitations can communicate about. More precisely, it studied how well listeners can identify what vocal imitations refer to.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Vocal Imitations of Non-Vocal Sounds

et al. 2016

Self Cite

View full text Add to dashboard Cite

Imitative behaviors are widespread in humans, in particular whenever two persons communicate and interact. Several tokens of spoken languages (onomatopoeias, ideophones, and phonesthemes) also display different degrees of iconicity between the sound of a word and what it refers to. Thus, it probably comes at no surprise that human speakers use a lot of imitative vocalizations and gestures when they communicate about sounds, as sounds are notably difficult to describe. What is more surprising is that vocal imitations of non-vocal everyday sounds (e.g. the sound of a car passing by) are in practice very effective: listeners identify sounds better with vocal imitations than with verbal descriptions, despite the fact that vocal imitations are inaccurate reproductions of a sound created by a particular mechanical system (e.g. a car driving by) through a different system (the voice apparatus). The present study investigated the semantic representations evoked by vocal imitations of sounds by experimentally quantifying how well listeners could match sounds to category labels. The experiment used three different types of sounds: recordings of easily identifiable sounds (sounds of human actions and manufactured products), human vocal imitations, and computational “auditory sketches” (created by algorithmic computations). The results show that performance with the best vocal imitations was similar to the best auditory sketches for most categories of sounds, and even to the referent sounds themselves in some cases. More detailed analyses showed that the acoustic distance between a vocal imitation and a referent sound is not sufficient to account for such performance. Analyses suggested that instead of trying to reproduce the referent sound as accurately as vocally possible, vocal imitations focus on a few important features, which depend on each particular sound category. These results offer perspectives for understanding how human listeners store and access long-term sound representations, and sets the stage for the development of human-computer interfaces based on vocalizations.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Vocal Imitations of Non-Vocal Sounds

et al. 2016

Self Cite

View full text Add to dashboard Cite

show abstract

“…While our research has shown that descriptive language can work well for communicating some audio concepts, studies have shown that vocal imitations are more effective for communicating audio concepts that are unidentifiable or synthesized abstract sounds (Lemaitre and Rocchesso 2014;Lemaitre et al 2013aLemaitre et al , 2013b)-similar to how a sketched drawing may communicate an abstract visual concept more easily than a sentence can. Similarly, vocal imitation can also be thought of as sketching in the auditory domain (i.e., "auditory sketching")-a quick, rough approximation of an audio idea that is accessible to many people.…”

Section: Synthassist: Vocal Imitation Evaluation and Explorationmentioning

confidence: 77%

Learning to Build Natural Audio Production Interfaces

Pardo

Cartwright

Seetharaman

et al. 2019

Arts

View full text Add to dashboard Cite

Improving audio production tools provides a great opportunity for meaningful enhancement of creative activities due to the disconnect between existing tools and the conceptual frameworks within which many people work. In our work, we focus on bridging the gap between the intentions of both amateur and professional musicians and the audio manipulation tools available through software. Rather than force nonintuitive interactions, or remove control altogether, we reframe the controls to work within the interaction paradigms identified by research done on how audio engineers and musicians communicate auditory concepts to each other: evaluative feedback, natural language, vocal imitation, and exploration. In this article, we provide an overview of our research on building audio production tools, such as mixers and equalizers, to support these kinds of interactions. We describe the learning algorithms, design approaches, and software that support these interaction paradigms in the context of music and audio production. We also discuss the strengths and weaknesses of the interaction approach we describe in comparison with existing control paradigms.

show abstract

“…Consider this excerpt of a conversation between two persons describing a sound, (gestures are indicated between brackets) [ 1 ]:…”

Section: Introductionmentioning

confidence: 99%

Rising tones and rustling noises: Metaphors in gestural depictions of sounds

et al. 2017

Self Cite

View full text Add to dashboard Cite

Communicating an auditory experience with words is a difficult task and, in consequence, people often rely on imitative non-verbal vocalizations and gestures. This work explored the combination of such vocalizations and gestures to communicate auditory sensations and representations elicited by non-vocal everyday sounds. Whereas our previous studies have analyzed vocal imitations, the present research focused on gestural depictions of sounds. To this end, two studies investigated the combination of gestures and non-verbal vocalizations. A first, observational study examined a set of vocal and gestural imitations of recordings of sounds representative of a typical everyday environment (ecological sounds) with manual annotations. A second, experimental study used non-ecological sounds whose parameters had been specifically designed to elicit the behaviors highlighted in the observational study, and used quantitative measures and inferential statistics. The results showed that these depicting gestures are based on systematic analogies between a referent sound, as interpreted by a receiver, and the visual aspects of the gestures: auditory-visual metaphors. The results also suggested a different role for vocalizations and gestures. Whereas the vocalizations reproduce all features of the referent sounds as faithfully as vocally possible, the gestures focus on one salient feature with metaphors based on auditory-visual correspondences. Both studies highlighted two metaphors consistently shared across participants: the spatial metaphor of pitch (mapping different pitches to different positions on the vertical dimension), and the rustling metaphor of random fluctuations (rapidly shaking of hands and fingers). We interpret these metaphors as the result of two kinds of representations elicited by sounds: auditory sensations (pitch and loudness) mapped to spatial position, and causal representations of the sound sources (e.g. rain drops, rustling leaves) pantomimed and embodied by the participants’ gestures.

show abstract

Non-Verbal Imitations as a Sketching Tool for Sound Design

Cited by 6 publications

References 23 publications

Vocal Imitations of Non-Vocal Sounds

Vocal Imitations of Non-Vocal Sounds

Learning to Build Natural Audio Production Interfaces

Rising tones and rustling noises: Metaphors in gestural depictions of sounds

Contact Info

Product

Resources

About