Towards a Gesture-Sound Cross-Modal Analysis

Caramiaux, Baptiste; Bevilacqua, Frédéric; Schnell, Norbert

doi:10.1007/978-3-642-12553-9_14

Cited by 38 publications

(36 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The term spectromorphology is used to characterize how the sonic content is shaped over time. Note that such a concept of shape, or morphology can be linked to the concept of sound object defined by Schaeffer [Schaeffer 1966], and its link with gestural description is consistent with recent results as mentioned in Section 1 [Godøy 2006;Godøy et al 2006a;Nymoen et al 2011;Caramiaux et al 2010b]. …”

Section: Experiments 2: Gesture Responsessupporting

confidence: 84%

“…We computed the norm of the velocity profiles using functional data analysis techniques [Ramsay and Silverman 1997]. As shown in previous similar studies, velocity is one of the relevant information gestures to consider [Leman et al 2009;Caramiaux et al 2010a;2010b]. Velocity profiles are directly related to the kinetic energy that participants engage during their performance.…”

Section: Discussionmentioning

confidence: 99%

“…Focusing on qualities instead of timing, other studies investigated free body gestures related to sound stimuli [Godøy et al 2006b;2006a;Nymoen et al 2010;Caramiaux et al 2010b;Nymoen et al 2011]. Godøy et al [Godøy et al 2006b] studied how piano performance mimicry can give insight on pianists' musical expression.…”

Section: Introductionmentioning

confidence: 99%

“…Alternatively, such behavioral approaches can use computational methods for cross-modal gesture-sound analysis. Caramiaux et al [Caramiaux et al 2010b] analyzed cross-modal relationships between kinematic gesture and sound features using Canonical Correlation Analysis (CCA). They found consistent control strategies for a restricted set of short and abstract sounds.…”

Section: Introductionmentioning

confidence: 99%

“…Other experiments on gesture responses to sound stimuli also showed two intrinsic gesturesound relationships. First, sound energy or spectral features were correlated with movement's velocity [Caramiaux et al 2010b]. Second, pitch was correlated with the hand's vertical position [Nymoen et al 2011].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

The Role of Sound Source Perception in Gestural Sound Description

Caramiaux

Bevilacqua

Bianco

et al. 2014

ACM Trans. Appl. Percept.

View full text Add to dashboard Cite

We investigated gesture description of sound stimuli performed during a listening task. Our hypothesis is that the strategies in gestural responses depend on the level of identification of the sound source, and specifically on the identification of the action causing the sound. To validate our hypothesis, we conducted two experiments. In the first experiment, we built two corpora of sounds. The first corpus contains sounds with identifiable causal actions. The second contains sounds where no causal actions could be identified. These corpora properties were validated through a listening test. In the second experiment, participants performed arm and hand gestures synchronously while listening to sounds taken from these corpora. Afterwards, we conducted interviews asking participants to verbalize their experience, watching their own video recordings. They were questioned on their perception of the listened sounds and on their gestural strategies. We showed that for the sounds where causal action can be identified, participants mainly mimic the action that has produced the sound. In the other case, when no action can be associated to the sound, participants trace contours related to sound acoustic features. We also found that the inter-participants gesture variability is higher for causal sounds compared to non-causal sounds. Variability demonstrates that in the first case, participants have several ways of producing the same action whereas in the second case, the sound features tend to make the gesture responses consistent.

show abstract

Section: Experiments 2: Gesture Responsessupporting

confidence: 84%