Drawing sounds: representing tones and chords spatially

Abstract

Research on the crossmodal correspondences has revealed that seemingly unrelated perceptual information can be matched across the senses in a manner that is consistent across individuals. An interesting extension of this line of research is to study how sensory information biases action. In the present study, we investigated whether different sounds (i.e. tones and piano chords) would bias participants' hand movements in a free movement task. Right-handed participants were instructed to move a computer mouse in order to represent three tones and two chords. They also had to rate each sound in terms of three visual analogue scales (slow-fast, unpleasant-pleasant, and weak-strong). The results demonstrate that tones and chords influence hand movements, with higher-(lower-)pitched sounds giving rise to a significant bias towards upper (lower) locations in space. These results are discussed in terms of the literature on forward models, embodied cognition, crossmodal correspondences, and mental imagery. Potential applications sports and rehabilitation are discussed briefly.

The strength of weak embodiment

“…In humans, another example of grounding can be found in cross-linguistic research, which shows that the understanding of some concepts (e.g. emotion) is closely related to the perception of physical space (Montoro, Contreras, Elosúa, & Marmolejo-Ramos, 2015;Salgado-Montejo et al, 2016). Therefore, it is reasonable to hypothesise that, while some elements of human cognition are more or less universal, they are still grounded in a material reality.…”

“…Some studies have demonstrated clear links between concrete and abstract domains by asking participants to perform various body movements, or to place valenced objects in space (positive items go "up", negative items go "down"). This has been shown to have an effect on memory, emotional state, and semantic space processes (Casasanto, 2009;Salgado-Montejo et al, 2016;Vigliocco, Vinson, Lewis, & Garrett, 2004). Lakoff and Johnson (1980) proposed that something as abstract as metaphors can be grounded in bodily experience.…”

“…A large body of research has indicated that people exhibit consistent cross-modal correspondences between different stimuli that imply very different sensory modalities. For example, people consistently match high-pitched sounds with small, bright objects that are located high up in space (Salgado-Montejo et al, 2016;Spence, 2011). There is no need to propose more 'computational models', where the concepts tend to be obscure, immaterial, and reliant on external behavior.…”

Better ears with eyes open: effects of multisensory stimulation with nonconscious visual stimuli on auditory learning

“…Also fundamental is the correspondence of sensory features, e.g. size and timbre, size and pitch, and the particularly well studied correspondence between spatial elevation and pitch (Melara & O'Brien, 1987;Ben-Artzi & Marks, 1995;Patching & Quinlan, 2002;Maeda et al, 2004;Widmann et al, 2004;Evans and Treisman, 2010;Spence, 2011;Salgado-Montejo et al, 2016). Interestingly, this ability may still be true when one of these features is nonconscious.…”

Ouvidos melhores de olhos abertos: efeitos da estimulação multissensorial com estímulo visual não consciente sobre a aprendizagem auditiva

“…De fato, esta é uma das correspondências mais estudadas (Melara & O'Brien, 1987;Ben-Artzi & Marks, 1995;Patching & Quinlan, 2002;Maeda et al, 2004;Widmann et al, 2004, Salgado-Montejo et al, 2016. Contudo, apesar destas demonstrações, inclusive em bebês, Parise (2016) levanta dúvidas sobre este mapeamento natural, que pode ter relações bayesianas de uma vida sendo exposta ao mesmo tipo de relação (Parise, 2016).…”