Natural cross-modal mappings between visual and auditory features

Evans, Karla K.; Treisman, Anne

doi:10.1167/10.1.6

Cited by 275 publications

(428 citation statements)

References 29 publications

Supporting

Mentioning

360

Contrasting

Unclassified

Order By: Relevance

“…The results obtained in our study also go further in terms of explaining the origin of the spatial mapping of pitch which underlies a well-documented series of effects (e.g. Evans & Treisman, 2010;Patching & Quinlan, 2002;Rusconi et al, 2006). They challenge the idea of a general, amodal spatial representation of pitch shared across all modalities, along with other spatial representations of magnitudes (e.g.…”

Section: General Discussion and Conclusionsupporting

confidence: 79%

“…they were told that a high pitch sound is more likely to be correlated with a high visual target, Chiou & Rich, 2012) or at least trained by using the predicates 'high' and 'low' to describe the sounds (Evans & Treisman, 2010). Here we purposively provided only neutral information about these characteristics, to avoid biasing participants on the classification of the tactile stimuli.…”

Section: General Discussion and Conclusionmentioning

confidence: 99%

See 1 more Smart Citation

Differentiated audio-tactile correspondences in sighted and blind individuals.

Deroy¹,

Fasiello²,

Hayward³

et al. 2016

Journal of Experimental Psychology: Human Perception and Perfor

View full text Add to dashboard Cite

The aim of the present study is to investigate whether the crossmodal correspondence robustly documented between auditory pitch and visual elevation has analogues in the audio-tactile domain. Across 4 experiments, the compatibility effects between intuitively congruent pairs of stimuli (i.e., outward tactile movement, going from the inside of the finger toward the fingertip and increasing pitch, or inward tactile movement and decreasing pitch) and incongruent pairs stimuli (i.e., the reverse associations) were measured. Two methods were compared to assess the behavioral effects of such a correspondence: One where participants have to respond to either the auditory or tactile stimulus presented simultaneously, while ignoring the other (speeded classification task), and the other where the auditory and tactile stimuli are presented sequentially and associated to different response buttons (implicit association test). No significant compatibility effect was observed under the speeded classification task. The implicit association test revealed a significant compatibility effect. This effect was similar in the conditions where the finger was placed vertically and horizontally. However, this implicit association between pitch and tactile movements was not observed in blind participants. These results have methodological implications for the explanation and testing of crossmodal correspondences, and the origin of the widely discussed association between pitch and vertical elevation.

show abstract

Section: General Discussion and Conclusionsupporting

confidence: 79%

Section: General Discussion and Conclusionmentioning

confidence: 99%

Differentiated audio-tactile correspondences in sighted and blind individuals.

Deroy¹,

Fasiello²,

Hayward³

et al. 2016

Journal of Experimental Psychology: Human Perception and Perfor

View full text Add to dashboard Cite

show abstract

“…Previous experimental evidence demonstrates that human adults are sensitive to these relationships, even within age and sex morphs [11,[16][17][18]. For instance, experimentally lowering F 0 and formant dispersion in young men's voices increases the speaker's perceived masculinity, size and age [19].…”

Section: Introductionmentioning

confidence: 99%

Three-month-old human infants use vocal cues of body size

et al. 2017

View full text Add to dashboard Cite

Differences in vocal fundamental (F 0 ) and average formant (F n ) frequencies covary with body size in most terrestrial mammals, such that larger organisms tend to produce lower frequency sounds than smaller organisms, both between species and also across different sex and life-stage morphs within species. Here we examined whether three-month-old human infants are sensitive to the relationship between body size and sound frequencies. Using a violation-of-expectation paradigm, we found that infants looked longer at stimuli inconsistent with the relationship-that is, a smaller organism producing lower frequency sounds, and a larger organism producing higher frequency sounds-than at stimuli that were consistent with it. This effect was stronger for fundamental frequency than it was for average formant frequency. These results suggest that by three months of age, human infants are already sensitive to the biologically relevant covariation between vocalization frequencies and visual cues to body size. This ability may be a consequence of developmental adaptations for building a phenotype capable of identifying and representing an organism's size, sex and life-stage.

show abstract

“…Many studies have demonstrated the existence of a relationship between auditory pitch and different visual features, such as spatial elevation (Ben-Artzi & Marks, 1995;Bernstein & Edelstein, 1971;Evans & Treisman, 2010;Melara & O'Brien, 1987;Patching & Quinlan, 2002;Rusconi, Kwan, Giordano, Umiltà, & Butterworth, 2006), brightness (Ludwig, Adachi, & Matzuzawa, 2011;Marks, 1987) or lightness (Hubbard, 1996;Marks, 1987;Martino & Marks, 1999;Melara, 1989;Mondloch & Maurer, 2004), size (Evans & Treisman, 2010;Gallace & Spence, 2006;Mondloch & Maurer, 2004), angularity of shape (Marks, 1987), direction of movement (Clark & Brownell, 1976), and even spatial frequency (Evans & Treisman, 2010).…”

mentioning

confidence: 99%

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

Klapetek

Ngo

Spence

2012

Atten Percept Psychophys

View full text Add to dashboard Cite

The "pip-and-pop effect" refers to the facilitation of search for a visual target (a horizontal or vertical bar whose color changes frequently) among multiple visual distractors (tilted bars also changing color unpredictably) by the presentation of a spatially uninformative auditory cue synchronized with the color change of the visual target. In the present study, the visual stimuli in the search display changed brightness instead of color, and the crossmodal congruency between the pitch of the auditory cue and the brightness of the visual target was manipulated. When cue presence and cue congruency were randomly varied between trials (Experiment 1), both congruent cues (low-frequency tones synchronized with dark target states or highfrequency tones synchronized with bright target states) and incongruent cues (the reversed mapping) facilitated visual search performance equally, relative to a no-cue baseline condition. However, when cue congruency was blocked and the participants were informed about the pitch-brightness mapping in the cue-present blocks (Experiment 2), performance was significantly enhanced when the cue and target were crossmodally congruent as compared to when they were incongruent. These results therefore suggest that the crossmodal congruency between auditory pitch and visual brightness can influence performance in the pip-and-pop task by means of top-down facilitation.

show abstract

Natural cross-modal mappings between visual and auditory features

Cited by 275 publications

References 29 publications

Differentiated audio-tactile correspondences in sighted and blind individuals.

Differentiated audio-tactile correspondences in sighted and blind individuals.

Three-month-old human infants use vocal cues of body size

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

Contact Info

Product

Resources

About