Visuoauditory mappings between high luminance and high pitch are shared by chimpanzees (Pan troglodytes) and humans

Abstract

Humans share implicit preferences for certain cross-sensory combinations; for example, they consistently associate higher-pitched sounds with lighter colors, smaller size, and spikier shapes. In the condition of synesthesia, people may experience such cross-modal correspondences to a perceptual degree (e.g., literally seeing sounds). So far, no study has addressed the question whether nonhuman animals share cross-modal correspondences as well. To establish the evolutionary origins of cross-modal mappings, we tested whether chimpanzees (Pan troglodytes) also associate higher pitch with higher luminance. Thirty-three humans and six chimpanzees were required to classify black and white squares according to their color while hearing irrelevant background sounds that were either high-pitched or low-pitched. Both species performed better when the background sound was congruent (high-pitched for white, low-pitched for black) than when it was incongruent (low-pitched for white, high-pitched for black). An inherent tendency to pair high pitch with high luminance hence evolved before the human lineage split from that of chimpanzees. Rather than being a culturally learned or a linguistic phenomenon, this mapping constitutes a basic feature of the primate sensory system. multisensory integration | audio-visual correspondences | comparative cognition | sound symbolism | language evolution H umans share systematic, implicit preferences for pairing sounds with visual sensations (1, 2); for example, they consistently choose higher-pitched sounds to better fit lighter colors (3-5), smaller size (4, 6), spikier shape (7), and locations higher in space (7). Such phenomena are already observed in young children (4, 6-10), and they are also reflected in language. For example, the term dunkler Ton ("dark sound") in German refers to a low-pitched sound, and both kiiroi koe ("yellow voice") in Japanese and voces blancas ("white voices") in Spanish describe high-pitched voices. Some individuals may even experience crossmodal associations to a perceptual degree; for example, they may literally see sounds (5). This neurodevelopmental condition is known as synesthesia (11-13).Cross-modal correspondences have been extensively described and analyzed, given their possible relevance for understanding the mechanisms of perception. However, so far nothing is known about when and why such correspondences (e.g., high pitch = bright) evolved. Associations might be mediated by language, cultural learning, and higher cognitive processes (1, 14), for example, by exposure to metaphors during development. If this is true for all associations, then humans might be the only species that show cross-modal correspondences and synesthesia. However, if cross-modal correspondences are a more basic, ancient feature of the perceptual system, then other animals should show such associations as well.Here we tested whether chimpanzees (Pan troglodytes) share cross-modal correspondences with humans. We assessed pitchluminance mapping (i.e., the mapping of high pitch ...

Does crossmodal correspondence modulate the facilitatory effect of auditory cues on visual search?

“…Contrary to our expectations, though, the results of Experiment 1 revealed that the crossmodal correspondence between auditory pitch and visual brightness-the existence of which has been demonstrated by a number of researchers in adults, young children, and even chimpanzees (e.g., Hubbard, 1996;Ludwig et al, 2011;Marks, 1987;Martino & Marks, 1999;Melara, 1989;Mondloch & Maurer, 2004)-did not influence participants' performance on the pip-and-pop task, at least not when the congruency of the crossmodal correspondence (i.e., congruent vs. incongruent) was varied randomly on a trial-by-trial basis. The results of Experiment 1 therefore seem to corroborate the view that it is mainly the temporal information concerning the auditory signal, and not some other quality, such as its pitch, that is integrated with the visual signal.…”

“…Further support for this notion comes from experiments, in which this crossmodal correspondence influenced the behavior of prelinguistic children (Mondloch & Maurer, 2004), and even of chimpanzees (Ludwig et al, 2011).…”

“…If the correspondence between auditory pitch and visual brightness were to be semantic in nature, it could possibly never manifest itself at a preattentive stage of perceptual processing (see also Gallace & Spence, 2006), while a structural or statistical correspondence might (see Ernst, 2007;Parise & Spence, 2009). However, mounting evidence is showing that the crossmodal correspondence between pitch and brightness cannot be purely semantic in nature, since it has been observed in both prelinguistic children (Mondloch & Maurer, 2004) and, more recently, in chimpanzees (Ludwig et al, 2011). On the other hand, Mondloch and Maurer (2004;Maurer & Mondloch, 2005) have hypothesized that the correspondence in adults may be a remainder of crossmodal neural connections that may have been present at birth and influenced the organization of perception and language before they were pruned out or placed under inhibitory control in the course of development.…”

“…Many studies have demonstrated the existence of a relationship between auditory pitch and different visual features, such as spatial elevation (Ben-Artzi & Marks, 1995;Bernstein & Edelstein, 1971;Evans & Treisman, 2010;Melara & O'Brien, 1987;Patching & Quinlan, 2002;Rusconi, Kwan, Giordano, Umiltà, & Butterworth, 2006), brightness (Ludwig, Adachi, & Matzuzawa, 2011;Marks, 1987) or lightness (Hubbard, 1996;Marks, 1987;Martino & Marks, 1999;Melara, 1989;Mondloch & Maurer, 2004), size (Evans & Treisman, 2010;Gallace & Spence, 2006;Mondloch & Maurer, 2004), angularity of shape (Marks, 1987), direction of movement (Clark & Brownell, 1976), and even spatial frequency (Evans & Treisman, 2010).…”

Crossmodal correspondences between odors and contingent features: odors, musical notes, and geometrical shapes

“…Now, this might be attributable to researchers having looked preferentially at those crossmodal influences that have been formed by associating various cues relative to the same object (e.g., following the unity assumption, considered as a crucial factor in multisensory research since Welch & Warren, 1986; see also Vatakis & Spence, 2007). However, effects that are not limited to these cases have been demonstrated in those broader crossmodal correspondences between sometimes distinctly presented objects whose dimensions happen to feel congruent: Higher-pitched sounds can facilitate the detection of brighter objects, even without coming from or being attributed to the same source or object (e.g., Klapetek, Ngo, & Spence, 2012;Ludwig, Adachi, & Matzuzawa, 2011). As we detail below, this draws attention to the need for further investigation in the case of associations holding between olfaction and musical features, and it indicates a few of the directions along which research could profitably be pursued (see Table 2 and below).…”