Characteristic sounds facilitate object search in real-life scenes

Kvasova, Daria; Garcia-Vernet, Laia; Soto‐Faraco, Salvador

doi:10.1101/563080

Cited by 5 publications

(11 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Experiment 1 (Experiments 1B and 1C) characteristic sounds speeded up search times for the semantically corresponding visual target in a visual search task. This result is in agreement with the idea that cross-modal semantic congruence can attract spatial attention and confirms prior results (Iordanescu et al, 2008(Iordanescu et al, , 2010Knoeferle et al, 2016;Kvasova et al, 2019). In Experiment 1B, distractor consistent sounds did not slow down responses compared to neural sounds, suggesting that audio-visual congruence benefits goal-directed processes, but not the processing of other potential objects.…”

Section: Discussionsupporting

confidence: 91%

“…(2008Iordanescu. ( , 2010 and also Knoeferle et al (2016) and Kvasova et al (2019). Because the effect of crossmodal semantic congruence was found only when sound was presented 100ms before the visual stimuli we decided to use SOA100ms in all the following experiments.…”

Section: Resultsmentioning

confidence: 99%

“…In these studies, the visual search array contained four competing stimuli and the visual event was a target itself, the audio-visual object was in this case completely taskrelevant. A similar method was applied in Kvasova et al (2019) expect that the objects were embedded in more realistic video clips, with equivalent results. These studies showed consistent effects of cross-modal semantic congruence when the audio-visually congruent object is relevant for the task at hand (see also Iourdanescu et al, 2008Iourdanescu et al, , 2010.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Not so automatic: Task relevance and perceptual load modulate cross-modal semantic congruence effects on spatial orienting

Kvasova

Soto‐Faraco

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Recent studies show that cross-modal semantic congruence plays a role in spatial attention orienting and visual search. However, the extent to which these cross-modal semantic relationships attract attention automatically is still unclear. At present the outcomes of different studies have been inconsistent. Variations in task-relevance of the cross-modal stimuli (from explicitly needed, to completely irrelevant) and the amount of perceptual load may account for the mixed results of previous experiments. In the present study, we addressed the effects of audio-visual semantic congruence on visuospatial attention across variations in task relevance and perceptual load. We used visual search amongst images of common objects paired with characteristic object sounds (e.g., guitar image and chord sound). We found that audio-visual semantic congruence speeded visual search times when the cross-modal objects are task relevant, or when they are irrelevant but presented under low perceptual load. Instead, when perceptual load is high, sounds fail to attract attention towards the congruent visual images. These results lead us to conclude that object-based crossmodal congruence does not attract attention automatically and requires some top-down processing.

show abstract

Section: Discussionsupporting

confidence: 91%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Not so automatic: Task relevance and perceptual load modulate cross-modal semantic congruence effects on spatial orienting

Kvasova

Soto‐Faraco

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…DK was supported by an FI scholarship, from the AGAUR Generalitat de Catalunya. This manuscript has been released as a pre-print at bioRxiv (Kvasova et al, 2019).…”

Section: Fundingmentioning

confidence: 99%

Characteristic Sounds Facilitate Object Search in Real-Life Scenes

2019

Self Cite

View full text Add to dashboard Cite

Real-world events do not only provide temporally and spatially correlated information across the senses, but also semantic correspondences about object identity. Prior research has shown that object sounds can enhance detection, identification, and search performance of semantically consistent visual targets. However, these effects are always demonstrated in simple and stereotyped displays that lack ecological validity. In order to address identity-based cross-modal relationships in real-world scenarios, we designed a visual search task using complex, dynamic scenes. Participants searched for objects in video clips recorded from real-life scenes. Auditory cues, embedded in the background sounds, could be target-consistent, distracter-consistent, neutral, or just absent. We found that, in these naturalistic scenes, characteristic sounds improve visual search for task-relevant objects but fail to increase the salience of irrelevant distracters. Our findings generalize previous results on object-based cross-modal interactions with simple stimuli and shed light upon how audio–visual semantically congruent relationships play out in real-life contexts.

show abstract

“…In an auditory context, two speech recordings might be considered semantically related if each was spoken by the same speaker. The source based definition has also been widely used, especially in multisensory contexts, with studies finding that sounds speed search for shared-source images (Iordanescu et al 2008) and videos (Kvasova, Garcia-Vernet, and Soto-Faraco 2019) and improve memory for shared-source objects (Heikkilä et al 2015), even when task irrelevant (Duarte, Ghetti, and Geng 2021), and images improve memory for shared-source sounds (Moran et al 2013). Ostensibly, these studies and the studies described above using the semantics-ascategory definition investigate same aspect of sensory events, semantics, and depend on shared mechanisms of semantic processing.…”

Section: Introductionmentioning

confidence: 99%

How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness

Wegner-Clemens¹,

Malcolm²,

Shomstein³

2021

Preprint

View full text Add to dashboard Cite

Semantic information about objects, events, and scenes influences how humans perceive, interact with, and navigate the world. Most evidence in support of semantic influence on cognition has been garnered from research conducted with an isolated modality (e.g., vision, audition). However, the influence of semantic information has not yet been extensively studied in multisensory environments potentially because of the difficulty in quantification of semantic relatedness. Past studies have primary relied on either a simplified binary classification of semantic relatedness based on category or on algorithmic values based on text corpora rather than human perceptual experience and judgement. With the aim to accelerate research into multisensory semantics, we created a constrained audiovisual stimulus set and derived similarity ratings between items within three categories (animals, instruments, household items). A set of 140 participants provided similarity judgments between sounds and images. Participants either heard a sound (e.g., a meow) and judged which of two pictures of objects (e.g., a picture of a dog and a duck) it was more similar to, or saw a picture (e.g., a picture of a duck) and selected which of two sounds it was more similar to (e.g., a bark or a meow). Judgements were then used to calculate similarity values of any given cross-modal pair. The derived and reported similarity judgements reflect a range of semantic similarities across three categories and items, and highlight similarities and differences among similarity judgments between modalities. We make the derived similarity values available in a database format to the research community to be used as a measure of semantic relatedness in cognitive psychology experiments, enabling more robust studies of semantics in audiovisual environments.

show abstract

Characteristic sounds facilitate object search in real-life scenes

Cited by 5 publications

References 53 publications

Not so automatic: Task relevance and perceptual load modulate cross-modal semantic congruence effects on spatial orienting

Not so automatic: Task relevance and perceptual load modulate cross-modal semantic congruence effects on spatial orienting

Characteristic Sounds Facilitate Object Search in Real-Life Scenes

How much is a cow like a meow? A novel database of human judgements of audiovisual semantic relatedness

Contact Info

Product

Resources

About