Talking about Relations: Factors Influencing the Production of Relational Descriptions

Abstract

In a production experiment (Experiment 1) and an acceptability rating one (Experiment 2), we assessed two factors, spatial position and salience, which may influence the production of relational descriptions (such as “the ball between the man and the drawer”). In Experiment 1, speakers were asked to refer unambiguously to a target object (a ball). In Experiment 1a, we addressed the role of spatial position, more specifically if speakers mention the entity positioned leftmost in the scene as (first) relatum. The results showed a small preference to start with the left entity, which leaves room for other factors that could influence spatial reference. Thus, in the following studies, we varied salience systematically, by making one of the relatum candidates animate (Experiment 1b), and by adding attention capture cues, first subliminally by priming one relatum candidate with a flash (Experiment 1c), then explicitly by using salient colors for objects (Experiment 1d). Results indicate that spatial position played a dominant role. Entities on the left were mentioned more often as (first) relatum than those on the right (Experiments 1a–d). Animacy affected reference production in one out of three studies (in Experiment 1d). When salience was manipulated by priming visual attention or by using salient colors, there were no significant effects (Experiments 1c, d). In the acceptability rating study (Experiment 2), participants expressed their preference for specific relata, by ranking descriptions on the basis of how good they thought the descriptions fitted the scene. Results show that participants preferred most the description that had an animate entity as the first mentioned relatum. The relevance of these results for models of reference production is discussed.

Gradations of Error Severity in Automatic Image Descriptions

“…Yun et al 2013). Animacy also plays an important role in referring expression generation (Baltaretu et al, 2016;Vogels et al, 2013). We believe that animacy might also play a (yet to be determined) role in quality judgments, and our intuition is: 2 Hypothesis 1: The perceived quality of descriptions with people-related errors is lower than the perceived quality of descriptions with clothing-related errors.…”

Rethinking symbolic and visual context in Referring Expression Generation

“…More closely related to REG, different ways of integrating visual context have been found to increase system performances in the inverse task of Referring Expression Comprehension (Nagaraja et al, 2016 ; Zhang et al, 2018 ; Wang et al, 2019 ). For human reference production, different parameters of visual scenes have been shown to affect form and content of produced utterances, including e.g., spatial structures (Baltaretu et al, 2016 ; Koolen, 2019 ), properties of co-occurring objects (Koolen et al, 2015 ) and visual salience (Fukumura et al, 2010 ; Clarke et al, 2013a , b , 2015 ; Vogels et al, 2013 ), the latter being connected to more general work on how visual context contributes to attention allocation (Oliva et al, 2003 ; Torralba et al, 2006 ).…”

Occluding the face diminishes the conceptual accessibility of an animate agent

“…A wealth of research demonstrates that visual perceptual factors affect Subject selection (Antón-Méndez, 2017;Baltaretu, Krahmer, van Wijk, & Maes, 2016;Coco & Keller, 2009;Coco & Keller, 2015;Flores d'Arcais, 1975;Forrest, 1996;Gleitman et al, 2007;Hwang & Kaiser, 2015;Kuchinsky, 2009;Myachykov, Garrod, & Scheepers, 2018;Myachykov, Thompson, Garrod, & Scheepers, 2012;Osgood, 1971;Osgood & Bock, 1977;Sridhar, 1988;Tomlin, 1997;Vogels et al, 2013a). For example, in Gleitman et al (2007), participants viewed illustrations with two animate entities, such as a cat licking a dog.…”