Task demands and individual variation in referring expressions

Abstract

Aiming to improve the human-likeness of natural language generation systems, this study investigates different sources of variation that might influence the production of referring expressions (REs), namely the effect of task demands and inter-intra-individual variation. We collected REs using a discrimination game and varied the instructions, telling speakers that they would get points for being fast, creative, clear, or no incentive would be mentioned. Our results show that taskdemands affected REs production (number of words, number of attributes), and we observe a considerable amount of variation among the length of REs produced by single speakers, as well as among the REs of different speakers referring to the same targets.

The Task Matters: Comparing Image Captioning and Task-Based Dialogical Image Description

“…These observations show that the MeetUp descriptions are more focused on the task, less broad, contain much more referring expressions, which are longer then the ones in the non-task-driven set-up. Table 4 displays the most frequent adjectives in both datasets in the spirit of (Baltaretu and Ferreira, 2016), who compared type and frequency of adjectives in a similar task design. It clearly shows a trend that seems to be present in the overall data: MDs cover a broader range of object properties or image attributes than the DDs.…”

Pragmatic descriptions of perceptual stimuli

“…They score points if other participants can successfully identify the entity from the referring expressions. Baltaretu and Castro Ferreira (2016) modified the original prompt by asking participants to play fast (FA), be creative (CR), be clear and thorough (CT), or just to provide descriptions without any additional goal (NO). These different prompts had an effect on the length of the expressions (with longer expressions in the CR and CT conditions), and on the amount of adjectives used (with more adjectives in the CR-condition than in the FA-condition).…”

“…Table 3 shows an example description for each category. Baltaretu and Castro Ferreira (2016), showing the difference between the different prompts: Fast, Creative, clear and thorough, and no specific emphasis.…”

“…There is already some prior work showing that the way that crowd workers are prompted for a description can have a strong influence on form of the descriptions. Baltaretu and Castro Ferreira (2016) present results from a study manipulating a crowdsourcing task to get different kinds of referential expressions for the same entity. They experimented with different task prompts within the ReferIt-game (Kazemzadeh et al, 2014).…”

On task effects in NLG corpus elicitation: a replication study using mixed effects modeling

“…The quality of the system output depends to a large extent on the quality of the data that is used to train the system, which in turn depends on the way that data is collected. A recent trend in NLG is to study task effects in the creation of corpora for natural language generation (Baltaretu and Castro Ferreira, 2016;van Miltenburg et al, 2017;Ilinykh et al, 2018). However, there does not seem to be an established methodology to investigate whether differences in task design lead to any significant differences in the output.…”