Evaluating information presentation strategies for spoken recommendations

Winterboer, Andi; Moore, Johanna D.

doi:10.1145/1297231.1297260

Cited by 15 publications

(19 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, demonstrating perception is in some ways a necessary prerequisite to task-based evaluation: if participants do not notice any difference between the forms of output, it is unlikely to make any great difference to their task performance. A specific instance where the results of an overhearer study agreed with those from a task-based study is found in the pair of FLIGHTS studies mentioned above (Demberg and Moore 2006;Winterboer and Moore 2007).…”

Section: User Evaluation Of Generated Outputsupporting

confidence: 56%

“…Another possible explanation is that this effect arises from the experimental setting: the subjects were judging appropriateness with reference to a random user model, rather than on one based on their own preferences. However, the FLIGHTS studies (Demberg and Moore 2006;Winterboer and Moore 2007) used a very similar design, and the users in those studies did respond to all forms of tailoring. So the use of a hypothetical user model cannot be the only issue.…”

Section: Resultsmentioning

confidence: 99%

“…However, in retrospect it might have been better to allow the subjects to use their own preferences rather than those of a hypothetical user; a task-based study would also provide more robust results than the current overhearer study. In the future, it would therefore be informative to carry out further studies that address these two limitations: users may be more likely to pay attention to tailoring that is based on their own preferences, and a full interactive study would allow taskbased measures to be gathered similar to those used by Carenini and Moore (2006) and Winterboer and Moore (2007). This would in turn provide stronger evidence as to the role of non-verbal behaviour in user tailoring.…”

Section: Discussionmentioning

confidence: 99%

“…In a similar study evaluating the output of the MATCH restaurant recommendation system (Walker et al 2004), participants preferred recommendations that were tailored to their own user preferences over those that were tailored to the preferences of some other user. In an overhearer-style user evaluation of the FLIGHTS system-in which the subjects judged the appropriateness of simulated interactions for a hypothetical target user- Demberg and Moore (2006) found that the UMSR approach was preferred over the SR approach; as noted above, the subsequent Winterboer and Moore (2007) study confirmed that the UMSR system also performed better in a task-based evaluation.…”

Section: User Evaluation Of Generated Outputmentioning

confidence: 90%

“…2.1) affected user behaviour: descriptions tailored to the user's own preferences were significantly more effective than non-tailored descriptions at persuading a user to select a new option that was presented by the system. In an evaluation of the FLIGHTS system, Winterboer and Moore (2007) found that users of the system who heard summaries generated by a process that combined user modelling with stepwise refinement (UMSR) were significantly more likely both to choose the optimal flight and to do it more quickly than users who heard summaries that used stepwise refinement only (SR).…”

Section: User Evaluation Of Generated Outputmentioning

confidence: 99%

See 4 more Smart Citations

User preferences can drive facial expressions: evaluating an embodied conversational agent in a recommender dialogue system

Foster

Oberlander

2010

User Model User-Adap Inter

View full text Add to dashboard Cite

Tailoring the linguistic content of automatically generated descriptions to the preferences of a target user has been well demonstrated to be an effective way to produce higher-quality output that may even have a greater impact on user behaviour. It is known that the non-verbal behaviour of an embodied agent can have a significant effect on users' responses to content presented by that agent. However, to date no-one has examined the contribution of non-verbal behaviour to the effectiveness of user tailoring in automatically generated embodied output. We describe a series of experiments designed to address this question. We begin by introducing a multimodal dialogue system designed to generate descriptions and comparisons tailored to user preferences, and demonstrate that the user-preference tailoring is detectable to an overhearer when the output is presented as synthesised speech. We then present a multimodal corpus consisting of the annotated facial expressions used by a speaker to accompany the generated tailored descriptions, and verify that the most characteristic positive and negative expressions used by that speaker are identifiable when resynthesised on an artificial talking head. Finally, we combine the corpus-derived facial displays with the tailored descriptions to test whether the addition of the non-verbal channel improves users' ability to detect the intended tailoring, comparing two strategies for selecting the displays: one based on a simple corpus-derived rule, and one This article integrates and extends the work described in Foster and White (2005) and Foster (2007a,b). M. E. Foster (B)123 342 M. E. Foster, J. Oberlander making direct use of the full corpus data. The performance of the subjects who saw displays selected by the rule-based strategy was not significantly different than that of the subjects who got only the linguistic content, while the subjects who saw the data-driven displays were significantly worse at detecting the correctly tailored output. We propose a possible explanation for this result, and also make recommendations for developers of future systems that may make use of an embodied agent to present user-tailored content.

show abstract

Section: User Evaluation Of Generated Outputsupporting

confidence: 56%

Section: Resultsmentioning

confidence: 99%