Evaluation of the Naturalness of Chatbot Applications

Atiyah, Ayah; Jusoh, Shaidah; Alghanim, Firas

doi:10.1109/jeeit.2019.8717455

Cited by 10 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…x -- [17] 2016 --x [20] 2016 --x [50] 2016 -x - [34] 2016 x -- [2] 2016 -x x [78] 2016 x -x [58] 2016 --x [6] 2016 --x [22] 2017 -x x [77] 2017 x x - [39] 2017 x x - [60] 2017 x -- [21] 2017 -x - [63] 2017 x -- [76] 2017 x -- [54] 2017 x -- [79] 2017 --x [28] 2017 -x - [53] 2017 -x - [49] 2018 -x x [67] 2018 --x [51] 2018 -x x [52] 2018 x -- [59] 2018 -x x [44] 2018 -x x [42] 2018 -x x [19] 2018 -x x [71] 2018 -x x [8] 2018 -x x [35] 2018 x -- [31] 2018 -x x [72] 2018 -x x [13] 2019 -x x [41] 2019 -x - [45] 2019 x -- [11] 2019 --x [24] 2019 x x - [29] 2019 x x - [56] 2019 x -- [33] 2019 x -x [46] 2019 -x - [10] 2019 -x x [64] 2020 -x x [3] 2020 -x x [9] 2020 -x - [4] 2020 x -- [80] 2020 x x -…”

Section: Resultsmentioning

confidence: 99%

Trends & Methods in Chatbot Evaluation

Casas

Tricot

Khaled

et al. 2020

Companion Publication of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Section: Resultsmentioning

confidence: 99%

Trends & Methods in Chatbot Evaluation

Casas

Tricot

Khaled

et al. 2020

Companion Publication of the 2020 International Conference on Multimodal Interaction

View full text Add to dashboard Cite

“…25 Other researcher suggested to evaluate the conversational experience 26 or the naturalness in open domain chatbots. 27 These evaluation approaches are close to the ISO definition of usability. 22 Beyond usability, chatbot evaluations focus on the quality of the dialogue.…”

Section: Related Workmentioning

confidence: 86%

Evaluation Metrics for Health Chatbots: A Delphi Study

et al. 2021

View full text Add to dashboard Cite

Background In recent years, an increasing number of health chatbots has been published in app stores and described in research literature. Given the sensitive data they are processing and the care settings for which they are developed, evaluation is essential to avoid harm to users. However, evaluations of those systems are reported inconsistently and without using a standardized set of evaluation metrics. Missing standards in health chatbot evaluation prevent comparisons of systems, and this may hamper acceptability since their reliability is unclear. Objectives The objective of this paper is to make an important step toward developing a health-specific chatbot evaluation framework by finding consensus on relevant metrics. Methods We used an adapted Delphi study design to verify and select potential metrics that we retrieved initially from a scoping review. We invited researchers, health professionals, and health informaticians to score each metric for inclusion in the final evaluation framework, over three survey rounds. We distinguished metrics scored relevant with high, moderate, and low consensus. The initial set of metrics comprised 26 metrics (categorized as global metrics, metrics related to response generation, response understanding and aesthetics). Results Twenty-eight experts joined the first round and 22 (75%) persisted to the third round. Twenty-four metrics achieved high consensus and three metrics achieved moderate consensus. The core set for our framework comprises mainly global metrics (e.g., ease of use, security content accuracy), metrics related to response generation (e.g., appropriateness of responses), and related to response understanding. Metrics on aesthetics (font type and size, color) are less well agreed upon—only moderate or low consensus was achieved for those metrics. Conclusion The results indicate that experts largely agree on metrics and that the consensus set is broad. This implies that health chatbot evaluation must be multifaceted to ensure acceptability.

show abstract

“…The study conducted by McTear [38] explores how the conversational search interface is relevant today and identified some key takeaways such as usage of conversational search interface in messaging apps, personalised chat experience, and search interfaces which learn from previous experiences. Another interesting fact is that, currently, most of the conversational agents are passive [39] and provide a list of replies, which in turn puts more cognitive pressure on the end users. The study conducted by Dubiel et al [39] investigated the effect of active search support and summarisation of search results in a conversational interface.…”

Section: Conversational Search Interfacementioning

confidence: 99%

“…Another interesting fact is that, currently, most of the conversational agents are passive [39] and provide a list of replies, which in turn puts more cognitive pressure on the end users. The study conducted by Dubiel et al [39] investigated the effect of active search support and summarisation of search results in a conversational interface. When the uses of conversational search interface increases, the scope of creating tools, datasets and academic resources also increases.…”

Section: Conversational Search Interfacementioning

confidence: 99%

An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface

2022

View full text Add to dashboard Cite

In the realm of information, conversational search is a relatively new trend. In this study, we have developed, implemented, and evaluated a multiview conversational image search system to investigate user search behaviour. We have also explored the potential for reinforcement learning to learn from user search behaviour and support the user in the complex information seeking process. A conversational image search system may mimic a natural language discussion with a user via text or speech, and then assist the user in locating the required picture via a dialogue-based search. We modified and improved a dual-view search interface that displays discussions on one side and photos on the other. Based on the states, incentives, and dialogues in the initial run, we developed a reinforcement learning model and a customized search algorithm in the back end that predicts which reply and images would be provided to the user among a restricted set of fixed responses. Usability of the system was validated using methodologies such as Chatbot Usability Questionnaire, System Usability Scale, and User Experience Questionnaire, and the values were tabulated. The result of this usability experiment proved that most of the users found the system to be very usable and helpful for their image search.

show abstract

Evaluation of the Naturalness of Chatbot Applications

Cited by 10 publications

References 10 publications

Trends & Methods in Chatbot Evaluation

Trends & Methods in Chatbot Evaluation

Evaluation Metrics for Health Chatbots: A Delphi Study

An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface

Contact Info

Product

Resources

About