An Experimental Comparison of the Usability of Rule-based and Natural Language Processing-based Chatbots

Lim, Yeji; Lim, Jeonghun; Cho, Nam Jae

doi:10.14329/apjis.2020.30.4.832

Cited by 4 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A Shakespearean-styled chatbot increased user engagement as well as perceived product value, but user satisfaction decreased [PS9]. As for chatbot use for online shopping, the researchers found that the participants' expressed re-use intentions and the level of recommendation to others were not as high as expected [PS24].…”

Section: Design and Proceduresmentioning

confidence: 89%

“…The most frequently reported weak point was that Dr. Joy failed to meet all user intents and to cover a much broader range of content domains. [PS24] The purpose of the study is to compare rule-based and natural language processing-based chatbots in terms of usefulness, usability, searchability, reliability and attractiveness.…”

Section: Partially Personal Assistant In Healthcare Domainmentioning

confidence: 99%

“…[PS12], [PS14], [PS15],[PS18],[PS24] and standardized questionnaires are the two main usability scales used. However, the adoption of usability scales varies a lot in chatbot usability experiments, because it mainly depends on the research goal and chatbot type.Researchers developed multiple questions for self-designed questionnaires according to their research topics and measurements.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Experimentation for Chatbot Usability Evaluation: A Secondary Study

et al. 2022

View full text Add to dashboard Cite

Interest in chatbot development is on the rise. As a usability evaluation is an essential step in chatbot development, the number of experimental studies on chatbot usability has grown as well. As a result, we think a systematic mapping study is opportune. We analyzed more than 700 sources and retrieved 28 primary studies. By aggregating the research questions and examining the characteristics and metrics used to evaluate the usability of chatbots in experiments, it is possible to identify the state of the art in chatbot usability experimentation. We conducted a systematic mapping study to identify the research questions, characteristics, and metrics used to evaluate the usability of chatbots in experiments. Most experiments adopted a within-subjects design. On the other hand, few experiments provided raw data, and only one of the identified papers was part of a family of experiments. Effectiveness, efficiency, and satisfaction are usability characteristics used to identify how well users can learn and use chatbots to achieve their goals and how satisfied users are during the interaction. Generally, the experimental results revealed that chatbots have several advantages (e.g., they provide a real-time response and they improve ease of use) and some shortcomings (e.g., natural language processing, which is rated as the weakness most in need of improvement). This research offers an overview of chatbot usability experimentation. The increasing interest in this area is very recent, as works did not start to be published until 2018. Chatbot usability experiments should be more replicable to improve the reliability and transparency of the experimental results.INDEX TERMS Usability, chatbots, experiments, family of experiments, systematic mapping study

show abstract

Section: Design and Proceduresmentioning

confidence: 89%

Section: Partially Personal Assistant In Healthcare Domainmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Experimentation for Chatbot Usability Evaluation: A Secondary Study

et al. 2022

View full text Add to dashboard Cite

show abstract

“…These experiments measured usability characteristics referred to effectiveness, efficiency and satisfaction as shown in Table 1. Some of the primary studies ( [19], [26], [27], [28], [29], [30], [31], [32], [33]) consider all three of these aspects, others ( [24], [25], [34], [35], [36], [37], [38], [39], [40]) investigate efficiency and satisfaction, others again ( [41], [42], [43], [44], [45]) consider only satisfaction, whereas only one ( [46]) evaluated both effectiveness and satisfaction. Satisfaction is again the usability characteristic of most concern to researchers since it is evaluated most often.…”

Section: Related Workmentioning

confidence: 99%

“…Within the primary studies, chatbots are dedicated to various domains. Most chatbots are used as personal assistants [25], [26], [29], [30], [34], [37], [38], [39], [40], [42], [43], [44], [45], [46], [47], especially in the healthcare domain [32], [35], [38], [44], some act as recommenders [27], [28], [31], emotion-aware conversational agents [41] and e-commerce chatbots [33], [36]. Nevertheless, none of the above chatbots is applied as a modelling tool like the SOCIO chatbot.…”

Section: Related Workmentioning

confidence: 99%

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Ren

Castro

Santos

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants). Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately). Method: We conduct a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings. Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise -albeit slightly less complete. Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

show abstract

Conversational Process Modelling: State of the Art, Applications, and Implications in Practice

Klievtsova,

Benzin,

Kampik

et al. 2023

Lecture Notes in Business Information Processing

View full text Add to dashboard Cite

Chatbots such as ChatGPT have caused a tremendous hype lately. For BPM applications, it is often not clear how to apply chatbots to generate business value. Hence, this work aims at the systematic analysis of existing chatbots for their support of conversational process modelling as process-oriented capability. Application scenarios are identified along the process life cycle. Then a systematic literature review on conversational process modelling is performed. The resulting taxonomy serves as input for the identification of application scenarios for conversational process modelling, including paraphrasing and improvement of process descriptions. The application scenarios are evaluated for existing chatbots based on a real-world test set from the higher education domain. It contains process descriptions as well as corresponding process models, together with an assessment of the model quality. Based on the literature and application scenario analyses, recommendations for the usage (practical implications) and further development (research directions) of conversational process modelling are derived.

show abstract

An Experimental Comparison of the Usability of Rule-based and Natural Language Processing-based Chatbots

Cited by 4 publications

References 0 publications

Experimentation for Chatbot Usability Evaluation: A Secondary Study

Experimentation for Chatbot Usability Evaluation: A Secondary Study

Using the SOCIO Chatbot for UML Modelling: A Family of Experiments

Conversational Process Modelling: State of the Art, Applications, and Implications in Practice

Contact Info

Product

Resources

About