Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state-of-the-art in chatbot usability and applied human–computer interaction techniques, to analyze how to evaluate chatbot usability. Method: We have conducted a systematic mapping study, by searching the main scientific databases. The search retrieved 170 references and 21 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is still a very incipient field of research where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

Background: The use of chatbots has increased considerably in recent years. These are used in different areas and by a wide variety of users. Due to this fact, it is essential to incorporate usability in their development. Aim: Our objective is to identify the state of the art in chatbot usability and applied human-computer interaction techniques, to analyze how to evaluate chatbots usability. Method: A systematic mapping study has been conducted, searching the main scientific databases. The search retrieved 170 citations and 19 articles were retained as primary studies. Results: The works were categorized according to four criteria: usability techniques, usability characteristics, research methods and type of chatbots. Conclusions: Chatbot usability is a very incipient field of research, where the published studies are mainly surveys, usability tests, and rather informal experimental studies. Hence, it becomes necessary to perform more formal experiments to measure user experience, and exploit these results to provide usability-aware design guidelines.

Context: Recent developments in natural language processing have facilitated the adoption of chatbots in typically collaborative software engineering tasks (such as diagram modelling). Families of experiments can assess the performance of tools and processes and, at the same time, alleviate some of the typical shortcomings of individual experiments (e.g., inaccurate and potentially biased results due to a small number of participants). Objective: Compare the usability of a chatbot for collaborative modelling (i.e., SOCIO) and an online web tool (i.e., Creately). Method: We conduct a family of three experiments to evaluate the usability of SOCIO against the Creately online collaborative tool in academic settings. Results: The student participants were faster at building class diagrams using the chatbot than with the online collaborative tool and more satisfied with SOCIO. Besides, the class diagrams built using the chatbot tended to be more concise -albeit slightly less complete. Conclusion: Chatbots appear to be helpful for building class diagrams. In fact, our study has helped us to shed light on the future direction for experimentation in this field and lays the groundwork for researching the applicability of chatbots in diagramming.

Interest in chatbot development is on the rise. As a usability evaluation is an essential step in chatbot development, the number of experimental studies on chatbot usability has grown as well. As a result, we think a systematic mapping study is opportune. We analyzed more than 700 sources and retrieved 28 primary studies. By aggregating the research questions and examining the characteristics and metrics used to evaluate the usability of chatbots in experiments, it is possible to identify the state of the art in chatbot usability experimentation. We conducted a systematic mapping study to identify the research questions, characteristics, and metrics used to evaluate the usability of chatbots in experiments. Most experiments adopted a within-subjects design. On the other hand, few experiments provided raw data, and only one of the identified papers was part of a family of experiments. Effectiveness, efficiency, and satisfaction are usability characteristics used to identify how well users can learn and use chatbots to achieve their goals and how satisfied users are during the interaction. Generally, the experimental results revealed that chatbots have several advantages (e.g., they provide a real-time response and they improve ease of use) and some shortcomings (e.g., natural language processing, which is rated as the weakness most in need of improvement). This research offers an overview of chatbot usability experimentation. The increasing interest in this area is very recent, as works did not start to be published until 2018. Chatbot usability experiments should be more replicable to improve the reliability and transparency of the experimental results.INDEX TERMS Usability, chatbots, experiments, family of experiments, systematic mapping study

After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.INDEX TERMS Chatbot, usability, family of experiments.