“Ciao AI”: The Italian adaptation and validation of the Chatbot Usability Scale

Abstract

Chatbot-based tools becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently an eleven-item satisfaction inventory was developed (the chatBot Usbaliy Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and languages i.e., English, German, Dutch, and Spanish. This scale forms a solid platform enabling designers to rapidly assess chatbots both during and after the design process. The present work aims to adapt and validate the BUS-11 inventory in Italian. A total of 1360 questionnaires were collected which related to a total of 10 Italian chatbot-based systems using the BUS-11 inventory and also using the lite version of the Usability Metrics for User eXperience for convergent validity purposes. The Italian version of the BUS-11 was adapted in terms of the wording of one item, and a Multi-Group Confirmatory Factorial Analysis was performed to establish the factorial structure of the scale and compare the effects of the wording adaptation. Results indicate that the adapted Italian version of the scale matches the expected factorial structure of the original scale. The Italian BUS-11 is highly reliable (Cronbach alpha: 0.921), and it correlates to other measures of satisfaction (e.g., UMUX-Lite, τb=0.67; p< .001 ) by also offering specific insights regarding the chatbots’ characteristics. The Italian BUS-11 can be confidently used by chatbot designers to assess the satisfaction of their users during formative or summative tests.

Unleashing the Potential of Turkish Chatbots: A Study on the Validity and Reliability of the Bot Usability Scale

“…The TBUS scale aims to assess the different aspects that contribute to the conversational system's quality, including Accessibility (F1, average loading of items: 0.64), Perceived quality of chatbot functions (F2, average loading of items: 0.84), Perceived quality of conversation and information providing (F3, average loading of items: 0.82), Perceived privacy and security (F4), and Time Response (F5). In line with ndings from previous validation studies (Borsci, Prati, et al, 2022;, there exists a robust correlation between Factor 2 and 3 on the TBUS scale. This indicates that the participants had di culty distinguishing between factors related to functionality quality (items 3-5) and those associated with conversation and information quality (items 6-9).…”

“…The expected ve factors of the BUS-11 identi ed in previous validation studies(Borsci, Prati, et al, 2022; were con rmed in the present experiment (see Table…”