A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

Borsci, Simone; Schmettow, Martin; Malizia, Alessio; Chamberlain, Alan; Velde, Frank van der

doi:10.1007/s00779-022-01690-0

Cited by 21 publications

(9 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Both have not been specifically designed for health CAs. We identified two methods that rely upon these two, but have been adapted to the peculiarities of CAs: Bot usability scale (BUS-11, [ 30 ]) and heuristics [ 31 ] for CA design. The set of heuristics for CA design comprises 11 criteria.…”

Section: Resultsmentioning

confidence: 99%

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Denecke

2023

Healthcare

View full text Add to dashboard Cite

Evaluating conversational agents (CAs) that are supposed to be applied in healthcare settings and ensuring their quality is essential to avoid patient harm and ensure efficacy of the CA-delivered intervention. However, a guideline for a standardized quality assessment of health CAs is still missing. The objective of this work is to describe a framework that provides guidance for development and evaluation of health CAs. In previous work, consensus on categories for evaluating health CAs has been found. In this work, we identify concrete metrics, heuristics, and checklists for these evaluation categories to form a framework. We focus on a specific type of health CA, namely rule-based systems that are based on written input and output, have a simple personality without any kind of embodiment. First, we identified relevant metrics, heuristics, and checklists to be linked to the evaluation categories through a literature search. Second, five experts judged the metrics regarding their relevance to be considered within evaluation and development of health CAs. The final framework considers nine aspects from a general perspective, five aspects from a response understanding perspective, one aspect from a response generation perspective, and three aspects from an aesthetics perspective. Existing tools and heuristics specifically designed for evaluating CAs were linked to these evaluation aspects (e.g., Bot usability scale, design heuristics for CAs); tools related to mHealth evaluation were adapted when necessary (e.g., aspects from the ISO technical specification for mHealth Apps). The resulting framework comprises aspects to be considered not only as part of a system evaluation, but already during the development. In particular, aspects related to accessibility or security have to be addressed in the design phase (e.g., which input and output options are provided to ensure accessibility?) and have to be verified after the implementation phase. As a next step, transfer of the framework to other types of health CAs has to be studied. The framework has to be validated by applying it during health CA design and development.

show abstract

Section: Resultsmentioning

confidence: 99%

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Denecke

2023

Healthcare

View full text Add to dashboard Cite

show abstract

“…The advancement in artificial intelligence research and applications is pushing chatbots and conversational systems in sensitive and high-risk domains such as education, training, and rehabilitation. This is particularly relevant in Europe, in which evidence of the trustworthiness of products is an essential condition for market access, especially for AIdriven tools [5][6][7]. In such context, the BUS-11 could be the basis to generate evidence about the perceived quality of AI and algorithmic chatbots, as this scale offers a quick and standardized assessment tool that designers can use to avoid unsatisfactory and potentially problematic solutions to reach the market.…”

Section: Discussionmentioning

confidence: 99%

“…For this reason, experts are forced to assess the interaction with chatbots using only qualitative and non-comparable observations or by using scales developed for point-and-click interaction that don't consider aspects such as conversational quality. Acknowledging this issue, Borsci and colleagues [5][6][7], Borsci, Schmettow, et al, 2022) recently proposed and validated a scale in English, German, Spanish, and Dutch to assess the satisfaction of end-users-i.e., perceived usability (ISO 9241-11, 2018)-after the interaction with a chatbot: the ChatBot Usability Scale (BUS).…”

Section: Introductionmentioning

confidence: 99%

“…In line with previous studies on the BUS scale (Borsci, Schmettow, et al, 2022), the Italian version of the scale is expected to have a high level of reliability, e.g., Cronbach's alpha over 0.7, and to be positively correlated with the UMUX-Lite scale [3], i.e., convergent validity. It should also be noted that the BUS questionnaire was previously assessed in a digital form with items presented to participants in a randomized order [5][6][7], Borsci, Schmettow, et al, 2022). Nevertheless, the scale could also be used as a paper and pencil questionnaire with a predefined order of the items.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

“Ciao AI”: The Italian adaptation and validation of the Chatbot Usability Scale

Borsci¹,

Prati²,

Federici³

et al. 2022

Preprint

View full text Add to dashboard Cite

Chatbot-based tools becoming pervasive in multiple domains from commercial websites to rehabilitation applications. Only recently an eleven-item satisfaction inventory was developed (the chatBot Usbaliy Scale, BUS-11) to help designers in the assessment process of their systems. The BUS-11 has been validated in multiple contexts and languages i.e., English, German, Dutch, and Spanish. This scale forms a solid platform enabling designers to rapidly assess chatbots both during and after the design process. The present work aims to adapt and validate the BUS-11 inventory in Italian. A total of 1360 questionnaires were collected which related to a total of 10 Italian chatbot-based systems using the BUS-11 inventory and also using the lite version of the Usability Metrics for User eXperience for convergent validity purposes. The Italian version of the BUS-11 was adapted in terms of the wording of one item, and a Multi-Group Confirmatory Factorial Analysis was performed to establish the factorial structure of the scale and compare the effects of the wording adaptation. Results indicate that the adapted Italian version of the scale matches the expected factorial structure of the original scale. The Italian BUS-11 is highly reliable (Cronbach alpha: 0.921), and it correlates to other measures of satisfaction (e.g., UMUX-Lite, τb=0.67; p< .001 ) by also offering specific insights regarding the chatbots’ characteristics. The Italian BUS-11 can be confidently used by chatbot designers to assess the satisfaction of their users during formative or summative tests.

show abstract

“…The BUS-15 is a 15-question validated survey meant to assess and compare the satisfaction of users after interactions with conversational chatbots. 18 It covers a range of themes related to accessibility, quality of chatbot functions, quality of conversation and information provided, perceived privacy and security, and response time. Of the 15, we chose the 7 questions that most aligned with the study goals (to review how the chatbot responds to medical prompts, particularly questions related to genetics) to be completed by domain experts (two genetic counselors and two clinical geneticists).…”

Section: Methodsmentioning

confidence: 99%

Evaluating ChatGPT as an Agent for Providing Genetic Education

Walton,

Gracefo,

Sutherland

et al. 2023

Preprint

View full text Add to dashboard Cite

SummaryGenetic disorders are complex and can greatly impact an individual’s health and well-being. In this study, we assess the ability of ChatGPT, a language model developed by OpenAI, to answer questions related to three specific genetic disorders: BRCA1, MLH1, and HFE. ChatGPT has shown it can supply articulate answers to a wide spectrum of questions. However, its ability to answer questions related to genetic disorders has yet to be evaluated. The aim of this study is to perform both quantitative and qualitative assessments of ChatGPT’s performance in this area. The ability of ChatGPT to provide accurate and useful information to patients was assessed by genetic experts. Here we show that ChatGPT answered 64.7% of the 68 genetic questions asked and was able to respond coherently to complex questions related to the three genes/conditions. Our results reveal that ChatGPT can provide valuable information to individuals seeking information about genetic disorders, however, it still has some limitations and inaccuracies, particularly in understanding human inheritance patterns. The results of this study have implications for both genomics and medicine and can inform future developments in this area. AI platforms, like ChatGPT, have significant potential in the field of genomics. As these technologies become integrated into consumer-facing products, appropriate oversight is required to ensure accurate and safe delivery of medical information. With such oversight and training specifically for genetic information, these platforms could have the potential to augment some clinical interactions.

show abstract

A confirmatory factorial analysis of the Chatbot Usability Scale: a multilanguage validation

Cited by 21 publications

References 35 publications

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare

“Ciao AI”: The Italian adaptation and validation of the Chatbot Usability Scale

Evaluating ChatGPT as an Agent for Providing Genetic Education

Contact Info

Product

Resources

About