Did chatbots miss their “Apollo Moment”? Potential, gaps, and lessons from using collaboration assistants during COVID-19

Srivastava, Biplav

doi:10.1016/j.patter.2021.100308

Cited by 11 publications

(11 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Beside quality of access to technology, literary and accessibility issues are also issues to be tackled in chatbot development. Srivastava [ 42 ] reviewed gaps found in using chatbots during COVID-19, and one of them was “inaccessible information,” that is, most of the chatbots created assumed that the users were literate, experienced with digital technology, and did not have any disabilities. These assumptions prevented a considerable part of the society from benefiting from chatbot technology.…”

Section: Discussionmentioning

confidence: 99%

Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study

Chagas¹,

Pagano²,

Prates³

et al. 2023

JMIR Hum Factors

View full text Add to dashboard Cite

Background The potential of chatbots for screening and monitoring COVID-19 was envisioned since the outbreak of the disease. Chatbots can help disseminate up-to-date and trustworthy information, promote healthy social behavior, and support the provision of health care services safely and at scale. In this scenario and in view of its far-reaching postpandemic impact, it is important to evaluate user experience with this kind of application. Objective We aimed to evaluate the quality of user experience with a COVID-19 chatbot designed by a large telehealth service in Brazil, focusing on the usability of real users and the exploration of strengths and shortcomings of the chatbot, as revealed in reports by participants in simulated scenarios. Methods We examined a chatbot developed by a multidisciplinary team and used it as a component within the workflow of a local public health care service. The chatbot had 2 core functionalities: assisting web-based screening of COVID-19 symptom severity and providing evidence-based information to the population. From October 2020 to January 2021, we conducted a mixed methods approach and performed a 2-fold evaluation of user experience with our chatbot by following 2 methods: a posttask usability Likert-scale survey presented to all users after concluding their interaction with the bot and an interview with volunteer participants who engaged in a simulated interaction with the bot guided by the interviewer. Results Usability assessment with 63 users revealed very good scores for chatbot usefulness (4.57), likelihood of being recommended (4.48), ease of use (4.44), and user satisfaction (4.38). Interviews with 15 volunteers provided insights into the strengths and shortcomings of our bot. Comments on the positive aspects and problems reported by users were analyzed in terms of recurrent themes. We identified 6 positive aspects and 15 issues organized in 2 categories: usability of the chatbot and health support offered by it, the former referring to usability of the chatbot and how users can interact with it and the latter referring to the chatbot’s goal in supporting people during the pandemic through the screening process and education to users through informative content. We found 6 themes accounting for what people liked most about our chatbot and why they found it useful—3 themes pertaining to the usability domain and 3 themes regarding health support. Our findings also identified 15 types of problems producing a negative impact on users—10 of them related to the usability of the chatbot and 5 related to the health support it provides. Conclusions Our results indicate that users had an overall positive experience with the chatbot and found the health support relevant. Nonetheless, qualitative evaluation of the chatbot indicated challenges and directions to be pursued in improving not only our COVID-19 chatbot but also health chatbots in general.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study

Chagas¹,

Pagano²,

Prates³

et al. 2023

JMIR Hum Factors

View full text Add to dashboard Cite

show abstract

“…They make a list of 1019 occupations publicly available 2 . In (Srivastava andRossi 2018, 2020), we consider the translation setting from English to an intermediate language, and then from it back to English. Inputs are primed with gendered pronouns and checked for variation.…”

Section: Bias In Machine Translatorsmentioning

confidence: 99%

“…In (Srivastava andRossi 2020, 2018), we proposed a technique to rate automated machine language translators for gender bias. Further, we created visualizations to communicate ratings (Bernagozzi et al 2021b) to users, and conducted user studies to determine how users perceive trust in the presence of such ratings (Bernagozzi et al 2021a).…”

Section: Initial Approach -Rating Machine Translatorsmentioning

confidence: 99%

“…they use -be it text (Blodgett et al 2020), audio (Koenecke et al 2020) or image (Antun et al 2020). In fact, this has been argued as one of the key factors hampering the adoption of AI techniques during an emergency like COVID-19 (Srivastava 2021) or mainstream treatment like breast cancer (Seedat and Taylor-Phillip 2022). In this context, we explore how the trustworthiness properties of AI could be assessed and communicated, without access to its code or data (i.e., the black box setting), to promote growth.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Advances in Automatically Rating the Trustworthiness of Text Processing Services

Srivastava¹,

Kausik²,

Bernagozzi³

et al. 2023

Preprint

View full text Add to dashboard Cite

AI services are known to have unstable behavior when subjected to changes in data, models or users. Such behaviors, whether triggered by omission or commission, lead to trust issues when AI works with humans. The current approach of assessing AI services in a black box setting, where the consumer does not have access to the AI's source code or training data, is limited. The consumer has to rely on the AI developer's documentation and trust that the system has been built as stated. Further, if the AI consumer reuses the service to build other services which they sell to their customers, the consumer is at the risk of the service providers (both data and model providers). Our approach, in this context, is inspired by the success of nutritional labeling in food industry to promote health and seeks to assess and rate AI services for trust from the perspective of an independent stakeholder. The ratings become a means to communicate the behavior of AI systems so that the consumer is informed about the risks and can make an informed decision. In this paper, we will first describe recent progress in developing rating methods for text-based machine translator AI services that have been found promising with user studies. Then, we will outline challenges and vision for a principled, multi-modal, causality-based rating methodologies and its implication for decision-support in real-world scenarios like health and food recommendation.

show abstract

“…One of the first systematic reviews of the effectiveness of chatbots during COVID-19 was conducted by Biplav Srivastava, AI Institute, University of South Carolina [30]. Srivastava observed minimal chatbot use during COVID-19 and asked if chatbots missed their "Apollo Moment".…”

Section: Literature Reviewmentioning

confidence: 99%

Proactive Chatbot Framework Based on the PS2CLH Model: An AI-Deep Learning Chatbot Assistant for Students

Almada

Patel

2022

Lecture Notes in Networks and Systems

View full text Add to dashboard Cite

Nowadays, universities are using more technologies dealing with students' interactions. The chatbot supported with Artificial Intelligence (AI) -Deep Learning (DL) technology exhibited a better ability and efficiency in various assistant situations. However, the effectiveness of the education chatbot is still not satisfactory. This paper proposes a new chatbot framework that integrated students' learning profiles and enhanced chatbot components to improve student interaction. The new chatbot framework uses knowledge from the PS2CLH model and AI -DL to build a proactive chatbot for assisting students' learning on their academic subjects and controllable learning factors. One of the principal novelties of the chatbot framework lies in the studentlecturer/assistant facilitator. The proactive chatbot applies multimodality to students' learning process to retain students' attention and explain the content in different ways using text, image, video, and audio to assist students and improve their learning experience effectively. Furthermore, the chatbot proactively suggests controllable learning factors for students to work on, improving their academic performance. The testing results demonstrated that the proactive chatbot offered sound accuracy and more effective learning support than other chatbots.

show abstract

Did chatbots miss their “Apollo Moment”? Potential, gaps, and lessons from using collaboration assistants during COVID-19

Cited by 11 publications

References 30 publications

Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study

Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study

Advances in Automatically Rating the Trustworthiness of Text Processing Services

Proactive Chatbot Framework Based on the PS2CLH Model: An AI-Deep Learning Chatbot Assistant for Students

Contact Info

Product

Resources

About