What Are the Users’ Needs? Design of a User-Centered Explainable Artificial Intelligence Diagnostic System

He, Xiaohua; Hong, Yeyi; Xi, Zheng; Zhang, Yong

doi:10.1080/10447318.2022.2095093

Cited by 18 publications

(29 citation statements)

References 74 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Publication types included 23 (72%) journal papers ( Tables 2 and 3 ) [ 10 , 12 , 15 , 17 , 34 , 37 , 39 , 41 , 43 , 45 - 49 , 52 - 62 ] and 9 (28%) conference papers ( Table 4 ) [ 13 , 35 , 36 , 38 , 40 , 42 , 44 , 50 , 51 ]. Study designs included quantitative research (22/32, 69%) [ 12 , 13 , 15 , 34 , 37 , 39 , 40 , 42 - 44 , 47 - 52 , 54 , 55 , 57 - 59 , 61 ], qualitative research (2/32, 6%) [ 35 , 60 ], and mixed methods studies (4/32, 12%) [ 38 , 41 , 45 , 46 ], in addition to systematic reviews (4/32, 12%) [ 17 , 36 , 53 , 56 ]. Most studies chose general practice (8/32, 25%) [ 34 , 37 , 40 , 41 , 46 , 49 , 54 , 55 ] as the target medical field.…”

Section: Resultsmentioning

confidence: 99%

Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review

He,

Zheng,

Ding

2023

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background Direct-to-consumer (DTC) health care artificial intelligence (AI) apps hold the potential to bridge the spatial and temporal disparities in health care resources, but they also come with individual and societal risks due to AI errors. Furthermore, the manner in which consumers interact directly with health care AI is reshaping traditional physician-patient relationships. However, the academic community lacks a systematic comprehension of the research overview for such apps. Objective This paper systematically delineated and analyzed the characteristics of included studies, identified existing barriers and design recommendations for DTC health care AI apps mentioned in the literature and also provided a reference for future design and development. Methods This scoping review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews guidelines and was conducted according to Arksey and O’Malley’s 5-stage framework. Peer-reviewed papers on DTC health care AI apps published until March 27, 2023, in Web of Science, Scopus, the ACM Digital Library, IEEE Xplore, PubMed, and Google Scholar were included. The papers were analyzed using Braun and Clarke’s reflective thematic analysis approach. Results Of the 2898 papers retrieved, 32 (1.1%) covering this emerging field were included. The included papers were recently published (2018-2023), and most (23/32, 72%) were from developed countries. The medical field was mostly general practice (8/32, 25%). In terms of users and functionalities, some apps were designed solely for single-consumer groups (24/32, 75%), offering disease diagnosis (14/32, 44%), health self-management (8/32, 25%), and health care information inquiry (4/32, 13%). Other apps connected to physicians (5/32, 16%), family members (1/32, 3%), nursing staff (1/32, 3%), and health care departments (2/32, 6%), generally to alert these groups to abnormal conditions of consumer users. In addition, 8 barriers and 6 design recommendations related to DTC health care AI apps were identified. Some more subtle obstacles that are particularly worth noting and corresponding design recommendations in consumer-facing health care AI systems, including enhancing human-centered explainability, establishing calibrated trust and addressing overtrust, demonstrating empathy in AI, improving the specialization of consumer-grade products, and expanding the diversity of the test population, were further discussed. Conclusions The booming DTC health care AI apps present both risks and opportunities, which highlights the need to explore their current status. This paper systematically summarized and sorted the characteristics of the included studies, identified existing barriers faced by, and made future design recommendations for such apps. To the best of our knowledge, this is the first study to systematically summarize and categorize academic research on these apps. Future studies conducting the design and development of such systems could refer to the results of this study, which is crucial to improve the health care services provided by DTC health care AI apps.

show abstract

Section: Resultsmentioning

confidence: 99%

Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review

He,

Zheng,

Ding

2023

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…Because underlying methods are generally not transparent, users of AI may overtrust or undertrust the system, leading to errors (Okamura & Yamada, 2020). Explainability methods can be complex for end users and have a similar result of overtrust (Ghassemi et al, 2021;He et al, 2022). Learning systems can be dynamic, such that performance may not be consistent.…”

Section: Unique Features Of Ai Technology That May Impact Well-beingmentioning

confidence: 99%

Six Human-Centered Artificial Intelligence Grand Challenges

Garibay

Winslow

Andolina

et al. 2023

International Journal of Human–Computer Interaction

153

View full text Add to dashboard Cite

Widespread adoption of artificial intelligence (AI) technologies is substantially affecting the human condition in ways that are not yet well understood. Negative unintended consequences abound including the perpetuation and exacerbation of societal inequalities and divisions via algorithmic decision making. We present six grand challenges for the scientific community to create AI technologies that are humancentered, that is, ethical, fair, and enhance the human condition. These grand challenges are the result of an international collaboration across academia, industry and government and represent the consensus views of a group of 26 experts in the field of human-centered artificial intelligence (HCAI). In essence, these challenges advocate for a human-centered approach to AI that (1) is centered in human wellbeing, (2) is designed responsibly, (3) respects privacy, (4) follows human-centered design principles, ( 5) is subject to appropriate governance and oversight, and (6) interacts with individuals while respecting human's cognitive capacities. We hope that these challenges and their associated research directions serve as a call for action to conduct research and development in AI that serves as a force multiplier towards more fair, equitable and sustainable societies.

show abstract

“…In addition to the nine categories of Liao et al [20], the question category transparency was added on a theoretical basis [17]. He et al [12] developed a user needs library based on the XAIQB, which they adapted to a medical context based on their literature review. They then designed an XAI prototype for the medical domain and analyzed the needs and preferences of consumer users with respect to explanations [12].…”

Section: Explainable Ai Question Bankmentioning

confidence: 99%

“…He et al [12] developed a user needs library based on the XAIQB, which they adapted to a medical context based on their literature review. They then designed an XAI prototype for the medical domain and analyzed the needs and preferences of consumer users with respect to explanations [12]. However, to the best of our knowledge, there is no study that empirically evaluated the XAIQB and its applicability for end-users directly.…”

Section: Explainable Ai Question Bankmentioning

confidence: 99%

Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank

Sipos,

Schäfer,

Glinka

et al. 2023

Mensch Und Computer 2023

View full text Add to dashboard Cite

Explainable Artificial Intelligence (XAI) is concerned with making the decisions of AI systems interpretable to humans. Explanations are typically developed by AI experts and focus on algorithmic transparency and the inner workings of AI systems. Research has shown that such explanations do not meet the needs of users who do not have AI expertise. As a result, explanations are often ineffective in making system decisions interpretable and understandable. We aim to strengthen a socio-technical view of AI by following a Human-Centered Explainable Artificial Intelligence (HC-XAI) approach, which investigates the explanation needs of end-users (i.e., subject matter experts and lay users) in specific usage contexts. One of the most influential works in this area is the XAI Question Bank (XAIQB) by Liao et al. The authors propose a set of questions that end-users might ask when using an AI system, which in turn is intended to help developers and designers identify and address explanation needs. Although the XAIQB is widely referenced, there are few reports of its use in practice. In particular, it is unclear to what extent the XAIQB sufficiently captures the explanation needs of end-users and what potential problems exist in the practical application of the XAIQB. To explore these open questions, we used the XAIQB as the basis for analyzing 12 think-aloud software explorations with subject matter experts, i.e., art historians. We investigated the suitability of the XAIQB as a tool for identifying explanation needs in a specific usage context. Our analysis revealed a number of explanation needs that were missing from the question bank, but that emerged repeatedly as our study participants interacted with an AI system. We also found that some of the XAIQB questions were difficult to distinguish and required interpretation during use. Our contribution is an extension of the XAIQB with 11 new questions. In addition, we have expanded the descriptions of all new and existing questions to facilitate their use. We hope that this extension will enable HCI researchers and practitioners to use the XAIQB in practice and may provide a basis for future studies on the identification of explanation needs in different contexts.CCS Concepts: • Human-centered computing → HCI design and evaluation methods.

show abstract

What Are the Users’ Needs? Design of a User-Centered Explainable Artificial Intelligence Diagnostic System

Cited by 18 publications

References 74 publications

Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review

Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review

Six Human-Centered Artificial Intelligence Grand Challenges

Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank

Contact Info

Product

Resources

About