The Role of Human Knowledge in Explainable AI

Abstract

As the performance and complexity of machine learning models have grown significantly over the last years, there has been an increasing need to develop methodologies to describe their behaviour. Such a need has mainly arisen due to the widespread use of black-box models, i.e., high-performing models whose internal logic is challenging to describe and understand. Therefore, the machine learning and AI field is facing a new challenge: making models more explainable through appropriate techniques. The final goal of an explainability method is to faithfully describe the behaviour of a (black-box) model to users who can get a better understanding of its logic, thus increasing the trust and acceptance of the system. Unfortunately, state-of-the-art explainability approaches may not be enough to guarantee the full understandability of explanations from a human perspective. For this reason, human-in-the-loop methods have been widely employed to enhance and/or evaluate explanations of machine learning models. These approaches focus on collecting human knowledge that AI systems can then employ or involving humans to achieve their objectives (e.g., evaluating or improving the system). This article aims to present a literature overview on collecting and employing human knowledge to improve and evaluate the understandability of machine learning models through human-in-the-loop approaches. Furthermore, a discussion on the challenges, state-of-the-art, and future trends in explainability is also provided.

Integrating Artificial Intelligence in Disease Diagnosis, Treatment, and Formulation Development: A Review

“…While they lack a formal mathematical definition, efforts have been made to differentiate these two concepts [32,33]. Explainability refers to the ability to communicate with humans in understandable terms [34], whereas interpretability concerns the ability to comprehend the reasoning behind a model's outputs [35]. Explainable AI (XAI) is used in health care to communicate transparent and understandable automated decision-making to impacted patients [36].…”

Navigating the Landscape of Robust and Secure Artificial Intelligence: A Comprehensive Literature Review

“…First, there is an indication to fully exploit the research findings of security vulnerability coordination as a way of accommodating AI technologies for connecting with new vulnerabilities. With AI increasingly being used in real-world systems, developing strategies to understand and resolve its inevitable peculiar security implications constitutes one of the most pressing issues [11], [12]. Red teaming presents a second area of benefit: improving red teaming capabilities is an effective approach.…”

“…Teams must steer by experimentation to 'fingerprint' such interdependencies, on the one hand, and devise contingent plans for unexpected behaviors caused by changes within systems. This throws light on the need for a non-linear perspective analysis that embodies the complex interdependencies within the AI [10], [11]. In the process of organizing their utilization in high-risk environments such as managing under-control highway systems or power grids, it should ensure the adaptability and safety of AI Systems.…”

Electronic Medical Record Data Analysis and Prediction of Stroke Disease Using Explainable Artificial Intelligence (XAI)

“…As previously explained, XAI is an AI model with more transparent processes than deep learning [7]. The decision-making process in XAI is also based on mathematical calculations that are easier for users to understand [11].…”

“…It cannot be denied that the Deep Learning used in previous studies has a high level of accuracy. However, this high level of accuracy is not accompanied by the transparency of data processing in Deep Learning [7]. Data processing that occurs in deep learning tends to be difficult for users to understand because of the complex architecture of its constituents [8].…”