Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence

Abstract

Recent years have seen a tremendous growth in Artificial Intelligence (AI)-based methodological development in a broad range of domains. In this rapidly evolving field, large number of methods are being reported using machine learning (ML) and Deep Learning (DL) models. Majority of these models are inherently complex and lacks explanations of the decision making process causing these models to be termed as 'Black-Box'. One of the major bottlenecks to adopt such models in mission-critical application domains, such as banking, e-commerce, healthcare, and public services and safety, is the difficulty in interpreting them. Due to the rapid proleferation of these AI models, explaining their learning and decision making process are getting harder which require transparency and easy predictability. Aiming to collate the current state-of-the-art in interpreting the black-box models, this study provides a comprehensive analysis of the explainable AI (XAI) models. To reduce false negative and false positive outcomes of these back-box models, finding flaws in them is still difficult and inefficient. In this paper, the development of XAI is reviewed meticulously through careful selection and analysis of the current state-of-the-art of XAI research. It also provides a comprehensive and in-depth evaluation of the XAI frameworks and their efficacy to serve as a starting point of XAI for applied and theoretical researchers. Towards the end, it highlights emerging and critical issues pertaining to XAI research to showcase major, model-specific trends for better explanation, enhanced transparency, and improved prediction accuracy.

Using driver monitoring to estimate readiness in automation: A conceptual model based on simulator experimental data

“…This work contributes to the state of the art on readiness assessment models (e.g., Kim et al, 2022;Mioch, 2017;Mariajoseph et al, 2020;Baek et al 2018;Deo & Trivedi, 2020;Du et al 2021), by proposing an applicable tool to estimate driver readiness, without relying on subjective assessment of readiness, as a ground truth. Another advantage of the proposed solution is the mechanistic nature of the model, since it allows researchers to directly observe the relationship between driver readiness estimation variables and the behaviour prediction, without relying on black box techniques (Hassija et al, 2024). The last contribution of this work is that the proposed methodology accounts for scenario variability, assuming that different safety-critical scenarios might require different readiness threshold values.…”

“…However, the parameter used as the ground truth on the training dataset of this study was a manual annotation of video data, where experts described drivers' state as "good" or "bad", based on their subjective interpretation (Deo & Trivedi, 2020;Du et al 2021). Although machine learning models can provide accurate predictors for the drivers' annotated state, according to Hassija et al (2024), as the artificial intelligence-based (AI) models become more complex, the relationship between the model's predictors and the predicted variable becomes too complex to be explained. Therefore, machine learning approaches provide limited value as a tool to theoretically define readiness, and the safety thresholds required by DSM systems.…”

“…Therefore, machine learning approaches provide limited value as a tool to theoretically define readiness, and the safety thresholds required by DSM systems. Although explainable AI techniques may provide a promising solution for this problem, the state of the art in the field of explainable AI still faces challenges such as balancing transparency and privacy, addressing the diversity of user needs, and creating effective explanations for complex models (Hassija et al, 2024).…”

Towards an Autism Spectrum Disorder (ASD) care agent for social interaction

“…However, people don't (fully) trust the recommendations of a CNN, as they are considered 'black-boxes' [13]. 'Black-box' means that the CNNs hide the recommendation generation methodology from the end users, resulting in the need of eXplainable artificial intelligence (XAI) [14]. XAI aims at making the recommendations of a DL algorithm understandable to an end user, so that he/ she can trust the system.…”

“…XAI aims at making the recommendations of a DL algorithm understandable to an end user, so that he/ she can trust the system. The use of XAI has been greatly advocated in the healthcare [14].…”