A Review on Explainability in Multimodal Deep Neural Nets

Joshi, Gargi; Walambe, Rahee; Kotecha, Ketan

doi:10.1109/access.2021.3070212

Cited by 122 publications

(50 citation statements)

References 177 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The factor of stress affecting any person should not be ignored for too long as it causes health issues, both mental and physical. Furthermore, using the power of artificial intelligence in healthcare that goes beyond our supervision will go a long way [ 46 ].…”

Section: Discussionmentioning

confidence: 99%

Employing Multimodal Machine Learning for Stress Detection

Walambe

Nayak

Bhardwaj

et al. 2021

Journal of Healthcare Engineering

Self Cite

View full text Add to dashboard Cite

In the current information age, the human lifestyle has become more knowledge-oriented, leading to sedentary employment. This has given rise to a number of health and mental disorders. Mental wellness is one of the most neglected, however crucial, aspects of today’s fast-paced world. Mental health issues can, both directly and indirectly, affect other sections of human physiology and impede an individual’s day-to-day activities and performance. However, identifying the stress and finding the stress trend for an individual that may lead to serious mental ailments is challenging and involves multiple factors. Such identification can be achieved accurately by fusing these multiple modalities (due to various factors) arising from a person’s behavioral patterns. Specific techniques are identified in the literature for this purpose; however, very few machine learning-based methods are proposed for such multimodal fusion tasks. In this work, a multimodal AI-based framework is proposed to monitor a person’s working behavior and stress levels. We propose a methodology for efficiently detecting stress due to workload by concatenating heterogeneous raw sensor data streams (e.g., face expressions, posture, heart rate, and computer interaction). This data can be securely stored and analyzed to understand and discover personalized unique behavioral patterns leading to mental strain and fatigue. The contribution of this work is twofold: firstly, proposing a multimodal AI-based strategy for fusion to detect stress and its level and, secondly, identifying a stress pattern over a period of time. We were able to achieve 96.09% accuracy on the test set in stress detection and classification. Further, we were able to reduce the stress scale prediction model loss to 0.036 using these modalities. This work can prove important for the community at large, specifically those working sedentary jobs, to monitor and identify stress levels, especially in current times of COVID-19.

show abstract

Section: Discussionmentioning

confidence: 99%

Employing Multimodal Machine Learning for Stress Detection

Walambe

Nayak

Bhardwaj

et al. 2021

Journal of Healthcare Engineering

Self Cite

View full text Add to dashboard Cite

show abstract

“…Besides, this work lacked in applying some of the techniques such as progressive resizing ( Bhatt, Ganatra & Kotecha, 2021a ), which can be applied to CNNs to carry out imaging-based diagnostics. Furthermore, visual ablation studies ( Bhatt, Ganatra & Kotecha, 2021b ; Joshi, Walambe & Kotecha, 2021 ; Gite et al, 2021 ) can be performed along with deep learning, which will significantly improve the detection of COVID-19 manifestations in the CXR images. Since only a limited number of CXR images are available for COVID-19 infection, out-of-distribution issues may arise, so more data from related distributions is needed for further evaluation.…”

Section: Discussionmentioning

confidence: 99%

KL-MOB: automated COVID-19 recognition using a novel approach based on image enhancement and a modified MobileNet CNN

Taresh

Zhu

Ali

et al. 2021

PeerJ Computer Science

View full text Add to dashboard Cite

The emergence of the novel coronavirus pneumonia (COVID-19) pandemic at the end of 2019 led to worldwide chaos. However, the world breathed a sigh of relief when a few countries announced the development of a vaccine and gradually began to distribute it. Nevertheless, the emergence of another wave of this pandemic returned us to the starting point. At present, early detection of infected people is the paramount concern of both specialists and health researchers. This paper proposes a method to detect infected patients through chest x-ray images by using the large dataset available online for COVID-19 (COVIDx), which consists of 2128 X-ray images of COVID-19 cases, 8,066 normal cases, and 5,575 cases of pneumonia. A hybrid algorithm is applied to improve image quality before undertaking neural network training. This algorithm combines two different noise-reduction filters in the image, followed by a contrast enhancement algorithm. To detect COVID-19, we propose a novel convolution neural network (CNN) architecture called KL-MOB (COVID-19 detection network based on the MobileNet structure). The performance of KL-MOB is boosted by adding the Kullback–Leibler (KL) divergence loss function when trained from scratch. The KL divergence loss function is adopted for content-based image retrieval and fine-grained classification to improve the quality of image representation. The results are impressive: the overall benchmark accuracy, sensitivity, specificity, and precision are 98.7%, 98.32%, 98.82% and 98.37%, respectively. These promising results should help other researchers develop innovative methods to aid specialists. The tremendous potential of the method proposed herein can also be used to detect COVID-19 quickly and safely in patients throughout the world.

show abstract

“…The advantage of multimodality is its ability to extract and combine critical and comprehensive information from a variety of sources, thereby allowing for a far richer representation of the problem at hand. Several applications of multimodal AI were proposed in the literature [215], [216], but despite their outstanding performance, there is a lack of social acceptance due to their black-box nature [217]. Recently, some research was conducted for explaining multimodal AI systems [178], [218]- [222].…”

Section: Multimodal Xaimentioning

confidence: 99%

Explainable Artificial Intelligence for Tabular Data: A Survey

2021

View full text Add to dashboard Cite

Machine learning techniques are increasingly gaining attention due to their widespread use in various disciplines across academia and industry. Despite their tremendous success, many such techniques suffer from the "black-box" problem, which refers to situations where the data analyst is unable to explain why such techniques arrive at certain decisions. This problem has fuelled interest in Explainable Artificial Intelligence (XAI), which refers to techniques that can easily be interpreted by humans. Unfortunately, many of these techniques are not suitable for tabular data, which is surprising given the importance and widespread use of tabular data in critical applications such as finance, healthcare, and criminal justice. Also surprising is the fact that, despite the vast literature on XAI, there are still no survey articles to date that focus on tabular data. Consequently, despite the existing survey articles that cover a wide range of XAI techniques, it remains challenging for researchers working on tabular data to go through all of these surveys and extract the techniques that are suitable for their analysis. Our article fills this gap by providing a comprehensive and up-to-date survey of the XAI techniques that are relevant to tabular data. Furthermore, we categorize the references covered in our survey, indicating the type of the model being explained, the approach being used to provide the explanation, and the XAI problem being addressed. Our article is the first to provide researchers with a map that helps them navigate the XAI literature in the context of tabular data. INDEX TERMSBlack-box models, Explainable Artificial Intelligence, Machine Learning, Model interpretability VOLUME 4, 2016This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2021.3116481, IEEE Access M. Sahakyan et al.: Explainable Artificial Intelligence for Tabular Data: A Survey• Decomposability, reflecting the degree to which each input, parameter, and calculation can be explained intuitively;• Algorithmic transparency, reflecting the degree to which the inner workings of the learning algorithm can be understood.For example, rule-based models [16] are considered transparent since they use a series of if-then rules that can easily be understood without the need for any further explanation. Unlike transparent models, black-box models are those that do not explain their predictions in a way that humans can understand [17]. Examples of black-box models include artificial neural networks [18] and gradient boosting [19]. Although black-box models are hard to interpret by humans, they tend to have higher prediction accuracy compared to their transparent counterparts. This trade-off between accuracy and transparency gives raise to the black-box explanation problem, which involves explaining the rationale behind the decisions made by black-box models. By providing such explanations, one can continue ...

show abstract

A Review on Explainability in Multimodal Deep Neural Nets

Cited by 122 publications

References 177 publications

Employing Multimodal Machine Learning for Stress Detection

Employing Multimodal Machine Learning for Stress Detection

KL-MOB: automated COVID-19 recognition using a novel approach based on image enhancement and a modified MobileNet CNN

Explainable Artificial Intelligence for Tabular Data: A Survey

Contact Info

Product

Resources

About