Black-Box Model Explained Through an Assessment of Its Interpretable Features

Ventura, Francesco; Cerquitelli, Tania; Giacalone, Francesco

doi:10.1007/978-3-030-00063-9_15

Cited by 10 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An exhaustive overview of the existing XAI techniques for NLP models, applied in different contexts, such as social networks, medical, and cybersecurity, is presented in [34]. Many works exploit feature-perturbation strategies in the explanation process, analyzing the model reactions to produce prediction-local explanations, like in [2,28,32,35,40,48]. This straightforward idea is very powerful but requires a careful selection of the input features to be perturbed.…”

Section: Domain-specific Approachesmentioning

confidence: 99%

“…T-EBAnO proposes a new local and global explanation process for state-of-the-art deep NLP models. By exploiting a perturbation-based strategy similar to that described in [48], which was successfully tailored to image data, T-EBAnO fills in the gap of missing customized solutions for explaining deep NLP models by introducing a totally redesigned architecture and experimental section. Specifically, we introduce (i) a novel feature extraction process specifically tailored to textual data and deep natural language models, (ii) new perturbation strategies, (iii) an improved version of the index proposed in [48] able to quantify the influence of the input feature over local predictions tested in a new domain (NLP), and (iv) novel class-based global explanations, besides extending the experiments to new models and use cases, and presenting a human evaluation of the exploited index and proposed explanations.…”

Section: Task-specific Approachesmentioning

confidence: 99%

“…To access the inner knowledge of the network, this technique needs to know the inner details of the model under analysis. However, the process can be easily adapted to be compliant with different deep architectures (e.g., as reported in [48]) and their hidden Then, separately for each feature extraction method, T-EBAnO tests pairwise combinations of features to create larger groups of tokens corresponding to more complex concepts. For instance, for Part-of-Speech, it creates a feature with the combination of Adjectives and Verbs, Adjectives and Nouns, etc.…”

Section: Interpretable Feature Extractionmentioning

confidence: 99%

“…After the extraction of the interpretable feature sets, a perturbation phase is performed by introducing noise and consequently assessing the impact of the perturbed features on the model outcomes. Adding noise to the model input is a well-known technique adopted by different state-of-the-art approaches [2,32,40,48] to study the model behavior through the effects on the outcomes. Different input data types require different perturbation strategies.…”

Section: Interpretable Feature Perturbationmentioning

confidence: 99%

See 3 more Smart Citations

Trusting deep learning natural-language models via local and global explanations

et al. 2022

Self Cite

View full text Add to dashboard Cite

Despite the high accuracy offered by state-of-the-art deep natural-language models (e.g., LSTM, BERT), their application in real-life settings is still widely limited, as they behave like a black-box to the end-user. Hence, explainability is rapidly becoming a fundamental requirement of future-generation data-driven systems based on deep-learning approaches. Several attempts to fulfill the existing gap between accuracy and interpretability have been made. However, robust and specialized eXplainable Artificial Intelligence solutions, tailored to deep natural-language models, are still missing. We propose a new framework, named T-EBAnO, which provides innovative prediction-local and class-based model-global explanation strategies tailored to deep learning natural-language models. Given a deep NLP model and the textual input data, T-EBAnO provides an objective, human-readable, domain-specific assessment of the reasons behind the automatic decision-making process. Specifically, the framework extracts sets of interpretable features mining the inner knowledge of the model. Then, it quantifies the influence of each feature during the prediction process by exploiting the normalized Perturbation Influence Relation index at the local level and the novel Global Absolute Influence and Global Relative Influence indexes at the global level. The effectiveness and the quality of the local and global explanations obtained with T-EBAnO are proved on an extensive set of experiments addressing different tasks, such as a sentiment-analysis task performed by a fine-tuned BERT model and a toxic-comment classification task performed by an LSTM model. The quality of the explanations proposed by T-EBAnO, and, specifically, the correlation between the influence index and human judgment, has been evaluated by humans in a survey with more than 4000 judgments. To prove the generality of T-EBAnO and its model/task-independent methodology, experiments with other models (ALBERT, ULMFit) on popular public datasets (Ag News and Cola) are also discussed in detail.

show abstract

Section: Domain-specific Approachesmentioning

confidence: 99%

Section: Task-specific Approachesmentioning

confidence: 99%

Section: Interpretable Feature Extractionmentioning

confidence: 99%

Section: Interpretable Feature Perturbationmentioning

confidence: 99%

See 2 more Smart Citations

Trusting deep learning natural-language models via local and global explanations

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…For instance, the work in [62] uses a model-agnostic perturbation-based technique that produces local explanations by training a simpler interpretable model that approximates the prediction locally. Instead, the work in [63] produces local explanations of deep convolutional neural networks by a process of perturbation over interpretable features extracted from the hypercolumns. The authors of [64] propose a model-agnostic explainability technique based on a game-theoretic approach that iteratively removes combinations of inputs features to measure the features' importance.…”

Section: Explainable Artificial Intelligencementioning

confidence: 99%

Attention to Fires: Multi-Channel Deep Learning Models for Wildfire Severity Prediction

et al. 2021

Self Cite

View full text Add to dashboard Cite

Wildfires are one of the natural hazards that the European Union is actively monitoring through the Copernicus EMS Earth observation program which continuously releases public information related to such catastrophic events. Such occurrences are the cause of both short- and long-term damages. Thus, to limit their impact and plan the restoration process, a rapid intervention by authorities is needed, which can be enhanced by the use of satellite imagery and automatic burned area delineation methodologies, accelerating the response and the decision-making processes. In this context, we analyze the burned area severity estimation problem by exploiting a state-of-the-art deep learning framework. Experimental results compare different model architectures and loss functions on a very large real-world Sentinel2 satellite dataset. Furthermore, a novel multi-channel attention-based analysis is presented to uncover the prediction behaviour and provide model interpretability. A perturbation mechanism is applied to an attention-based DS-UNet to evaluate the contribution of different domain-driven groups of channels to the severity estimation problem.

show abstract