The MAMe dataset: on the relevance of high resolution and variable shape image properties

Parés, Ferran; Arias-Duart, Anna; Garcia-Gasulla, Darío; Campo-Francés, Gema; Viladrich, Nina; Ayguadé, Eduard; Labarta, Jesús

doi:10.1007/s10489-021-02951-w

Cited by 2 publications

(2 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Specifically, two different architectures: VGG16 [21] and ResNet-18 [7]. The models were fine-tuned on the following datasets: the Dogs vs. Cats 1 dataset (binary problem), the MAMe [13] dataset (29 categories of art mediums and techniques) and the MIT67 [14] dataset (67 indoor scenes). The first two datasets were combined with pre-training on ImageNet [18] and the third one with the Places365-Standard dataset [24].…”

Section: Methodsmentioning

confidence: 99%

A Confusion Matrix for Evaluating Feature Attribution Methods

Arias-Duart

Mariotti

Garcia-Gasulla

et al. 2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

The increasing use of deep learning models in critical areas of computer vision and the consequent need for insights into model behaviour have led to the development of numerous feature attribution methods. However, these attributions must be both meaningful and plausible to end-users, which is not always the case. Recent research has emphasized the importance of faithfulness in attributions, as plausibility without faithfulness can result in misleading explanations and incorrect decisions. In this work, we propose a novel approach to evaluate the faithfulness of feature attribution methods by constructing an 'Attribution Confusion Matrix', which allows us to leverage a wide range of existing metrics from the traditional confusion matrix. This approach effectively introduces multiple evaluation measures for faithfulness in feature attribution methods in a unified and consistent framework. We demonstrate the effectiveness of our approach on various datasets, attribution methods, and models, emphasizing the importance of faithfulness in generating plausible and reliable explanations while also illustrating the distinct behaviour of different feature attribution methods.

show abstract

Section: Methodsmentioning

confidence: 99%

A Confusion Matrix for Evaluating Feature Attribution Methods

Arias-Duart

Mariotti

Garcia-Gasulla

et al. 2023

2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…• Architectures: AlexNet [15], VGG16 [16] and ResNet-18 [17]. • Datasets: the Dogs vs. Cats 4 , the Museum Artworks Medium dataset (MAMe) [18], the MIT67 [19] and the ILSVRC 2012 [20] (hereafter ImageNet).…”

Section: B Modelsmentioning

confidence: 99%

Focus! Rating XAI Methods and Finding Biases

Arias-Duart

Parés

Garcia-Gasulla

et al. 2022

2022 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)

View full text Add to dashboard Cite

AI explainability improves the transparency and trustworthiness of models. However, in the domain of images, where deep learning has succeeded the most, explainability is still poorly assessed. In the field of image recognition many feature attribution methods have been proposed with the purpose of explaining a model's behavior using visual cues. However, no metrics have been established so far to assess and select these methods objectively. In this paper we propose a consistent evaluation score for feature attribution methods-the Focus-designed to quantify their coherency to the task. While most previous work adds outof-distribution noise to samples, we introduce a methodology to add noise from within the distribution. This is done through mosaics of instances from different classes, and the explanations these generate. On those, we compute a visual pseudo-precision metric, Focus. First, we show the robustness of the approach through a set of randomization experiments. Then we use Focus to compare six popular explainability techniques across several CNN architectures and classification datasets. Our results find some methods to be consistently reliable (LRP, GradCAM), while others produce class-agnostic explanations (SmoothGrad, IG). Finally we introduce another application of Focus, using it for the identification and characterization of biases found in models. This empowers bias-management tools, in another small step towards trustworthy AI.

show abstract

The MAMe dataset: on the relevance of high resolution and variable shape image properties

Cited by 2 publications

References 29 publications

A Confusion Matrix for Evaluating Feature Attribution Methods

A Confusion Matrix for Evaluating Feature Attribution Methods

Focus! Rating XAI Methods and Finding Biases

Contact Info

Product

Resources

About