Text2Concept: Concept Activation Vectors Directly from Text

Moayeri, Mazda; Keivan, Rezaei,; Sanjabi, Maziar; Feizi, Soheil

doi:10.1109/cvprw59228.2023.00386

Cited by 2 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, recent advancements have seen the introduction of invertible CAVs to interpret vision models with non-negative CAVs [78], and Text2Concept which extends the CAV framework to NLP, allowing the extraction of interpretable vectors from text [79]. Concept Activation Regions (CARs) further generalize this framework by using a collection of CAVs to define the decision boundaries in DNN models [80].…”

Section: Concept Activation Vectors and Derivativesmentioning

confidence: 99%

Classification of Kinematic Data Using Explainable Artificial Intelligence (XAI) for Smart Motion

Yong¹

2021

Explainable Artificial Intelligence for Smart Cities

View full text Add to dashboard Cite

Given the complexity and lack of transparency in deep neural networks (DNNs), extensive efforts have been made to make these systems more interpretable or explain their behaviors in accessible terms. Unlike most reviews, which focus on algorithmic and model-centric perspectives, this work takes a "data-centric" view, examining how data collection, processing, and analysis contribute to explainable AI (XAI). We categorize existing work into three categories subject to their purposes: interpretations of deep models, referring to feature attributions and reasoning processes that correlate data points with model outputs; influences of training data, examining the impact of training data nuances, such as data valuation and sample anomalies, on decision-making processes; and insights of domain knowledge, discovering latent patterns and fostering new knowledge from data and models to advance social values and scientific discovery. Specifically, we distill XAI methodologies into data mining operations on training and testing data across modalities, such as images, text, and tabular data, as well as on training logs, checkpoints, models and other DNN behavior descriptors. In this way, our study offers a comprehensive, data-centric examination of XAI from a lens of data mining methods and applications.

show abstract

Section: Concept Activation Vectors and Derivativesmentioning

confidence: 99%

Classification of Kinematic Data Using Explainable Artificial Intelligence (XAI) for Smart Motion

Yong¹

2021

Explainable Artificial Intelligence for Smart Cities

View full text Add to dashboard Cite

show abstract

“…In the past few years, image-text contrastively pre-trained multimodal models such as CLIP (Radford et al 2021a) have shown tremendous ability to perform zero-shot classification (Mu et al 2021;Minderer et al 2022), imagetext retrieval (Diwan et al 2022;Thrush et al 2022) and image-captioning (Yu et al 2022;Li et al 2022;Mokady, Hertz, and Bermano 2021). These contrastive models are also used as a part of various state-of-the-art pipelines for downstream tasks such as segmentation (Wang et al 2021;Lüddecke and Ecker 2021), object-detection (Minderer et al 2022;Zhong et al 2021) and model interpretability (Moayeri et al 2023). However, recent works have shown that these models fail on visio-linguistic reasoning tasks, for example identifying the relative position between objects in an image.…”

Section: Introductionmentioning

confidence: 99%

Research in information systems at the University of Maryland

Basu

Hevner

Twenty-Third Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Few-shot classification (FSC) entails learning novel classes given only a few examples per class after a pretraining (or meta-training) phase on a set of base classes. Recent works have shown that simply fine-tuning a pretrained Vision Transformer (ViT) on new test classes is a strong approach for FSC. Fine-tuning ViTs, however, is expensive in time, compute and storage. This has motivated the design of parameter efficient fine-tuning (PEFT) methods which fine-tune only a fraction of the Transformer's parameters. While these methods have shown promise, inconsistencies in experimental conditions make it difficult to disentangle their advantage from other experimental factors including the feature extractor architecture, pre-trained initialization and fine-tuning algorithm, amongst others. In our paper, we conduct a large-scale, experimentally consistent, empirical analysis to study PEFTs for few-shot image classification. Through a battery of over 1.8k controlled experiments on large-scale few-shot benchmarks including META-DATASET (MD) and ORBIT, we uncover novel insights on PEFTs that cast light on their efficacy in finetuning ViTs for few-shot classification. Through our controlled empirical study, we have two main findings: (i) Finetuning just the LayerNorm parameters (which we call LN-TUNE) during few-shot adaptation is an extremely strong baseline across ViTs pre-trained with both self-supervised and supervised objectives, (ii) For self-supervised ViTs, we find that simply learning a set of scaling parameters for each attention matrix (which we call ATTNSCALE) along with a domain-residual adapter (DRA) module leads to state-of-the-art performance (while being ∼ 9× more parameter-efficient) on MD. Our extensive empirical findings set strong baselines and call for rethinking the current design of PEFT methods for FSC.

show abstract

Text2Concept: Concept Activation Vectors Directly from Text

Cited by 2 publications

References 14 publications

Classification of Kinematic Data Using Explainable Artificial Intelligence (XAI) for Smart Motion

Classification of Kinematic Data Using Explainable Artificial Intelligence (XAI) for Smart Motion

Research in information systems at the University of Maryland

Contact Info

Product

Resources

About