New Intent Discovery with Pre-training and Contrastive Learning

Zhang, Yuwei; Zhang, Haode; Zhan, Li-Ming; Wu, Xiao-Ming; Lam, Albert Y. S.

doi:10.48550/arxiv.2205.12914

Cited by 3 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of current methods are based on the pre-training and fine-tuning paradigm to transfer knowledge implicitly by initializing model parameters (Zhang et al 2021a). For example, Vaze et al (2022) proposed to use contrastive learning to pre-train their model, and Zhang et al (2022) combined supervised learning and masked language modeling to initialize their model. However, we think this paradigm is sub-optimal for GCD because models pre-trained on labeled data tend to be biased towards known categories.…”

Section: Transfer Learningmentioning

confidence: 99%

Transfer and Alignment Network for Generalized Category Discovery

An,

Tian,

Shi

et al. 2024

AAAI

View full text Add to dashboard Cite

Generalized Category Discovery (GCD) is a crucial real-world task that aims to recognize both known and novel categories from an unlabeled dataset by leveraging another labeled dataset with only known categories. Despite the improved performance on known categories, current methods perform poorly on novel categories. We attribute the poor performance to two reasons: biased knowledge transfer between labeled and unlabeled data and noisy representation learning on the unlabeled data. The former leads to unreliable estimation of learning targets for novel categories and the latter hinders models from learning discriminative features. To mitigate these two issues, we propose a Transfer and Alignment Network (TAN), which incorporates two knowledge transfer mechanisms to calibrate the biased knowledge and two feature alignment mechanisms to learn discriminative features. Specifically, we model different categories with prototypes and transfer the prototypes in labeled data to correct model bias towards known categories. On the one hand, we pull instances with known categories in unlabeled data closer to these prototypes to form more compact clusters and avoid boundary overlap between known and novel categories. On the other hand, we use these prototypes to calibrate noisy prototypes estimated from unlabeled data based on category similarities, which allows for more accurate estimation of prototypes for novel categories that can be used as reliable learning targets later. After knowledge transfer, we further propose two feature alignment mechanisms to acquire both instance- and category-level knowledge from unlabeled data by aligning instance features with both augmented features and the calibrated prototypes, which can boost model performance on both known and novel categories with less noise. Experiments on three benchmark datasets show that our model outperforms SOTA methods, especially on novel categories. Theoretical analysis is provided for an in-depth understanding of our model in general. Our code and data are available at https://github.com/Lackel/TAN.

show abstract

Section: Transfer Learningmentioning

confidence: 99%

Transfer and Alignment Network for Generalized Category Discovery

An,

Tian,

Shi

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…However, these methods only focus on the scenario where known and novel categories are of the same granularity. To discover fine-grained categories, a novel task called Fine-grained Category Discovery under Coarsegrained supervision (FCDC) was proposed by An et al (2022a). They also proposed a weighted selfcontrastive strategy to acquire fine-grained knowledge.…”

Section: Novel Category Discoverymentioning

confidence: 99%

“…Li et al (2020) proposed to utilize prototypes learned by clustering as their positive keys. Furthermore, An et al (2022a) 3 Method…”

Section: Contrastive Learningmentioning

confidence: 99%

“…However, labelling fine-grained categories can be time-consuming and labour-intensive since it requires more expert knowledge. To get out of this dilemma, a novel task called Fine-grained Category Discovery under Coarse-grained supervision (FCDC) was recently proposed by An et al (2022a). Taking Figure 1 Right as an example, FCDC aims at discovering fine-grained categories (e.g., Desktop and Tennis) using only coarse-grained (e.g., Computer and Sports) labeled data which are easier and cheaper to annotate.…”

Section: Introductionmentioning

confidence: 99%

“…To solve the FCDC task, previous methods mainly focus on instance-level discrimination to learn low-level features through contrastive learning (An et al, 2022a;Bukchin et al, 2021). Despite the improved performance, these instancebased methods fail to encode cluster-level semantic structures of data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery

An,

Tian,

Shi

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Discovering fine-grained categories from coarsely labeled data is a practical and challenging task, which can bridge the gap between the demand for fine-grained analysis and the high annotation cost. Previous works mainly focus on instance-level discrimination to learn low-level features, but ignore semantic similarities between data, which may prevent these models learning compact cluster representations. In this paper, we propose Denoised Neighborhood Aggregation (DNA), a self-supervised framework that encodes semantic structures of data into the embedding space. Specifically, we retrieve k-nearest neighbors of a query as its positive keys to capture semantic similarities between data and then aggregate information from the neighbors to learn compact cluster representations, which can make fine-grained categories more separatable. However, the retrieved neighbors can be noisy and contain many false-positive keys, which can degrade the quality of learned embeddings. To cope with this challenge, we propose three principles to filter out these false neighbors for better representation learning. Furthermore, we theoretically justify that the learning objective of our framework is equivalent to a clustering loss, which can capture semantic similarities between data to form compact fine-grained clusters. Extensive experiments on three benchmark datasets show that our method can retrieve more accurate neighbors (21.31% accuracy improvement) and outperform state-of-the-art models by a large margin (average 9.96% improvement on three metrics). Our code and data are available at https://github.com/Lackel/DNA.

show abstract

A Diffusion Weighted Graph Framework for New Intent Discovery

Shi,

An,

Tian

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

New Intent Discovery (NID) aims to recognize both new and known intents from unlabeled data with the aid of limited labeled data containing only known intents. Without considering structure relationships between samples, previous methods generate noisy supervisory signals which cannot strike a balance between quantity and quality, hindering the formation of new intent clusters and effective transfer of the pre-training knowledge. To mitigate this limitation, we propose a novel Diffusion Weighted Graph Framework (DWGF) to capture both semantic similarities and structure relationships inherent in data, enabling more sufficient and reliable supervisory signals. Specifically, for each sample, we diffuse neighborhood relationships along semantic paths guided by the nearest neighbors for multiple hops to characterize its local structure discriminately. Then, we sample its positive keys and weigh them based on semantic similarities and local structures for contrastive learning. During inference, we further propose Graph Smoothing Filter (GSF) to explicitly utilize the structure relationships to filter high-frequency noise embodied in semantically ambiguous samples on the cluster boundary. Extensive experiments show that our method outperforms state-of-the-art models on all evaluation metrics across multiple benchmark datasets. Code and data are available at https://github.com/yibai-shi/DWGF.

show abstract

New Intent Discovery with Pre-training and Contrastive Learning

Cited by 3 publications

References 0 publications

Transfer and Alignment Network for Generalized Category Discovery

Transfer and Alignment Network for Generalized Category Discovery

DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery

A Diffusion Weighted Graph Framework for New Intent Discovery

Contact Info

Product

Resources

About