NGAME: Negative Mining-aware Mini-batching for Extreme Classification

Dahiya, Kunal; Gupta, Nilesh; Saini, Deepak; Soni, Akshay; Wang, Yajun; Dave, Kushal; Jiao, Jian; K, Gururaj; Dey, Prasenjit; Singh, Amit Prakash; Hada, Deepesh; Jain, Vidit; Bhawna, Paliwal,; Mittal, Anshul; Mehta, Sonu; Ramjee, Ramachandran; Agarwal, Sumeet; Kar, Purushottam; Varma, Manik

doi:10.1145/3539597.3570392

Cited by 9 publications

(22 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MUFIN seeks to obtain an embedding xi ∈ R D for every datapoint X i and a classifier w l ∈ R D for every label l ∈ [L] so that w ⊤ l xi is indicative of the relevance of label l to datapoint i. Datapoints and labels each having multiple descriptors i.e. m i , m l ≥ 1 present opportunities to ease this process: (1) The neural architecture used to obtain datapoint embeddings xi can also be used to obtain label embeddings ẑl that can serve as a convenient warm start when learning w l and has been found to accelerate training in XC methods [5,29].…”

Section: Mufin Multimodal Extreme Classificationmentioning

confidence: 99%

“…Minibatches B were created over labels instead of datapoints by sampling labels randomly. This was observed to improve performance over rare labels [5,28]. Training with respect to all N datapoints for each label would have resulted in an Ω (N L) epoch complexity that is infeasible when N, L are both in the millions.…”

Section: = X1mentioning

confidence: 99%

“…(2) Bid-query prediction [5] where an advertisement with visual and textual descriptions has to be tagged with the list of user queries most likely to lead to a click on that ad.…”

Section: Introductionmentioning

confidence: 99%

“…categorization is done entirely using embeddings of datapoints and categories obtained from some neural architecture. However, XC research has shown that training classifiers alongside embedding architectures can offer improved results [3,5,41]. However, existing XC research focuses mostly on text-based categorization.…”

Section: Introductionmentioning

confidence: 99%

“…Bridging this gap requires architectures that offer multi-modal embeddings sufficiently expressive to perform categorization over millions of classes. Also required are routines that can train classifiers over millions of classes and still offer predictions in milliseconds as demanded by real-time applications [3,5,14]. This is usually possible only if training and inference scale logarithmically with the number of labels.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Multi-modal Extreme Classification

Mittal

Dahiya

Malani

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

This paper develops the MUFIN technique for extreme classification (XC) tasks with millions of labels where datapoints and labels are endowed with visual and textual descriptors. Applications of MUFIN to product-to-product recommendation and bid query prediction over several millions of products are presented. Contemporary multi-modal methods frequently rely on purely embedding-based methods. On the other hand, XC methods utilize classifier architectures to offer superior accuracies than embeddingonly methods but mostly focus on text-based categorization tasks. MUFIN bridges this gap by reformulating multimodal categorization as an XC problem with several millions of labels. This presents the twin challenges of devel-

show abstract

Section: Mufin Multimodal Extreme Classificationmentioning

confidence: 99%

Section: = X1mentioning

confidence: 99%

“…(2) Bid-query prediction [5] where an advertisement with visual and textual descriptions has to be tagged with the list of user queries most likely to lead to a click on that ad.…”

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Multi-modal Extreme Classification

Mittal

Dahiya

Malani

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

show abstract

Meta-classifier free negative sampling for extreme multilabel classification

Qaraei,

Babbar

2023

Mach Learn

View full text Add to dashboard Cite

Negative sampling is a common approach for making the training of deep models in classification problems with very large output spaces, known as extreme multilabel classification (XMC) problems, tractable. Negative sampling methods aim to find per instance negative labels with higher scores, known as hard negatives, and limit the computations of the negative part of the loss to these labels. Two well-known methods for negative sampling in XMC models are meta-classifier-based and Maximum Inner product Search (MIPS)-based adaptive methods. Owing to their good prediction performance, methods which employ a meta classifier are more common in contemporary XMC research. On the flip side, they need to train and store the meta classifier (apart from the extreme classifier), which can involve millions of additional parameters. In this paper, we focus on the MIPS-based methods for negative sampling. We highlight two issues which may prevent deep models trained by these methods to undergo stable training. First, we argue that using hard negatives excessively from the beginning of training leads to unstable gradient. Second, we show that when all the negative labels in a MIPS-based method are restricted to only those determined by MIPS, training is sensitive to the length of intervals for pre-processing the weights in the MIPS method. To mitigate the aforementioned issues, we propose to limit the labels selected by MIPS to only a few and sample the rest of the needed labels from a uniform distribution. We show that our proposed MIPS-based negative sampling can reach the performance of LightXML, a transformer-based model trained by a meta classifier, while there is no need to train and store any additional classifier. The code for our experiments is available at https://github.com/xmc-aalto/mips-negative-sampling.

show abstract

MatchXML: An Efficient Text-Label Matching Framework for Extreme Multi-Label Text Classification

Ye,

Sunderraman,

2024

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

NGAME: Negative Mining-aware Mini-batching for Extreme Classification

Cited by 9 publications

References 34 publications

Multi-modal Extreme Classification

Multi-modal Extreme Classification

Meta-classifier free negative sampling for extreme multilabel classification

MatchXML: An Efficient Text-Label Matching Framework for Extreme Multi-Label Text Classification

Contact Info

Product

Resources

About