Convex Surrogates for Unbiased Loss Functions in Extreme Classification With Missing Labels

Abstract

Extreme Classification (XC) refers to supervised learning where each training/test instance is labeled with small subset of relevant labels that are chosen from a large set of possible target labels. The framework of XC has been widely employed in web applications such as automatic labeling of web-encyclopedia, prediction of related searches, and recommendation systems.While most state-of-the-art models in XC achieve high overall accuracy by performing well on the frequently occurring labels, they perform poorly on a large number of infrequent (tail) labels. This arises from two statistical challenges, (i) missing labels, as it is virtually impossible to manually assign every relevant label to an instance, and (ii) highly imbalanced data distribution where a large fraction of labels are tail labels. In this work, we consider common loss functions that decompose over labels, and calculate unbiased estimates that compensate missing labels according to Natarajan et al. [26]. This turns out to be disadvantageous from an optimization perspective, as important properties such as convexity and lower-boundedness are lost. To circumvent this problem, we use the fact that typical loss functions in XC are convex surrogates of the 0-1 loss, and thus propose to switch to convex surrogates of its unbiased version. These surrogates are further adapted to the label imbalance by combining with label-frequency-based rebalancing.We show that the proposed loss functions can be easily incorporated into various different frameworks for extreme classification. This includes (i) linear classifiers, such as DiSMEC, on sparse input data representation, (ii) attention-based deep architecture, Atten-tionXML, learnt on dense Glove embeddings, and (iii) XLNet-based transformer model for extreme classification, APLC-XLNet. Our results demonstrate consistent improvements over the respective vanilla baseline models, on the propensity-scored metrics for precision and nDCG. This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.

Propensity-scored Probabilistic Label Trees

“…In this section, we empirically show the usefulness of the proposed plug-in approach by incorporating it into BR and PLT algorithms and comparing these algorithms to their vanilla versions and stateof-the-art methods, particularly those that focus on tail-labels performance: PFastreXML [11], ProXML [4], a variant of DiSMEC [3] with a re-balanced and unbiased loss function as implemented in PW-DiSMEC [20] (class-balanced variant), and Parabel [18]. We conduct a comparison on six well-established XMLC benchmark datasets from the XMLC repository [6], for which we use the original train and test splits.…”

On Missing Labels, Long-tails and Propensities in Extreme Multi-label Classification

“…If propensities are known, then they can be used to construct an unbiased, task or surrogate, loss l [35] in the sense that The construction of the unbiased counterpart depends on the form of propensities, e.g., the label-wise propensities (7) are sufficient for losses decomposable over labels [24] like Hamming loss or binary cross-entropy, but might not be for more complex losses without additional assumptions [30]. The unbiased losses can be used in training procedures [18,26] or for estimating the performance of classifiers. For some losses, such as Hamming loss or precision@𝑘, the Bayes classifier can be written as a function of the conditional label distributions 𝜂 𝑗 (𝑥).…”

“…For example, decision tree methods can directly use the propensity-scored variants of metrics such as precision@𝑘 or nDCG@𝑘 [18]. Alternatively, one can use unbiased or upper-bounded propensity-scored surrogate losses [26].…”

“…Of course, in XMLC, both interpretations can be combined, i.e., one would like to have a task loss that is adapted to tail labels, but calculate it in a way that takes missing labels into account. The closest to this in the literature is [26], where training uses a loss that combines unbiased estimates and classrebalancing, but still, evaluation is performed using vanilla and propensity-scored metrics, instead of a propensity-scored variant of a tail-weighted metric.…”

Speeding-up One-vs-All Training for Extreme Classification via Smart Initialization

“…for the large dataset from the extreme classification repository [5] (cf. Figures 1 in [31,4,28] for some examples), and for many types of data that is gathered at internet-scale [1].…”