Compressed labeling on distilled labelsets for multi-label learning

Abstract

Directly applying single-label classification methods to the multi-label learning problems substantially limits both the performance and speed due to the imbalance, dependence and high dimensionality of the given label matrix. Existing methods either ignore these three problems or reduce one with the price of aggravating another. In this paper, we propose a {0, 1} label matrix compression and recovery method termed "compressed labeling (CL)" to simultaneously solve or at least reduce these three problems. CL first compresses the original label matrix to improve balance and independence by preserving the signs of its Gaussian random projections. Afterward, we directly utilize popular binary classification methods (e.g., support vector machines) for each new label. A fast recovery algorithm is developed to recover the original labels from the predicted new labels. In the recovery algorithm, a "labelset distilling method" is designed to extract distilled labelsets (DLs), i.e., the frequently appeared label subsets from the original labels via recursive clustering and subtraction. Given a distilled and an original label vector, we discover that the signs of their random projections have an explicit joint distribution that can be quickly computed from a geometric inference. Based on this observation, the original label vector is exactly determined after performing a series of Kullback-Leibler divergence based hypothesis tests on the distribution about the new labels. CL significantly improves the balance Mach Learn (2012) 88:69-126 of the training samples and reduces the dependence between different labels. Moreover, it accelerates the learning process by training fewer binary classifiers for compressed labels, and makes use of label dependence via DLs based tests. Theoretically, we prove the recovery bounds of CL which verifies the effectiveness of CL for label compression and multi-label classification performance improvement brought by label correlations preserved in DLs. We show the effectiveness, efficiency and robustness of CL via 5 groups of experiments on 21 datasets from text classification, image annotation, scene classification, music categorization, genomics and web page classification.

Multilabel Classification via Co-Evolutionary Multilabel Hypernetwork

“…Recently, many algorithms that aim at solving the high dimensionality problem meanwhile capturing correlations among labels have been proposed. In [24], a compressed labeling (CL) method has been proposed to solve the imbalance, dependency and high dimensionality of the label space in multilabel learning. Zhou and Tao [25] propose a multilabel subspace ensemble (MSE) method to deal with the exponential-sized output space of multilabel learning.…”

“…Content may change prior to final publication. Err KLD y y g R i j y W (24) Thus, the calculation of Δwkj is expressed in (25), (26), (27), (28), (29) , (30) and (31) …”

Graph Based Semi-supervised Non-negative Matrix Factorization for Document Clustering

“…The MUR is directly inherited from our previous work [15]. According to [15][20] [21], MUR decreases the objective function (8), but it converges slowly and does not guarantee convergence to any local minimum. To overcome the slow convergence problem of MUR, Lin [9] first proposed to apply projected gradient descent method (PGD) to optimizing NMF.…”

Multi-label Classification with Output Kernels

“…There has been recent interest in multi-label methods that work in transformed label spaces [16][17][18][19][20][21], primarily based on low-dimensional projections of high dimensional label vectors. For example, random projections [16], maximum eigenvalue projections [18,17], and Gaussian random projections [21] provide techniques for mapping high dimensional label vectors to low dimensional codewords to improve the efficiency of multi-label learning.…”

“…For example, random projections [16], maximum eigenvalue projections [18,17], and Gaussian random projections [21] provide techniques for mapping high dimensional label vectors to low dimensional codewords to improve the efficiency of multi-label learning. Canonical correlation analysis (CCA) has also been considered for relating inputs to label projections [20].…”