Density-Difference Estimation

IEICE Trans. Inf. & Syst.

2014

Self Cite

SUMMARYThe goal of dimension reduction is to represent highdimensional data in a lower-dimensional subspace, while intrinsic properties of the original data are kept as much as possible. An important challenge in unsupervised dimension reduction is the choice of tuning parameters, because no supervised information is available and thus parameter selection tends to be subjective and heuristic. In this paper, we propose an information-theoretic approach to unsupervised dimension reduction that allows objective tuning parameter selection. We employ quadratic mutual information (QMI) as our information measure, which is known to be less sensitive to outliers than ordinary mutual information, and QMI is estimated analytically by a least-squares method in a computationally efficient way. Then, we provide an eigenvector-based efficient implementation for performing unsupervised dimension reduction based on the QMI estimator. The usefulness of the proposed method is demonstrated through experiments.

“…The key idea in LSQMI is to directly approximate the following density-difference function without density estimation of p(z, x), p(z), and p(x) [5]:…”

Section: Lsqmi Estimationmentioning

confidence: 99%

Unsupervised Dimension Reduction via Least-Squares Quadratic Mutual Information

Sainui

IEICE Trans. Inf. & Syst.

2014

Self Cite

“…More intuitively, good density estimators tend to be smooth and thus a densitydifference estimator obtained from such smooth density estimators tends to be over-smoothed [5,6]. The density difference can be estimated in a single shot using the least-squares density difference (LSDD) approach [2]. In this approach, we directly fit a model g(x) to the density difference under the square loss:…”

Section: Direct Estimation Of the Density Differencementioning

confidence: 99%

“…] by estimating p(x) − p ′ (x) using the least squares fitting method [9]. Hyperparameters are selected via cross validation.…”

Section: Ler := Min (Mcr 1 − Mcr)mentioning

confidence: 99%

“…In [9] it was proposed to directly estimate the density difference by fitting a model g(x) to the true density difference f (x) under a square loss:…”

Section: A2 Minimizing the Upper Boundmentioning

confidence: 99%

“…Thus, the above naive scheme may be improved by estimating the density difference directly and then taking its sign to obtain the class labels. Recently, a method was introduced to directly estimate the density difference, called the least-squares density difference (LSDD) estimator [2]. Thus, the use of LSDD for labeling is expected to improve the performance.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Clustering Unclustered Data: Unsupervised Binary Labeling of Two Datasets Having Different Class Balances

Plessis¹,

Niu

2013 Conference on Technologies and Applications of Artificial Intelligence

2013

Self Cite

Abstract. We consider the unsupervised learning problem of assigning labels to unlabeled data. A naive approach is to use clustering methods, but this works well only when data is properly clustered and each cluster corresponds to an underlying class. In this paper, we first show that this unsupervised labeling problem in balanced binary cases can be solved if two unlabeled datasets having different class balances are available. More specifically, estimation of the sign of the difference between probability densities of two unlabeled datasets gives the solution. We then introduce a new method to directly estimate the sign of the density difference without density estimation. Finally, we demonstrate the usefulness of the proposed method against several clustering methods on various toy problems and real-world datasets.

Learning under nonstationarity: covariate shift and class‐balance change

WIREs Computational Stats

Yamada

Plessis

2013

Self Cite

One of the fundamental assumptions behind many supervised machine-learning algorithms is that training and test data follow the same probability distribution. However, this important assumption is often violated in practice, for example, because of an unavoidable sample selection bias or nonstationarity of the environment. Owing to violation of the assumption, standard machine-learning methods suffer a significant estimation bias. In this article, we consider two scenarios of such distribution change-the covariate shift where input distributions differ and class-balance change where class-prior probabilities vary in classification-and review semi-supervised adaptation techniques based on importance weighting. FIGURE 1 | Covariate shift. Input distributions change but the conditional distribution of outputs given inputs does not change. (a) Input densities and importance, (b) Learning target function, training samples, and test samples.466