Metric learning for text documents

Abstract

Many algorithms in machine learning rely on being given a good distance metric over the input space. Rather than using a default metric such as the Euclidean metric, it is desirable to obtain a metric based on the provided data. We consider the problem of learning a Riemannian metric associated with a given differentiable manifold and a set of points. Our approach to the problem involves choosing a metric from a parametric family that is based on maximizing the inverse volume of a given data set of points. From a statistical perspective, it is related to maximum likelihood under a model that assigns probabilities inversely proportional to the Riemannian volume element. We discuss in detail learning a metric on the multinomial simplex where the metric candidates are pull-back metrics of the Fisher information under a Lie group of transformations. When applied to text document classification the resulting geodesic distance resemble, but outperform, the tfidf cosine similarity measure.

Manifold Learning for Multi-dimensional Auto-regressive Dynamical Models

“…When the training set is unlabeled, manifold learning has to be based on purely geometrical considerations. G. Lebanon [19] has recently suggested in the context of document retrieval an approach that seeks to maximize the inverse volume element associated with a metric around the given training set of points [22]:…”

“…By definition, the push-forward F * λ of a vector v ∈ T m M under a an automorphism F λ with parameter λ is given by [19]:…”

“…The volume element of the pullback metric g * in a point m ∈ M is the determinant of the Gramian [19]…”

“…If the training set of models is labeled, we can then find the parameter of the metric which optimizes classification performance by crossvalidation [8]. Otherwise, the metric which optimizes some purely geometric objective function can be sought (like, for instance, the inverse volume of the manifold around the data-points in D [19]). In particular, we consider here the class A R(2) of multidimensional autoregressive models of order 2.…”

Fine: Information embedding for document classification

“…A standard method for differentiating document classes is to form a probability distribution over a dictionary and use methods of information geometry to determine a similarity between data sets [2]. To the best of our knowledge, most metrics in document classification which are based on word probabilities [3] do not restrict the probability manifold to be a lower dimensional parametric manifold. As a result, a geodesic may go through many probability models that are not admissible in the context of document classification (e.g., a text with only the words 'the' and 'of').…”