The Trace Ratio Optimization Problem

Abstract

This paper considers the problem of optimizing the ratio Tr[V T AV ]/Tr[V T BV ] over all unitary matrices V with p columns, where A, B are two positive definite matrices. This problem is common in supervised learning techniques. However, because its numerical solution is typically expensive it is often replaced by the simpler optimization problem which consists of optimizing Tr[V T AV ] under the constraint that V T BV = I, the identity matrix. The goal of this paper is to examine this trace ratio optimization problem in detail, to consider different algorithms for solving it, and to illustrate the use of these algorithms for dimensionality reduction. Introduction.A number of techniques in machine learning and high-dimensional data analysis are based on optimizing a trace of the form Tr [V T AV ] under certain constraints on V . This defines a projector with the basis V , which is then used for various dimensionality reduction tasks.A particular case of this scenario is the well-known Fisher linear discriminant analysis (LDA) [4]. The method, which is a prototypical approach of supervised learning, defines a linear hyperplane which best separates two or more datasets. This is achieved by trying to maximize the ratio of two traces. The first of these (the numerator) represents the in-between scatter, which measures how well the classes are separated in the projected space. The second (the denominator) represents the within scatter, which measures how well clustered each class is in the projected space.Consider the illustration shown in Figure 1. We are given n points which lie on a two-dimensional plane, with each point being labeled either as a triangle or as a square. Our goal is to find a good one-dimensional projection of these points. Specifically, we would like the projector to separate as best as possible the two distinct "classes," i.e., to separate triangles from squares. The left side of the figure shows a

Sparse and kernel OPLS feature extraction based on eigenvalue problem solving

“…The minimization of LðUÞ is equivalent to the maximization of the second trace in the above expression, i.e., a ratio trace maximization problem (see, e.g., [40,41]). Obviously, the optimizer of (4) is not unique, since e.g., multiplying U by a constant does not affect the value of LðUÞ.…”

Patient-specific modeling of the neuroendocrine HPA-axis and its relation to depression: Ultradian and circadian oscillations

“…The classification showed good performance compared to the classical methods such as Carroll's Depression Scale (CDS) [52]. In [12], data were analyzed independently with mixture effects modeling [53], cluster analysis [54] as well as a two-dimensional hidden Markov chain modeling approach of the low frequency signal [55,56], all resulting in the same categorization of data. Hence, the index suggests a scientifically based measure for depression and moves the etiology of depression from considering cortisol to also including ACTH.…”

Rigorous optimisation of multilinear discriminant analysis with Tucker and PARAFAC structures

“…In particular, we consider trace-ratio optimisation of matrices and compare them to existing trace-ratio optimisation procedures that have been used for MDA. We note that other procedures for optimising the trace-ratio exist [15, 16]. However, none of these procedures incorporate the cross-product of Stiefel manifolds structure of matrices presently considered.…”