Gradient-based Learning Methods Extended to Smooth Manifolds Applied to Automated Clustering

Abstract

Grassmann manifold based sparse spectral clustering is a classification technique that consists in learning a latent representation of data, formed by a subspace basis, which is sparse. In order to learn a latent representation, spectral clustering is formulated in terms of a loss minimization problem over a smooth manifold known as Grassmannian. Such minimization problem cannot be tackled by one of traditional gradient-based learning algorithms, which are only suitable to perform optimization in absence of constraints among parameters. It is, therefore, necessary to develop specific optimization/learning algorithms that are able to look for a local minimum of a loss function under smooth constraints in an efficient way. Such need calls for manifold optimization methods. In this paper, we extend classical gradient-based learning algorithms on at parameter spaces (from classical gradient descent to adaptive momentum) to curved spaces (smooth manifolds) by means of tools from manifold calculus. We compare clustering performances of these methods and known methods from the scientific literature. The obtained results confirm that the proposed learning algorithms prove lighter in computational complexity than existing ones without detriment in clustering efficacy.

A Coordinate-Free Variational Approach to Fourth-Order Dynamical Systems on Manifolds: A System and Control Theoretic Viewpoint

“…The second-order system (27) represents an improvement to the first-order optimization method (28) and traces back to Polyak's heavy-ball method [40], which, however, came as a discrete-time iterative algorithm. The heavy-ball method (later known as Nesterov accelerated gradient optimization method) may indeed be studied through an associated continuous-time differential equation, as shown in, e.g., [41][42][43]. The dynamical system (27) would constitute a natural generalization of the second-order differential equation associated with Nesterov's iteration, under the proviso that one allows the friction coefficient µ to decrease in time as fast as 1 t .…”