Representing Deep Neural Networks Latent Space Geometries with Graphs

Abstract

Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.

Study of Manifold Geometry using Multiscale Non-Negative Kernel Graphs

“…Furthermore, NNK has also been used to understand convolutional neural networks (CNN) channel redundancy [18] and to propose an early stopping criterion for them [19]. Graph properties (not necessarily based on NNK graphs) have been also proposed for the understanding and interpretation of deep neural network performance [20], latent space geometry [21,22] and to improve model robustness [23]. The specific contribution of this work is to explore the effectiveness of NNK graphs in providing insights into the local geometry of the data, which can be useful in understanding the properties and structure of the whole dataset.…”

Zero-Crossing Point Detection of Sinusoidal Signal in Presence of Noise and Harmonics Using Deep Neural Networks

“…Deep neural networks (DNN) are the most accepted and essential machine learning model to solve either regression or classification problems [20]. Deep neural networks are an assembly of layers that can be mathematically described, in the literature, as a "network function" that associates an input tensor with an output tensor [21]. A DNN is developed to predict the zero-crossing point using four input features of a distorted sinusoidal signal, i.e., slope, intercept, correlation and RMSE.…”

Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation

“…Given N data points represented by feature vectors x, a graph is constructed by connecting each data point (node) to similar data points, so that the weight of an edge between two nodes is based on the similarity of the data points, with the absence of an edge (a zero weight) denoting least similarity. Conventionally, one defines similarity between data points using positive definite kernels [19] such as the Gaussian kernel with bandwidth σ of (4) or range normalized cosine kernel of (5).…”

“…The ability of graphs to define relationships between different types of entities allows us to describe and analyze complex patterns in data [3]. Recently, graphs have been used to understand and improve intermediate representations of deep learning models with application to various tasks, such as model regularization [4] and robustness [5], model distillation [6], and model interpretation [7], [8]. Since no graph is given a priori, these methods typically begin with a graph construction phase, where each graph node corresponds to an item in the training set, and the weight of an edge between two nodes is a function of the distance between their respective intermediate layer activations (i.e., their respective feature vectors).…”