General Probabilistic Surface Optimization and Log Density Estimation

Abstract

Probabilistic inference, such as density estimation and distribution transformation, is a fundamental and highly important problem that needs to be solved in many different domains. Recently, a lot of research was done to solve it using Deep Learning (DL) approaches, including unnormalized and energy models, as well as Generative Adversarial Networks, where DL has shown top approximation performance. In this paper we contribute a novel algorithm family, which generalizes all above, and allows to infer different statistical modalities (e.g. data likelihood and ratio between densities) from data samples. The proposed unsupervised technique, named Probabilistic Surface Optimization (PSO), views a neural network (NN) as a flexible surface which can be pushed according to loss-specific virtual stochastic forces, where a dynamical equilibrium is achieved when the point-wise forces on the surface become equal. Concretely, the surface is pushed up and down at points sampled from two different distributions, with overall up and down forces becoming functions of these two distribution densities and of force intensity magnitudes defined by loss of a particular PSO instance. The eventual force equilibrium upon convergence enforces the NN to be equal to various statistical functions depending on the used magnitude functions, such as data density. Furthermore, this dynamical-statistical equilibrium is extremely intuitive and useful, providing many implications and possible usages in probabilistic inference. Further, we connect PSO to numerous existing statistical works which are also PSO instances, and derive new PSO-based inference methods as demonstration of PSO exceptional usability. Likewise, based on the insights coming from the virtual-force perspective we analyze PSO stability and propose new ways to improve it. Finally, we present new instances of PSO, termed PSO-LDE, for data density estimation on logarithmic scale and also provide a new NN block-diagonal architecture for increased surface flexibility, which significantly improves estimation accuracy. Both PSO-LDE and the new architecture are combined together as a new density estimation technique. In our experiments we demonstrate this technique to be superior over state-of-the-art baselines in density estimation task for multi-modal 20D data.

Neural Spectrum Alignment: Empirical Study

“…As well, we showed various trends of the kernel dynamics as a result of the learning rate decay, accounting for which we argue may lead to a further progress in DL theory. The considered herein optimization scenario is 2D regression, yet it is consistent with our previous numerous experiments over different unsupervised losses and architecture types in [12], where considered data dimension was ranging between 20 and 100. Likewise, beyond GD similar trends were also previously observed for a stochastic gradient-descent (SGD) and Adam optimizer.…”

“…(1) The above formulation can be generalized to include unsupervised learning methods in [12] by eliminating labels Y i from the equations. Further, techniques with a model f θ (X) returning multidimensional outputs are out of scope for this paper, to simplify the formulation.…”