Neural Stochastic Differential Equations with Neural Processes Family Members for Uncertainty Estimation in Deep Learning

Abstract

Existing neural stochastic differential equation models, such as SDE-Net, can quantify the uncertainties of deep neural networks (DNNs) from a dynamical system perspective. SDE-Net is either dominated by its drift net with in-distribution (ID) data to achieve good predictive accuracy, or dominated by its diffusion net with out-of-distribution (OOD) data to generate high diffusion for characterizing model uncertainty. However, it does not consider the general situation in a wider field, such as ID data with noise or high missing rates in practice. In order to effectively deal with noisy ID data for credible uncertainty estimation, we propose a vNPs-SDE model, which firstly applies variants of neural processes (NPs) to deal with the noisy ID data, following which the completed ID data can be processed more effectively by SDE-Net. Experimental results show that the proposed vNPs-SDE model can be implemented with convolutional conditional neural processes (ConvCNPs), which have the property of translation equivariance, and can effectively handle the ID data with missing rates for one-dimensional (1D) regression and two-dimensional (2D) image classification tasks. Alternatively, vNPs-SDE can be implemented with conditional neural processes (CNPs) or attentive neural processes (ANPs), which have the property of permutation invariance, and exceeds vanilla SDE-Net in multidimensional regression tasks.

Stochastic Chaos and Markov Blankets

“…This follows from the capacity for deep neural networks to represent arbitrary nonlinear relationships between observed data (here, samples of the true system’s flow) and latent variables, e.g., the elements of the flow operator and the steady-state density [ 76 , 79 ]. A similar approach has already become popular in the deep learning literature in the form of “neural stochastic differential equations”, which parameterise the drift and diffusion terms of a stochastic differential equation using feedforward neural networks [ 80 ]. In the same vein, one could use a separate feedforward neural network to parameterise the different components of the flow operator and the self-information.…”