Fixed-point algorithms for inverse of residual rectifier neural networks

Abstract

A deep neural network with invertible hidden layers has a nice property of preserving all the information in the feature learning stage. In this paper, we analyse the hidden layers of residual rectifier neural networks, and investigate conditions for invertibility under which the hidden layers are invertible. A new fixed-point algorithm is developed to invert the hidden layers of residual networks. The proposed inverse algorithms are capable of inverting some residual networks which cannot be inverted by existing inverting algorithms. Furthermore, a special residual rectifier network is designed and trained on MNIST so that it can achieve comparable performance with the state-of-art performance while its hidden layers are invertible. 2020 Mathematics Subject Classification. Primary: 68T07. Key words and phrases. Invertible neural networks, fixed-point algorithms, invertible residual networks, invertible rectifier neural networks, inverse algorithm of neural networks.

Spectral Collocation Methods for Fractional Integro‐Differential Equations with Weakly Singular Kernels

“…erefore, e ective numerical methods have been widely used for solving this kind of equations in recent years, such as fractional di erential transform methods [1], Taylor expansion [2], operational Tau methods [3], Adomian decomposition methods [4], spline collocation methods [5], wavelets [6][7][8], piecewise polynomial collocation methods [9], and Laplace decomposition methods [10]. Recently, the kernels methods have also received much attention, for the detail, see [11][12][13]. It is well known, that fractional di erential operators are nonlocal and have weakly singular kernels, and so global methods; for example, spectral methods, could be better suited for solving numerically FDIEs.…”