Robust Learning with Implicit Residual Networks

Abstract

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show that this choice leads to the improved stability of both forward and backward propagations, has a favorable impact on the generalization power, and allows for control the robustness of the network with only a few hyperparameters. In addition, the proposed reformulation of ResNet does not introduce new parameters and can potentially lead to a reduction in the number of required layers due to improved forward stability. Finally, we derive the memory-efficient training algorithm, propose a stochastic regularization technique, and provide numerical results in support of our findings.

AI-Based Approach for Advancing the Understanding of Spatiotemporal Drought Characteristics

“…Such networks allow for simultaneous extraction of multi-scale features, enable the flow of information across the scales, often require less data and better adopt to a specific problem allowing the same model to be applied for various applications and types of data. We also intend to extend these architectures with our recent results on robust learning of dynamical systems (Reshniak and Webster 2021) and hierarchical design of neural networks with a goal to enable the simultaneous utilization of data coming from different sources and at different resolutions.…”

A Shooting Formulation of Deep Learning

“…A recent line of works, including deep equilibrium models [7] and implicit residual networks [32], has shown that it may not always be necessary to freely parametrize all the layers in the network. Specifically, in [7] and [32], the parameters of each layer are defined via an implicit equation motivated by weight tying thus improving expressiveness and reducing the number of parameters while decreasing the memory footprint via implicit differentiation. Instead, our work increases expressiveness and reduces the number of parameters via particle-based shooting.…”