Convolutional Neural Networks as 2-D systems

Abstract

This paper introduces a novel representation of convolutional Neural Networks (CNNs) in terms of 2-D dynamical systems. To this end, the usual description of convolutional layers with convolution kernels, i.e., the impulse responses of linear filters, is realized in state space as a linear time-invariant 2-D system. The overall convolutional Neural Network composed of convolutional layers and nonlinear activation functions is then viewed as 2-D version of a Lur e system, i.e., a linear dynamical system interconnected with static nonlinear components. One benefit of this 2-D Lur e system perspective on CNNs is that we can use robust control theory much more efficiently for Lipschitz constant estimation than previously possible.

Training Robust Neural Networks Using Lipschitz Bounds

“…Using our parameterization we can train Lipschitz-bounded 1D CNNs in an unconstrained training problem which we illustrated in the classification of ECG data from the MIT-BIH database. Future research includes the extension of our parameterization to 2D CNNs using a 2D systems approach as suggested in [15].…”

“…Note that 2D convolutions also admit a state space realization, namely as a 2D system [15], based on which our parameterization can potentially be extended to end-to-end Lipschitz-bounded 2D CNNs.…”

“…In this paper, we focus on 1D CNNs as a stepping stone to the more relevant, yet more involved case of 2D CNNs, which is beyond the scope of this paper. However, an extension based on a 2D systems representation [15] is possible and left as future work. CNNs typically consist of convolutions, nonlinear activation functions, pooling layers, and linear layers that are concatenated in a feedforward structure.…”

Outlook: How I Learned to Love Machine Learning (A Personal Perspective on Machine Learning in Process Systems Engineering)

“…16,69 Moreover, there is recent interesting work that is aiming to analyze neural networks as dynamical systems. 70 I also think that it is important to emphasize the origin of ML tools because I feel that there is a lot of confusion on how to best train chemical engineers (undergraduate and graduate) and PSE researchers so that they become proficient in ML. I was personally able to jump in the ML wagon quickly because of my training in statistics, linear algebra, and optimization.…”

“…As such, one can use CNNs for a wide range of nonobvious applications such as multivariate time series analysis . As another example, one can show that autoencoders networks are generalizations of PCA (under some specific conditions). , Moreover, there is recent interesting work that is aiming to analyze neural networks as dynamical systems …”