Theoretical Exploration of Solutions of Feedforward ReLU Networks

Abstract

This paper aims to interpret the mechanism of feedforward ReLU networks by exploring their solutions for piecewise linear functions through basic rules. The constructed solutions should be universal enough to explain the network architectures of engineering. In order for that, we borrow the methodology of theoretical physics to develop the theories. Some of the consequences of our theories include: Under geometric backgrounds, the solutions of both three-layer networks and deep-layer networks are presented, and the solution universality is ensured by several ways; We give clear and intuitive interpretations of each component of network architectures, such as the parameter-sharing mechanism for multi-output, the function of each layer, the advantage of deep layers, the redundancy of parameters, and so on. We explain three typical network architectures: the subnetwork of last three layers of convolutional networks, multi-layer feedforward networks, and the decoder of autoencoders. This paper is expected to provide a basic foundation of theories of feedforward ReLU networks for further investigations.

On a Mechanism Framework of Autoencoders

“…To data set D of m-dimensional space, if there exists no k-dimensional hyperplane l k for 0 ≤ k ≤ m−1 such that D ⊂ l k , then we say that its dimensionality is m. Otherwise, the dimensionality of D is the minimum dimensionality of a hyperplane that contains D. Proof. This lemma is simply a restatement of corollary 12 of Huang (2022) by the new terminology of definition 6. Definition 7.…”

“…As Huang (2022), this paper will continue to use the deductive way to develop theories. There had been some theoretical studies of autoencoders, such as Baldi & Hornik (1989) and Bourlard & Kamp (1988); however, they are mainly for the autoencoders in the early days, and are not much related to the contemporary cases.…”

“…Corollary 12 of Huang (2022) pointed out that if the encoder can realize a bijective map for data set D of the input space, then any element of D could be reconstructed by the decoder. Thus, the bijective map is a key point of autoencoders and will be discussed in detail by sections 2 and 3.…”

“…Proof. The proof is by proposition 5 of Huang (2022), where the method of determining the dimensionality of the intersection of two hyperplanes was given. The intersection of a k 1 -dimensional hyperplane l k 1 and a k 2 -dimensional hyperplane l k 2 with k 1 ≥ k 2 is k 2 − 1, provided that l k 1 ∩ l k 2 = ∅ and l k 2 l k 1 .…”

“…By lemma 3, the key point is the bijective map of encoders. We first present two elementary results by lemma 4 of Huang (2022), from which some concepts or construction methods are proposed and will be used throughout the remaining part of this paper.…”