Deep neural networks for rotation-invariance approximation and learning

Abstract

Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space. It is shown that deep nets have much better performance than shallow nets (with only one hidden layer) in terms of approximation accuracy and learning capabilities. In particular, for learning radial functions, it is shown that near-optimal rate can be achieved by deep nets but not by shallow nets. Our results illustrate the necessity of depth in neural network design for realization of rotation-invariance target functions.

Realizing Data Features by Deep Nets

“…The reason is that we focus on all activation functions satisfying (1) rather than piecewise activation functions. It should be also mentioned that similar covering number estimates for deep nets with tree structures has been studied in [6], [14], [25]. We highlight that different structures yield essentially non-trivial approaches.…”

“…We also exhibit that for some simple data features like the smoothness, deep nets performs essentially similar as shallow nets. 6 APPENDIX A: PROOF OF THEOREM 1 For 1 ≤ ℓ ≤ L, let W * F ℓ,w be the set of d ℓ × d ℓ−1 matrices with fixed structures and total F ℓ,w free parameters and B * F ℓ,b be the set of F ℓ,b -dimensional vectors with fixed structures and total F ℓ,b free parameters. Denote…”

“…A recent focus in deep nets approximation is to pursue the approximation ability of deep nets with fixed structures. Up till now, numerous theoretical results [39], [54], [6], [38] showed that the approximation ability of deep fully-connected neural networks can be maintained by deep nets with some special structures with much less free parameters.…”

“…We highlight that different structures yield essentially non-trivial approaches. In fact, due to tree structures, the approach in [6], [14], [25] is just to decouple layers by using the boundedness and Liptchiz property of activation functions. However, in estimating covering number of deep nets with arbitrarily fixed structure, we need a novel matrixvector transformation technique, as presented in Appendix A.…”

An improved deep convolutional neural network model with kernel loss function in image classification

“…Kernel methods have been developed as powerful tools for machine learning, statistical analysis and probability numeral calculations [16], [8]. In the recent years, scientists have paid large attention to kernel methods to study various learning frameworks, such as extreme learning [19], deep learning [2], Bayesian learning [10] and others [14], [4], [15]. We attempt to extend the kernel method to find a suitable loss function for CNN problem.…”