On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces

Abstract

Deep learning has been applied to various tasks in the field of machine learning and has shown superiority to other common procedures such as kernel methods. To provide a better theoretical understanding of the reasons for its success, we discuss the performance of deep learning and other methods on a nonparametric regression problem with a Gaussian noise. Whereas existing theoretical studies of deep learning have been based mainly on mathematical theories of well-known function classes such as Hölder and Besov classes, we focus on function classes with discontinuity and sparsity, which are those naturally assumed in practice. To highlight the effectiveness of deep learning, we compare deep learning with a class of linear estimators representative of a class of shallow estimators. It is shown that the minimax risk of a linear estimator on the convex hull of a target function class does not differ from that of the original target function class. This results in the suboptimality of linear methods over a simple but non-convex function class, on which deep learning can attain nearly the minimax-optimal rate. In addition to this extreme case, we consider function classes with sparse wavelet coefficients. On these function classes, deep learning also attains the minimax rate up to log factors of the sample size, and linear methods are still suboptimal if the assumed sparsity is strong. We also point out that the parameter sharing of deep neural networks can remarkably reduce the complexity of the model in our setting.

Biomedical Image Classification in a Big Data Architecture Using Machine Learning Algorithms

“…In contrast, ML, deep learning (DL), and AI excel at automatic pattern recognition from large amounts of biomedical image data. In particular, machine learning and deep learning algorithms (e.g., support vector machine, neural network, and convolutional neural network) have achieved impressive results in biomedical image classification [14][15][16][17][18][19][20][21][22][23]. Classification helps to organize biomedical image databases into image categories before diagnostics [24][25][26][27][28][29][30].…”

“…SVM is a margin-based classifier that achieves superior classification performance compared to other algorithms when the amount of dataset training is medium [34,51,60]. DL techniques are conquering the prevailing traditional approaches of the neural network; when it comes to the huge amount of dataset, applications requiring complex functions demanding increase accuracy with lower time complexities [22,66,67]. DL particularly CNN has shown an intrinsic ability to automatically extract the high-level representations from big data [36].…”

KTBoost: Combined Kernel and Tree Boosting

“…The idea that machine learning methods should be able learn both smooth as well as non-smooth functions has recently received attention also in other areas of machine learning. For instance, Imaizumi and Fukumizu [24] and Hayakawa and Suzuki [20] argue that one of the reasons for the superior predictive accuracy of deep neural networks, over e.g. kernel methods, is their ability to also learn non-smooth functions.…”

Rejoinder: On nearly assumption-free tests of nominal confidence interval coverage for causal parameters estimated by machine learning

“…In Section 6, KBW divide statistical procedures into structure-driven and methods-driven but also acknowledge that the boundary between these two categories is blurry. For example, even for the poster child of the methods-driven tools -deep neural networks -one common research direction is to prove some form of optimality or robustness under some assumptions, often quantified by smoothness, sparsity or other related complexity measures such as metric entropy (Schmidt-Hieber, 2020, Hayakawa and Suzuki, 2020, Barron and Klusowski, 2018.…”

“…In Section 6, KBW divide statistical procedures into structure-driven and methods-driven but also acknowledge that the boundary between these two categories is blurry. For example, even for the poster child of the methods-driven tools -deep neural networks -one common research direction is to prove some form of optimality or robustness under some assumptions, often quantified by smoothness, sparsity or other related complexity measures such as metric entropy (Schmidt-Hieber, 2020, Hayakawa and Suzuki, 2020, Barron and Klusowski, 2018.The discussants then state that higher order influence function (HOIF) based methods are 'structure-driven' because 'they typically rely on carefully constructed series estimates' and achieve 'better performance over appropriate Hölder spaces potentially at the expense of being more structure driven.' This statement misunderstands the motivation and goals of HOIF estimation.…”