Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

Chen, Lin; Xu, Sheng

doi:10.48550/arxiv.2009.10683

Cited by 16 publications

(25 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although our analysis only concerns overparametrized one-hiddenlayer ReLU neural networks, it can potentially apply to other types of neural networks in the NTK regime. Recently, it has been shown that overparametrized multi-layer networks correspond to the Laplace kernel [35,36]. As long as the trained neural networks can approximate the classifier induced by the NTK, our results can be naturally extended.…”

Section: Robustness and Calibration Errormentioning

confidence: 69%

See 1 more Smart Citation

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Hu¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration. introductionThe pursuit of better classifiers has fueled the progress of machine learning and deep learning research. The abundance of benchmark image datasets, e.g., MNIST, CIFAR, ImageNet, etc., provides test fields for all kinds of new classification models, especially those based on deep neural networks (DNN). With the introduction of CNN, ResNets, and transformers, DNN classifiers are constantly improving and catching up to the human-level performance. In contrast to the active innovations in model architecture, the training objective remains largely stagnant, with cross-entropy loss being the default choice. Despite its popularity, cross-entropy has been shown to be problematic in some applications. Among others, Yu et al. [1] argued that features learned from cross-entropy lack interpretability and proposed a new loss aiming for maximum coding rate reduction. Pang et al. [2] linked the use of cross-entropy to adversarial vulnerability and proposed a new classification loss

show abstract

Section: Robustness and Calibration Errormentioning

confidence: 69%

“…Proof of Theorem 3.2. By the equivalence of the RKHS generated by the Laplace kernel and N [35,36], it can be shown that N can be embedded into the Sobolev space W ν 2 for some ν > d/2. Consider the Hölder space C 0,α b for 0 < α ≤ 1 equipped with the norm…”

Section: E2 Proof Of Theorem 32mentioning

confidence: 99%

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Hu¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The proof of this result can be found in [6]. The similarity between neural tangent kernel and the Laplace kernel was first documented in [15], and a rigorous theory was developed in [10].…”

Section: A1 Proofs Of Lemma A1-lemma A3mentioning

confidence: 91%

“…For the theorem to work for learning with neural networks as we discussed in Section 5, it has been shown that the neural tangent kernel also satisfies the required property in different settings. The main argument is that neural tangent kernel is equivalent to kernel regression with the Laplace kernel as proved in [10]. We cite the following result for the two-layer neural network model.…”

Section: A1 Proofs Of Lemma A1-lemma A3mentioning

confidence: 98%

A Generalized Weighted Optimization Method for Computational Learning and Inversion

Engquist¹,

Ren²,

Yang³

2022

Preprint

View full text Add to dashboard Cite

The generalization capacity of various machine learning models exhibits different phenomena in the underand over-parameterized regimes. In this paper, we focus on regression models such as feature regression and kernel regression and analyze a generalized weighted least-squares optimization method for computational learning and inversion with noisy data. The highlight of the proposed framework is that we allow weighting in both the parameter space and the data space. The weighting scheme encodes both a priori knowledge on the object to be learned and a strategy to weight the contribution of different data points in the loss function. Here, we characterize the impact of the weighting scheme on the generalization error of the learning method, where we derive explicit generalization errors for the random Fourier feature model in both the under-and over-parameterized regimes. For more general feature maps, error bounds are provided based on the singular values of the feature matrix. We demonstrate that appropriate weighting from prior knowledge can improve the generalization capability of the learned model.

show abstract

“…For the special case of rectified linear unit (ReLU) activation function, a 1 (. ), it was shown that λ m ∼ m − d d−1 , and consequently the RKHS of the corresponding NT kernel is equivalent to that of the Laplace kernel (that is a special Matérn kernel with ν = 1 2 ) (Geifman et al, 2020;Chen & Xu, 2020). Our contribution includes generalizing this result by showing the equivalence between the RKHS of the NT kernel, under various smoothness of the activation functions, and that of a Matérn kernel with the corresponding smoothness (see Section 3.3).…”

Section: Contributionsmentioning

confidence: 99%

Uniform Generalization Bounds for Overparameterized Neural Networks

Vakili,

Bromberg,

Garcia

et al. 2021

Preprint

View full text Add to dashboard Cite

An interesting observation in artificial neural networks is their favorable generalization error despite typically being extremely overparameterized. It is well known that classical statistical learning methods often result in vacuous generalization errors in the case of overparameterized neural networks. Adopting the recently developed Neural Tangent (NT) kernel theory, we prove uniform generalization bounds for overparameterized neural networks in kernel regimes, when the true data generating model belongs to the reproducing kernel Hilbert space (RKHS) corresponding to the NT kernel. Importantly, our bounds capture the exact error rates depending on the differentiability of the activation functions. In order to establish these bounds, we propose the information gain of the NT kernel as a measure of complexity of the learning problem. Our analysis uses a Mercer decomposition of the NT kernel in the basis of spherical harmonics and the decay rate of the corresponding eigenvalues. As a byproduct of our results, we show the equivalence between the RKHS corresponding to the NT kernel and its counterpart corresponding to the Matérn family of kernels, that induces a very general class of models. We further discuss the implications of our analysis for some recent results on the regret bounds for reinforcement learning algorithms, which use overparameterized neural networks.

show abstract

Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS

Cited by 16 publications

References 17 publications

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Understanding Square Loss in Training Overparametrized Neural Network Classifiers

A Generalized Weighted Optimization Method for Computational Learning and Inversion

Uniform Generalization Bounds for Overparameterized Neural Networks

Contact Info

Product

Resources

About