Universality and Optimality of Structured Deep Kernel Networks

Abstract

Kernel based methods yield approximation models that are flexible, efficient and powerful. In particular, they utilize fixed feature maps of the data, being often associated to strong analytical results that prove their accuracy. On the other hand, the recent success of machine learning methods has been driven by deep neural networks (NNs). They achieve a significant accuracy on very high-dimensional data, in that they are able to learn also efficient data representations or data-based feature maps.In this paper, we leverage a recent deep kernel representer theorem to connect the two approaches and understand their interplay. In particular, we show that the use of special types of kernels yield models reminiscent of neural networks that are founded in the same theoretical framework of classical kernel methods, while enjoying many computational properties of deep neural networks. Especially the introduced Structured Deep Kernel Networks (SDKNs) can be viewed as neural networks with optimizable activation functions obeying a representer theorem. Analytic properties show their universal approximation properties in different asymptotic regimes of unbounded number of centers, width and depth. Especially in the case of unbounded depth, the constructions is asymptotically better than corresponding constructions for ReLU neural networks, which is made possible by the flexibility of kernel approximation.

Application of Deep Kernel Models for Certified and Adaptive RB-ML-ROM Surrogate Modeling

“…However, due to being kernel models that can be optimized instead of fixed activation functions, they allow for more flexibility and thus enable a potentially faster optimization. As analyzed in [10], these SDKNs enjoy universal approximation properties in various limit cases. The 2L-VKOGA [9] combines the use of an optimized two-layered kernel with greedy kernel algorithms, where we leverage the VKOGA algorithm [7]: Radial basis function kernels are a popular class of kernels which are given as k(x, z) = Φ( x − z ) for some radial basis function Φ : R d → R, e.g.…”

“…In order to make the best out of both approaches, the idea is to combine deep learning methods such as neural networks with the benefits of "shallow" kernel methods. In the following Section 4, we review two recently introduced algorithms in this direction: SDKNs [10] as well as 2L-VKOGA [9].…”