A Primer on Reproducing Kernel Hilbert Spaces

Abstract

Reproducing kernel Hilbert spaces are elucidated without assuming prior familiarity with Hilbert spaces. Compared with extant pedagogic material, greater care is placed on motivating the definition of reproducing kernel Hilbert spaces and explaining when and why these spaces are efficacious. The novel viewpoint is that reproducing kernel Hilbert space theory studies extrinsic geometry, associating with each geometric configuration a canonical overdetermined coordinate system. This coordinate system varies continuously with changing geometric configurations, making it well-suited for studying problems whose solutions also vary continuously with changing geometry. This primer can also serve as an introduction to infinite-dimensional linear algebra because reproducing kernel Hilbert spaces have more properties in common with Euclidean spaces than do more general Hilbert spaces.In several contexts, RKHS methods have been described as providing a unified framework [77,32,46,59]; although a subclass of problems was solved earlier by other techniques, a RKHS approach was found to be more elegant, have broader applicability, or offer new insight for obtaining actual solutions, either in closed form or numerically. Parzen denote a straight line in R 2 passing through the origin and intersecting the horizontal axis at an angle of θ radians; it is a one-dimensional subspace of R 2 . Fix an arbitrary point p = (p 1 , p 2 ) ∈ R 2 and define f (θ) to be the point on L(θ) closest to p with respect to the Euclidean metric. It can be shown thatVisualising f (θ) as the projection of p onto L(θ) shows that f (θ) depends continuously on the orientation of the line. While (1.2) veri- Pointwise Coordinates and Canonical CoordinatesFinite-dimensional RKHSs space also be finite dimensional? For convenience, we adopt the latter interpretation. While this interpretation is not standard, it is consistent with our emphasis on the embedding space playing a significant role in RKHS theory. Precisely, we say a RKHS is finite dimensional if the set X in §4 has finite cardinality. The Kernel of an Inner Product SubspaceCentral to RKHS theory is the following question. Let V ⊂ R n be endowed with an inner product. How can this configuration be described efficiently? The configuration involves three aspects: the vector space V , the orientation of V in R n , and the inner product on V . Importantly, the inner product is not defined on the whole of R n , unless V = R n .(An alternative viewpoint that will emerge later is that RKHS theory studies possibly degenerate inner products on R n , where V represents the largest subspace on which the inner product is not degenerate.) One way of describing the configuration is by writing down a basis {v 1 , · · · , v r } ⊂ V ⊂ R n for V and the corresponding Gram matrix G whose ijth element is G ij = v i , v j . Alternatively, the configuration is completely described by giving an orthonormal basis {u 1 , · · · , u r } ⊂ V ⊂ R n ; the Gram matrix associated with an orthonormal basis is the identity matrix and...

A review of machine learning models applied to genomic prediction in animal breeding

“…Where λ is a regularization parameter that controls tradeoffs between goodness of fit and model complexity, H represents a Hilbert space, and g 2 H is the square of the norm of g on H. The square of the norm measures the model complexity. According to Manton and Amblard (2014), RKHS theory can be used to solve three types of problems:…”

Variational inference as iterative projection in a Bayesian Hilbert space with application to robotic state estimation

“…Given an orthonormal basis for a subspace , we define and thus, for any , [33]. For a nonorthonormal basis, where is the entry in .…”

Optimal classification of Gaussian processes in homo- and heteroscedastic settings

“…A natural framework for these types of functional classification problems is the theory of Reproducing Kernel Hilbert Spaces (Cucker and Smale 2002;Berlinet and Thomas-Agnan 2004;Manton and Amblard 2015). For this reason, in the next section we provide an overview of properties of these types of spaces that will be used later in this article to derive optimal rules for the classification of Gaussian processes.…”