On the maximum entropy property of the first-order stable spline kernel and its implications

Abstract

A new nonparametric approach for system identification has been recently proposed where the impulse response is seen as the realization of a zero-mean Gaussian process whose covariance, the so-called stable spline kernel, guarantees that the impulse response is almost surely stable. Maximum entropy properties of the stable spline kernel have been pointed out in the literature. In this paper we provide an independent proof that relies on the theory of matrix extension problems in the graphical model literature and leads to a closed form expression for the inverse of the first order stable spline kernel as well as to a new factorization in the form U W U ⊤ with U upper triangular and W diagonal. Interestingly, all first-order stable spline kernels share the same factor U and W admits a closed form representation in terms of the kernel hyperparameter, making the factorization computationally inexpensive. Maximum likelihood properties of the stable spline kernel are also highlighted. These results can be applied both to improve the stability and to reduce the computational complexity associated with the computation of stable spline estimators.

Maximum entropy properties of discrete-time first-order stable spline kernel

“…We first formulate the MaxEnt problem solved by the DT SS-1 kernel without Gaussian and uniform sampling assumptions. Then, we extend the result of [11] and link it to our former result: under general sampling assumption, we show that the SS-1 kernel matrix is the solution of a maximum entropy covariance extension problem [12] with band constraints. This results in the well-known tridiagonal structure of the kernel's inverse, which can be also used for efficient numerical implementations [13], [15,Section 5].…”

“…The arguments in [10] were however quite involved, mainly due to the infinite-dimensional nature of the problem and the fact that the differential entropy rate of a generic CT stochastic process is not well-defined. Another recent contribution is [11] where, under Gaussian and uniform sampling assumptions, it is shown that the SS-1 kernel matrix can be given a MaxEnt covariance completion interpretation [12], that is then exploited to derive its special structure (namely that it admits a tridiagonal inverse with closed form representation as well as factorization).…”

Maximum Entropy Kernels for System Identification

“…Proof: By Theorem III.2, the maximum entropy completion of Σ (1) n (x) can be recursively computed starting from the maximum entropy completions of the nested principal submatrices of smaller size. The statement can thus be proved by induction on the dimension n of the completion.…”

Robust EM kernel-based methods for linear system identification

“…Its derivative with respect to λ is where c is a constant. Now consider the following factorization of the first order stable spline kernel [10]:…”