On the Estimation of Transfer Functions, Regularizations and Gaussian Processes – Revisited

Abstract

Intrigued by some recent results on impulse response estimation by kernel and nonparametric techniques, we revisit the old problem of transfer function estimation from input-output measurements. We formulate a classical regularization approach, focused on finite impulse response (FIR) models, and find that regularization is necessary to cope with the high variance problem. This basic, regularized least squares approach is then a focal point for interpreting other techniques, like Bayesian inference and Gaussian process regression. The main issue is how to determine a suitable regularization matrix (Bayesian prior or kernel). Several regularization matrices are provided and numerically evaluated on a data bank of test systems and data sets. Our findings based on the data bank are as follows: The classical regularization approach with carefully chosen regularization matrices shows slightly better accuracy and clearly better robustness in estimating the impulse response than the standard approach -the prediction error method/maximum likelihood (PEM/ML) approach. If the goal is to estimate a model of given order as well as possible, a low order model is often better estimated by the PEM/ML approach, and a higher order model is often better estimated by model reduction on a high order regularized FIR model estimated with careful regularization. Moreover, an optimal regularization matrix that minimizes the mean square error matrix is derived and studied. The importance of this result lies in that it gives the theoretical upper bound on the accuracy that can be achieved for this classical regularization approach.

Kernel methods in system identification, machine learning and function estimation: A survey

“…(The routine is implemented as impulseest.m in the 2012b version of Ljung (2013). ) This method of estimating impulse response, possibly followed by a model reduction of the high order FIR model, has been extensively tested in Monte Carlo simulations in Chen et al (2012). They clearly show that the approach is a viable alternative to the classical ML/PEM methods, and may in some cases provide better models.…”

“…The proof consists of straightforward calculations, see Chen et al (2012). Noting the expression of (21) and invoking Lemma 2, the question what P and γ give the best MSE of the regularized estimate has a clear answer: the equation σ 2 P = γ θ 0 θ T 0 needs to be satisfied.…”

“…In the ''classical, frequentist'' framework, this is a question of trade-off between bias and variance, and can be handled by various model validation techniques. This is often carried out by resorting to complexity measures, such as the Akaike's criterion (AIC) (Akaike, 1974) or cross validation (CV), but some inefficiencies related to these classical approaches have been recently pointed out (Chen, Ohlsson, & Ljung, 2012;Pillonetto, Chiuso, & De Nicolao, 2011;. In particular, it has been shown that sample properties of PEM approaches, equipped e.g.…”

Some Classical and Some New Ideas for Identification of Linear Systems

“…System Identification is about building mathematical models of dynamical systems from observed inputoutput signals, like we did in (2). This problem area contains a number of considerations, like…”

“…If the prior (before Y has been observed) covariance matrix of θ is P , then it is known that the maximum a posteriori (after Y has been observed) estimate of θ is given by (36a). [See [2] for all technical details in this section. ]…”