Integrated pre-processing for Bayesian nonlinear system identification with Gaussian processes

Frigola, Roger; Rasmussen, Carl Edward

doi:10.1109/cdc.2013.6760734

Cited by 50 publications

(30 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A main drawback of GP-NARX however is that it cannot account for the observation noise in x t , leading to the errors-in-variables problem. To address this issue, we could (i) conduct datapreprocessing to remove the noise from data [221]; (ii) adopt GPs considering input noise [200]; and (iii) employ the more powerful state space models (SSM) [217] introduced below.…”

Section: E Scalable Recurrent Gpmentioning

confidence: 99%

When Gaussian Process Meets Big Data: A Review of Scalable GPs

Liu

Ong

Shen

et al. 2020

IEEE Trans. Neural Netw. Learning Syst.

566

320

View full text Add to dashboard Cite

The vast quantity of information brought by big data as well as the evolving computer hardware encourages success stories in the machine learning community. In the meanwhile, it poses challenges for the Gaussian process (GP) regression, a well-known non-parametric and interpretable Bayesian model, which suffers from cubic complexity to data size. To improve the scalability while retaining desirable prediction quality, a variety of scalable GPs have been presented. But they have not yet been comprehensively reviewed and analyzed in order to be well understood by both academia and industry. The review of scalable GPs in the GP community is timely and important due to the explosion of data size. To this end, this paper is devoted to the review on state-of-the-art scalable GPs involving two main categories: global approximations which distillate the entire data and local approximations which divide the data for subspace learning. Particularly, for global approximations, we mainly focus on sparse approximations comprising prior approximations which modify the prior but perform exact inference, posterior approximations which retain exact prior but perform approximate inference, and structured sparse approximations which exploit specific structures in kernel matrix; for local approximations, we highlight the mixture/product of experts that conducts model averaging from multiple local experts to boost predictions. To present a complete review, recent advances for improving the scalability and capability of scalable GPs are reviewed. Finally, the extensions and open issues regarding the implementation of scalable GPs in various scenarios are reviewed and discussed to inspire novel ideas for future research avenues.Index Terms-Gaussian process regression, big data, scalability, sparse approximations, local approximations Haitao Liu is with the Rolls-

show abstract

Section: E Scalable Recurrent Gpmentioning

confidence: 99%

When Gaussian Process Meets Big Data: A Review of Scalable GPs

Liu

Ong

Shen

et al. 2020

IEEE Trans. Neural Netw. Learning Syst.

566

320

View full text Add to dashboard Cite

show abstract

“…A particular advantage of expressing both the regularization and the loss in the ℓ 2 norm is that the solution of the corresponding optimization problem is obtained by solving a system of linear equations and an attractive trade-off between regularization bias and variance of the estimates is present (Goethals et al, 2005). LS-SVMs are also related to Kriging (Krige, 1966) in geostatistics and Gaussian processes (GPs) in machine learning, e.g., Frigola and Rasmussen (2013) and Kocijan, Girard, Banko, and Murray-Smith (2005), whose approaches can be seen as different variants of the reproducing kernel Hilbert space (RKHS) theory based function estimators. The relation between these methods is analyzed in Pillonetto et al (2014) and Van Gestel et al (2002).…”

Section: Introductionmentioning

confidence: 99%

An instrumental least squares support vector machine for nonlinear system identification

et al. 2015

View full text Add to dashboard Cite

International audienceLeast-Squares Support Vector Machines (LS-SVMs), originating from Statistical Learning and Reproducing Kernel HilbertSpace (RKHS) theories, represent a promising approach to identify nonlinear systems via nonparametric estimation of theinvolved nonlinearities in a computationally and stochastically attractive way. However, application of LS-SVMs and otherRKHS variants in the identification context is formulated as a regularized linear regression aiming at the minimization of the ℓ2loss of the prediction error. This formulation corresponds to the assumption of an auto-regressive noise structure, which is oftenfound to be too restrictive in practical applications. In this paper, Instrumental Variable (IV) based estimation is integratedinto the LS-SVM approach, providing, under minor conditions, consistent identification of nonlinear systems regarding thenoise modeling error. It is shown how the cost function of the LS-SVM is modified to achieve an IV-based solution. Although,a practically well applicable choice of the instrumental variable is proposed for the derived approach, optimal choice of thisinstrument in terms of the estimates associated variance still remains to be an open problem. The effectiveness of the proposedIV based LS-SVM scheme is also demonstrated by a Monte Carlo study based simulation example

show abstract

“…Nevertheless, neither AR-based nor VAR-based models capture non-linearity. For that reason, substantial effort has been put into non-linear models for time series forecasting based on kernel methods [Chen et al(2008)Chen, Wang, and Harris], ensembles [Bouchachia and Bouchachia(2008)], or Gaussian processes [Frigola and Rasmussen(2014)]. Still, these approaches apply predetermined non-linearities and may fail to recognize different forms of non-linearity for different MTS.…”

Section: Introductionmentioning

confidence: 99%

Temporal pattern attention for multivariate time series forecasting

2019

View full text Add to dashboard Cite

Forecasting of multivariate time series data, for instance the prediction of electricity consumption, solar power production, and polyphonic piano pieces, has numerous valuable applications. However, complex and non-linear interdependencies between time steps and series complicate this task. To obtain accurate prediction, it is crucial to model long-term dependency in time series data, which can be achieved by recurrent neural networks (RNNs) with an attention mechanism. The typical attention mechanism reviews the information at each previous time step and selects relevant information to help generate the outputs; however, it fails to capture temporal patterns across multiple time steps. In this paper, we propose using a set of filters to extract time-invariant temporal patterns, similar to transforming time series data into its "frequency domain". Then we propose a novel attention mechanism to select relevant time series, and use its frequency domain information for multivariate forecasting. We apply the proposed model on several real-world tasks and achieve state-of-the-art performance in all of these with a single exception. Our source code is available at https://github.com/gantheory/TPA-LSTM.* indicates equal contribution.

show abstract

Integrated pre-processing for Bayesian nonlinear system identification with Gaussian processes

Cited by 50 publications

References 9 publications

When Gaussian Process Meets Big Data: A Review of Scalable GPs

When Gaussian Process Meets Big Data: A Review of Scalable GPs

An instrumental least squares support vector machine for nonlinear system identification

Temporal pattern attention for multivariate time series forecasting

Contact Info

Product

Resources

About