Growing and Pruning Selective Ensemble Regression for Nonlinear and Nonstationary Systems

Abstract

For a selective ensemble regression (SER) scheme to be effective in online modeling of fastarriving nonlinear and nonstationary data, it must not only be capable of maintaining a most up to date and diverse base model set but also be able to forget old knowledge no longer relevant. Based on these two important principles, in this paper, we propose a novel growing and pruning SER (GAP-SER) for timevarying nonlinear data. Specifically, during online operation, newly emerging process state is automatically identified and a local linear model is fitted to it. This adaptive growing strategy therefore maintains a most up to date and diverse local model set. The online prediction model is then constructed as a selective ensemble from the local linear model set based on a probability metric. Moreover, a pruning strategy is derived to remove 'unwanted' out of date local linear models in order to achieve low online computational complexity without sacrificing online modeling accuracy. A chaotic time series prediction and two real-world data sets are used to demonstrate the superior online modeling performance of the proposed GAP-SER over a range of benchmark schemes for nonlinear and nonstationary systems, in terms of online prediction accuracy and computational complexity. INDEX TERMS Nonlinear and nonstationary data, local linear model, growing model, pruning model, selective ensemble.

Multi-Output Selective Ensemble Identification of Nonlinear and Nonstationary Industrial Processes

“…The significance level α µ and α Σ are usually set to small values, e.g., 0.05, 0.01, and they can be different according to the process data characteristics. Similar to our previous work [25], [26], the selection of window size W G is a trade-off between the adaptive ability of capturing the local characteristics and the accuracy of local model. The proposed local learning procedure is summarized in Algorithm 1.…”

“…During online operation, when the newest data sample {x(t next ), y(t next )} is available, the data window shift one sample ahead, and the corresponding learning procedure can then be carried out. Unlike our previous work for single-output modeling [26], we consider multi-output modeling via multivariate statistics. This local learning procedure automatically encodes a newly emerging process state in the memory as a new local linear model.…”

“…The GAP-SER can not only learn the newly emerging concept but also forget the past accumulated old concepts that are no longer relevant, hence balancing well the so-called stability-plasticity dilemma of a learning system. The experimental results of [26] shows that the GAP-SER typically outperforms the SEMLM, in terms of both online prediction accuracy and computational complexity.…”

“…Hence, a potential problem associated with the SEMLM is that for fast time varying systems, over a long period of online adaptation, the local model set may grow to be very large, which may impose high online computational complexity. To overcome this drawback and in particular to fully consider the aforementioned short-term evolving principle of biological learning system, the work [26] further proposed a growing and pruning SER (GAP-SER) for nonlinear and nonstationary data modeling. The GAP-SER can not only learn the newly emerging concept but also forget the past accumulated old concepts that are no longer relevant, hence balancing well the so-called stability-plasticity dilemma of a learning system.…”

“…At the level of acquiring new knowledge, the newly emerging process state is automatically identified and a multi-output local linear model is fitted to it. Note that the growing strategy of [26] cannot be applied to the multi-output system as it is based on the singleoutput statistical testing. We develop a new adaptive local learning approach with the appropriate statistics for multioutput hypothesis testing in order to construct a highly diverse and independent multi-output local model set.…”