Online Sequential Extreme Learning Machine with Generalized Regularization and Adaptive Forgetting Factor for Time-Varying System Prediction

Abstract

Many real world applications are of time-varying nature and an online learning algorithm is preferred in tracking the realtime changes of the time-varying system. Online sequential extreme learning machine (OSELM) is an excellent online learning algorithm, and some improved OSELM algorithms incorporating forgetting mechanism have been developed to model and predict the time-varying system. But the existing algorithms suffer from a potential risk of instability due to the intrinsic ill-posed problem; besides, the adaptive tracking ability of these algorithms for complex time-varying system is still very weak. In order to overcome the above two problems, this paper proposes a novel OSELM algorithm with generalized regularization and adaptive forgetting factor (AFGR-OSELM). In the AFGR-OSELM, a new generalized regularization approach is employed to replace the traditional exponential forgetting regularization to make the algorithm have a constant regularization effect; consequently the potential illposed problem of the algorithm can be completely avoided and a persistent stability can be guaranteed. Moreover, the AFGR-OSELM adopts an adaptive scheme to adjust the forgetting factor dynamically and automatically in the online learning process so as to better track the dynamic changes of the time-varying system and reduce the adverse effects of the outdated data in time; thus it tends to provide desirable prediction results in time-varying environment. Detailed performance comparisons of AFGR-OSELM with other representative algorithms are carried out using artificial and real world data sets. The experimental results show that the proposed AFGR-OSELM has higher prediction accuracy with better stability than its counterparts for predicting time-varying system.

Cholesky Factorization Based Online Sequential Extreme Learning Machines with Persistent Regularization and Forgetting Factor

“…Recently, some papers [27,28,39] take the following cost function in online sequential RELM with forgetting factor:…”

“…OS-ELM has been successfully adopted in some applications, but it still has some drawbacks. Firstly, OS-ELM may encounter ill-conditioning problems, and resulting fluctuating generalization performances of SLFN if the number of hidden nodes L in SLFN is not set appropriately [26][27][28]. Secondly, OS-ELM does not take timeliness of samples into account, so it cannot be directly employed in time-varying or nonstationary environments.…”

“…Despite the widespread application of ReOS-ELM, it does not consider the timeliness of samples. To take this into account, Zhang et al and Du et al separately designed online sequential RELM with a forgetting factor, viz., SF-ELM [34] and RFOS-ELM [35]; Guo and Xu referred to them as FR-OSELM [27,28]. After stating the real optimization cost function in FR-OSELM and theoretically analyzing FR-OSELM, Guo and Xu pointed out that the regularization term in the cost function of FR-OSELM will be forgotten and tends to zero as time passes; thus, FR-OSELM will probably run into ill-conditioning problems and become unstable after a long period.…”

“…Recently, online sequential extreme learning machines with persistent regularization and forgetting factors (OSELM-PRFFs) were put forward [27,28,39]; this can avoid the potential singularity or ill-posed problem of FR-OSELM. Moreover, two kinds of recursive calculation schemes for OSELM-PRFF have been developed.…”

“…Moreover, two kinds of recursive calculation schemes for OSELM-PRFF have been developed. One is FP-ELM, which directly calculates precise inverse of matrix during every update of model [39]; the other includes FGR-OSELM [27] and AFGR-OSELM [28] which compute the recursively approximate inverse of the involved matrix to reduce computational burden. These online sequential RELM have been applied to some examples perfectly.…”