Learning rates of multi-kernel regression by orthogonal greedy algorithm

Greedy Criterion in Orthogonal Greedy Learning

“…However, this claim can not be applicable to greedy learning since the estimator is based on the samples with observational noises. Therefore, researches usually adopt a suitable number of iteration in OGL to avoid the overfitting/underfitting [2,6]. Since OGL always searches the most correlative atom and realizes the optimal approximation capability of the space spanned by the selected atoms in each greedy step, its generalization capability becomes sensitive to the number of iterations.…”

“…Thus, a slight turbulence of the number of atoms may lead to a great change of the generalization capability, which can be witnessed in Fig.1. Furthermore, the l 0 -based complexity regularization strategy [2] is only for the benefit of theoretical analysis and the applicable range of the l 1 -based adaptive stopping criterion [6] is quite restricted, which makes it be difficult to persuade the programmers to utilize OGL. Recalling that a possible reason of this problem is OGL searches the new atom according to SGD, an advisable idea is to weaken the level of greed by taking the "greedy-metric" issue into account.…”

“…Barron et al [2] suggested to select k that minimizes a l 0 based complexity regularization strategy, which often needs a full running before the best parameter is selected. Chen et al [6] proposed a stopping criterion also leads to a long iterative procedure in practice and sometimes does not work. In short, all the aforementioned study of stopping-criterion attempted to design a terminal rule by controlling the number of iterations directly.…”

Prediction-based Termination Rule for Greedy Learning with Massive Data

“…In recent works, the orthogonal greedy algorithm (OGA) has been revitalized as a computationally convenient alternative for learning (Barron et al (2008), Chen, Li, and Pan (2013a), Chen et al (2013b)). As a different learning paradigm, it starts from a null model and explores f ( X ) based on a series of expanding subspaces.…”

“…Moreover, as Barron et al (2008) recommended a large value for κ , rule (3) tends to select a k ′ that may be overly small in application. Addressing the same issue under a slightly different framework, Chen et al (2013a, 2013b) proposed terminating OGA when…”