Prediction-based Termination Rule for Greedy Learning with Massive Data

Abstract

The appearance of massive data has become increasingly common in contemporary scientific research. When sample size n is huge, classical learning methods become computationally costly for the regression purpose. Recently, the orthogonal greedy algorithm (OGA) has been revitalized as an efficient alternative in the context of kernel-based statistical learning. In a learning problem, accurate and fast prediction is often of interest. This makes an appropriate termination crucial for OGA. In this paper, we propose a new termination rule for OGA via investigating its predictive performance. The proposed rule is conceptually simple and convenient for implementation, which suggests an Ofalse(n/lognfalse) number of essential updates in an OGA process. It therefore provides an appealing route to conduct efficient learning for massive data. With a sample dependent kernel dictionary, we show that the proposed method is strongly consistent with an Ofalse(logn/nfalse) convergence rate to the oracle prediction. The promising performance of the method is supported by both simulation and real data examples.

Greedy Criterion in Orthogonal Greedy Learning

“…• Based on the "δ-greedy threshold" criterion, we propose an adaptive terminate rule for OGL and then provide a complete learning system called δ-thresholding orthogonal greedy learning (δ-TOGL). Different from classical termination rules that devote to searching the appropriate number of iterations based on the bias-variance balance principle [2], [49], our study implies that the balance can also be attained through setting a suitable greedy threshold. This phenomenon reveals the essential importance of the "greedy criterion" issue.…”

“…• Termination rule : this issue depicts how to terminate the learning process. The termination rule is regarded as the main difference between greedy approximation and learning, which has been recently studied [2], [8], [29], [49]. For example, Barron et al [2] proposed an l 0 -based complexity regularization strategy as the termination rule, and Chen et al [8] provided an l 1 -based adaptive termination rule.…”

“…(III.9) defines an interpolation space and is a natural assumption for the regression function in greedy learning [2]. This assumption has already been adopted to analyze the learning capability of greedy learning [2], [29], [49]. The Theorem III.1 illustrates the performance of TOGL and consequently, reveals the feasibility of the greedy criterion in Eq.…”

“…The termination rule is crucial for OGL, but designing an effective termination rule is a tricky problem. All the aforementioned studies [2], [8], [49] of the termination rule attempted to design a termination rule by controlling the number of iterations directly. Since the generalization capability of OGL is sensitive to the number of iterations, the results are at times inadequate.…”