Nyström subsampling method for coefficient-based regularized regression

Abstract

Kernel methods are attractive in data analysis as they can model nonlinear similarities between observations and provide means to rich representations, both of which are useful for the regression problems in general domains. Despite their popularity, they suffer from two primary inherent drawbacks. One drawback is the positive definiteness requirement of the kernel functions, which greatly restricts their applications to some real data analysis. The other drawback is their poor scalability in massive data scenarios. In this paper, we aim to address these two problems by considering the Nyström subsampling approach for coefficient-based regularized regression (or Nyström CRR for short). Nyström subsampling is an effective approach to reduce the space and time complexity by constructing a low-rank approximation of the original kernel matrix through column subsampling. Coefficient-based regularized regression can provide a simple paradigm for designing indefinite kernel methods. We show that a combination of these two schemes is not only computationally efficient but also statistically consistent with a mini-max optimal rates of convergence. We explicitly determine the lower bound of subsampling level as a function of the sample size such that the mini-max optimal convergence rates can be preserved. From our analysis, the subsampling level plays a role as a trade-off between computational and asymptotic behaviors of Nyström CRR, and hence is pivotal for algorithmic performances. In order to choose an appropriate subsampling level, we develop an incremental Nyström CRR for nested subsampling sets. The proposed algorithm can greatly reduce the cost of cross-validation, and even allows to compute the whole solution path through all possible subsampling levels. Based on our empirical studies, the incremental Nyström CRR can perform effective model selections and achieve the state-of-the-art results on both synthetic and real data sets.

Efficient kernel canonical correlation analysis using Nyström approximation

“…Note that initial theoretical work on the Nyström method focused on bounding the discrepancy between a given Gram matrix and its sub-sampled version [12,17], but these results do not provide direct information on the statistical behavior of downstream learning tasks. Recent researches have shifted toward studying the impact of Nyström approximations on specific learning tasks [23,26,28,32,38]. For example, Rudi et al [32] performed the first analysis for Nyström kernel ridge regression, demonstrating that the Nyström method can maintain guaranteed learning rates.…”

“…For example, Rudi et al [32] performed the first analysis for Nyström kernel ridge regression, demonstrating that the Nyström method can maintain guaranteed learning rates. Lu et al [26] expanded this work by considering the misspecified case, while Ma et al [28] explored coefficient-based kernel regression. However, most of these results have been obtained within the context of supervised learning, leaving the understanding of the unsupervised setting relatively limited.…”