Divide-and-Conquer Learning with Nyström: Optimal Rate and Algorithm

Abstract

Kernel Regularized Least Squares (KRLS) is a fundamental learner in machine learning. However, due to the high time and space requirements, it has no capability to large scale scenarios. Therefore, we propose DC-NY, a novel algorithm that combines divide-and-conquer method, Nyström, conjugate gradient, and preconditioning to scale up KRLS, has the same accuracy of exact KRLS and the minimum time and space complexity compared to the state-of-the-art approximate KRLS estimates. We present a theoretical analysis of DC-NY, including a novel error decomposition with the optimal statistical accuracy guarantees. Extensive experimental results on several real-world large-scale datasets containing up to 1M data points show that DC-NY significantly outperforms the state-of-the-art approximate KRLS estimates.

Distributed Randomized Sketching Kernel Learning

“…Compared with (Liu, Liu, and Wang 2021;Yin et al 2021;Lin, Wang, and Zhou 2020;Yin et al 2020a; N 2 N N 0.5 N 0.5 / Exp DKRR (Lin, Wang, and Zhou 2020) N 2.25 N 1.5 N 0.75 N 0.25 / Pro DKRR-CM (Lin, Wang, and Zhou 2020) N…”

“…The representative distributed KRR includes DKRR (Guo, Lin, and Shi 2019;Chang, Lin, and Zhou 2017;Lin, Guo, and Zhou 2017;Zhang, Duchi, andWainwright 2015, 2013) based on divide-and-conquer, DKRR-RF (Li, Liu, and Wang 2019) based on DKRR and random features (Rudi, Camoriano, and Rosasco 2016), and DKRR-NY-PCG (Yin et al 2020a) based on DKRR and Nyström-PCG (Rudi, Carratino, and Rosasco 2017), which derive the optimal learning rate in expectation. However, they have a restricted limitation in the number of local processors p, that is, to derive the optimal learning rate, p should be restricted to a constant at the most popular case (ζ = 1/2, γ = 1).…”

“…Remark 2. Optimal learning rates for DKRR (Lin, Guo, and Zhou 2017;Guo, Lin, and Shi 2019), DKRR-NY-PCG (Yin et al 2020a), and DKRR-RF (Li, Liu, and Wang 2019) in expectation have been established. However, they have a strict restriction on the number of local processors p.…”

“…with the optimal learning rate, where 2(ζ − 1)γ + 1 > 0. Compared to DKRR-NY-PCG (Yin et al 2020a), DKRR-RS reduces the time complexity and space complexity by factors of N 1−γ 2ζ+γ and N γ 2ζ+γ with the optimal learning rate, where 1 − γ ≥ 0.…”

“…The main principle is to characterize statistical and computational trade-offs, that is, to sacrifice statistical accuracy to gain computational benefits. The representative methods include Nyström (Yin et al 2021(Yin et al , 2020aRudi, Carratino, and Rosasco 2017;Li, Kwok, and Lu 2010) which constructs the approximate kernel matrix with a few anchor points, random features (Liu, Liu, and Wang 2021;Li, Liu, and Wang 2019;Rudi, Camoriano, and Rosasco 2016;Rahimi and Recht 2007), iterative optimization (Lin and Cevher 2020;Lin, Lei, and Zhou 2019;Shalev Shwartz et al 2011), distributed learning (Liu, Liu, and Wang 2021;Lin, Wang, and Zhou 2020;Wang 2019;Guo, Lin, and Shi 2019;Chang, Lin, and Zhou 2017;Lin, Guo, and Zhou 2017;Zhang, Duchi, andWainwright 2015, 2013) which divides the training data into some subsets for processing on local processors and carry out necessary communications, and randomized sketching (Lin and Cevher 2020;Lian, Liu, and Fan 2021;Liu, Shang, and Cheng 2019;Yang, Pilanci, and Wainwright 2017) which projects the kernel matrix into a small one based on the sketch matrix. The above studies show that randomized sketching and distributed learning have outstanding effects in kernel methods.…”