Online Manifold Regularization by Dual Ascending Procedure

Abstract

We propose a novel online manifold regularization framework based on the notion of duality in constrained optimization. The Fenchel conjugate of hinge functions is a key to transfer manifold regularization from offline to online in this paper. Our algorithms are derived by gradient ascent in the dual function. For practical purpose, we propose two buffering strategies and two sparse approximations to reduce the computational complexity. Detailed experiments verify the utility of our approaches. An important conclusion is that our online MR algorithms can handle the settings where the target hypothesis is not fixed but drifts with the sequence of examples. We also recap and draw connections to earlier works. This paper paves a way to the design and analysis of online manifold regularization algorithms.

Online Semi-Supervised Learning With Multiple Regularization Terms

“…Based on the above analysis, we can see that the semisupervised regularization term of S 3 VMs problem as given in Eq. ( 1) is not convex.…”

“…S 3 VMs learn a large margin hyper-plane classifier using labeled training data like SVMs, but simultaneously force this hyper-plane to be far away from the unlabeled data. If you believe there is a ''gap'' or low-density region between the underlying distributions of the different classes, then S 3 VMs can help because it selects a rule with exactly those properties.…”

“…Deriving great benefit from one of the most popular supervised learning method support vector machine (SVM) [16], a number of studies have been proposed to deal with semi-supervised learning problems [12]- [14]. For instance, S 3 VMs [17], LapSVMs [18], co-regularization [18] and SVM-2K [20] extend SVMs from supervised learning to semi-supervised learning by adding a semi-supervised regularization term. Co-LapSVMs [21] and MvLapSVMs [22] add multiple semi-supervised terms to the basic SVMs.…”

A Dual Perspective of Sparse and Robust Online Learning Algorithm

“…In this paper, we use the aggressive gradient ascent (AGA) step on t α which yield aggressive dual ascent and attain better sparsity [12]. The dual ascending procedure on round t can be written as…”

Online Coregularization for Multiview Semisupervised Learning

“…In previous approaches based on coregularization [16, 19], the distance function d (·, ·) is often defined as a square function: $\begin{array}{c}d\left(\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{1}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{1}0.75em0.75em)}\rangle ,\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{2}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{2}0.75em0.75em)}\rangle \right)={\left(\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{1}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{1}0.75em0.75em)}\rangle -\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{2}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{2}0.75em0.75em)}\rangle \right)}^{\mathrm{normal2}}.\end{array}$ The distance function is defined as an absolute function (using l 1 norm) in this paper (this idea is also adopted by Szedmak and Shawe-Taylor [18] and Sun et al [20]): $\begin{array}{c}d\left(\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{1}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{1}0.75em0.75em)}\rangle ,\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{2}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{2}0.75em0.75em)}\rangle \right)=\left|\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{1}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{1}0.75em0.75em)}\rangle -\langle {\mathrm{bold\omega }}^{0.75em0.75em(\mathrm{2}0.75em0.75em)},{\mathrm{boldx}}_t^{0.75em0.75em(\mathrm{2}0.75em0.75em)}\rangle \right|.\end{array}$ Furthermore, (3) is composed of tw...…”