Online Proximal Learning Over Jointly Sparse Multitask Networks With $\ell _{\infty, 1}$ Regularization

Abstract

Modeling relations between local optimum parameter vectors to estimate in multitask networks has attracted much attention over the last years. This work considers a distributed optimization problem with jointly sparse structure among nodes, that is, the local solutions have the same sparse support set. Several mixed norm have been proposed to address the jointly sparse structure in the literature. Among several candidates, the (reweighted) ∞,1-norm is element-wise separable, it is more convenient to evaluate their approximate proximal operators. Thus by introducing a (reweighted) ∞,1-norm penalty term at each node, and using a proximal gradient method to minimize the regularized cost, we devise a proximal multitask diffusion LMS algorithm which can promote joint-sparsity. Analyses are provided to characterize the algorithm behavior in the mean and mean-square sense. Simulation results are presented to show its effectiveness, as well as the accuracy of the theoretical findings.

Online Convex Optimization of a Multi-task Fuzzy Rule-based Evolving System

“…More recent works based on OGD and first order algorithms were proposed by [19,20]. They explored new structural regularizations and prove theoretical convergence improvements.…”

Transient Analysis of Clustered Multitask Diffusion RLS Algorithm

“…During the past decade, distributed detection, estimation, and tracking problems have attracted substantial attention in the context of adaptive networks with diffusion strategies [1][2][3]. Several multitask strategies for adaptation and learning over networks have been recently proposed based on the diffusion least-mean-squares (DLMS) algorithm [4][5][6][7][8][9][10][11]. The DLMS algorithm for multitask networks was first proposed in [4,5], and studied in asynchronous networks in [6].…”

“…A new multitask learning formulation using a common latent representation was presented in [7], as well as a unified framework to analyze its performance. Both 1-norm regularization and ∞,1-norm regularization were introduced into multitask networks in [9] and [10], respectively. Recently, the performance of multitask DLMS algorithm has been analyzed in the presence of communication delays [11].…”