Distributed Primal-Dual Optimization for Online Multi-Task Learning

Abstract

Conventional online multi-task learning algorithms suffer from two critical limitations: 1) Heavy communication caused by delivering high velocity of sequential data to a central machine; 2) Expensive runtime complexity for building task relatedness. To address these issues, in this paper we consider a setting where multiple tasks are geographically located in different places, where one task can synchronize data with others to leverage knowledge of related tasks. Specifically, we propose an adaptive primal-dual algorithm, which not only captures task-specific noise in adversarial learning but also carries out a projection-free update with runtime efficiency. Moreover, our model is well-suited to decentralized periodic-connected tasks as it allows the energy-starved or bandwidth-constraint tasks to postpone the update. Theoretical results demonstrate the convergence guarantee of our distributed algorithm with an optimal regret. Empirical results confirm that the proposed model is highly effective on various real-world datasets.

A survey on data‐efficient algorithms in big data era

“…(vi) In multi-task multi-view learning, each task exploits multi-view data. Recent years witness extensive studies on streaming data, known as online multi-task learning [209], this class of methods is used when training data in multiple tasks arrive sequentially, hence (vii) in multi-task online learning, each task is to process sequential data.…”

“…Given the nature of its process, MTL has been studied under the decentralized settings where each machine learns a separate, but related, task. In this vein, multiple parallel and distributed MTL models have been introduced in the recent literature [209][210][211]. Recently, research in MTL using DNN has produced a wide spectrum of approaches that have yielded impressive results on some tasks and application such as image processing [212], NLP [213] and biomedicine [214].…”

Federated Coordinate Descent for Privacy-Preserving Multiparty Linear Regression

“…The most critical research on parallel learning is to parallelize Gradient Descent [19], [50], [56], [68] for its widespread adoption on machine learning tasks. Based on these researches, a great many high-level machine learning models can be transferred to the distributed learning paradigms, for example, distributed ridge regression [47], distributed multi-task learning [57], distributed tensor decomposition [21], and distributed deep learning [2]. However, parallel learning assumes that data is centralized in one party, which has multiple computing clusters.…”