Efficient and Scalable Multi-Task Regression on Massive Number of Tasks

Abstract

Many real-world large-scale regression problems can be formulated as Multi-task Learning (MTL) problems with a massive number of tasks, as in retail and transportation domains. However, existing MTL methods still fail to offer both the generalization performance and the scalability for such problems. Scaling up MTL methods to problems with a tremendous number of tasks is a big challenge. Here, we propose a novel algorithm, named Convex Clustering Multi-Task regression Learning (CCMTL), which integrates with convex clustering on the k-nearest neighbor graph of the prediction models. Further, CCMTL efficiently solves the underlying convex problem with a newly proposed optimization method. CCMTL is accurate, efficient to train, and empirically scales linearly in the number of tasks. On both synthetic and real-world datasets, the proposed CCMTL outperforms seven state-of-the-art (SoA) multi-task learning methods in terms of prediction accuracy as well as computational efficiency. On a real-world retail dataset with 23, 812 tasks, CCMTL requires only around 30 seconds to train on a single thread, while the SoA methods need up to hours or even days.

Learning an Interpretable Graph Structure in Multi-Task Learning

“…Our work is similar in spirit to the graph fused Lasso (GFL) [23] and the Convex Clustering Multi-Task Learning (CCMTL) [22]. However, both GFL and CCMTL separate multi-task learning and graph structure estimation.…”

“…We evaluate the performance of GAMTL and RBF-GAMTL against six SOTA multi-task learning methodologies (namely MTRL [2], MSSL [7], BMSL [8], TAT [14], GFL [23] and CCMTL [22]) on both synthetic data and real-world applications. Among the six competitors, MTRL and BMSL learn a task covariance matrix, MSSL targets a task precision matrix which can be interpreted as a graph Laplacian.…”

“…contains only non-positive off-diagonal entries and is zero rowsum, whereas both constraints are necessary to define a valid graph Laplacian [11]. On the other hand, most existing graph structure multi-task learning methodologies assume that the graph topology is known a priori (e.g., [6], [20], [21]) or can be simply predefined from the observation data (e.g., [22], [23]). Unfortunately, in many real-world scenarios, a graph structure is either unavailable or hard to be predefined correctly due to its complex nature [24].…”

“…The computational complexity of Algorithm 1 is defined by the complexity of computing W and computing A. Computing W requires to solve the equation defined in Section III-B1, whose complexity is O(d 3 T 3 + d 2 N + T 3 ) 4 . When the matrix A is sparse, the solution can be efficiently computed using CMG [30] and is shown to have a linear empirical complexity in T [22]. The complexity of computing A is proportional to O(T 2 ) since it requires to evaluate function over a vector w (inside Algorithm 2) of size O(T 2 ) 5 .…”

“…The complexity of computing A is proportional to O(T 2 ) since it requires to evaluate function over a vector w (inside Algorithm 2) of size O(T 2 ) 5 . The complexity could be reduce to O(T 2 + d 2 N + d 3 T ) by enforcing sparsity on A in solving for W , where the last term is only derived empirically [22].…”