TOMATO: A Topic-Wise Multi-Task Sparsity Model

Abstract

The Multi-Task Learning (MTL) leverages the interrelationship across tasks and is useful for applications with limited data. Existing works articulate different task relationship assumptions, whose validity is vital to successful multi-task training. We observe that, in many scenarios, the interrelationship across tasks varies across different groups of data (i.e., topic), which we call within-topic task relationship hypothesis. In this case, current MTL models with homogeneous task relationship assumption cannot fully exploit different task relationships among different groups of data. Based on this observation, in this paper, we propose a generalized topic-wise multi-task architecture, to capture the within-topic task relationship, which can be combined with any existing MTL designs. Further, we propose a new specialized MTL design, topic-task-sparsity, along with two different types of sparsity constraints. The architecture, combined with the topic-task-sparsity design, constructs our proposed TOMATO model. The experiments on both synthetic and 4 real-world datasets show that our proposed models consistently outperform 6 state-of-the-art models and 2 baselines with improvement from 5% to 46% in terms of task-wise comparison, demonstrating the validity of the proposed within-topic task relationship hypothesis. We release the source codes and datasets of TOMATO at: https://github.com/JasonLC506/MTSEM. CCS CONCEPTS • Computing methodologies → Multi-task learning.

Structured Sparse Multi-Task Learning with Generalized Group Lasso

“…Hard-sharing approaches [7,26] use a single network, which forces all tasks to own the same hidden space and allows for modeling task-specificity only in the top layer. To remove redundant parameters across tasks, [25] combines l 1 -norm and l 2,1 -norm, while [8] combines Group Lasso and Adaptive Group Lasso; [39] introduces a novel topic-task-element penalty to promote topic-level sparsity; [26] learns a sparse sharing structure by extracting sub-nets from the base network. On the other hand, soft-sharing approaches [18] use a separate network for each task and task relationship is captured by jointly regularizing the weights of these networks, which is usually represented as a tensor by concatenating layer-wise weight matrices from multiple tasks.…”

“…Different from the aforementioned shallow MTL methods, deep MTL uses neural networks to model the complex nonlinear structure of real data. A number of methods with sparsity-inducing regularizations [25,26,39] are proposed to capture complex nonlinear relations among tasks in a compact way. For current structured sparse MTL methods, there are three main problems: 1) Each model works under a specific assumption on the structured sparsity of parameters, that limits its generalization ability to tackle various real applications.…”