Learning Sparse Sharing Architectures for Multiple Tasks

Abstract

Most existing deep multi-task learning models are based on parameter sharing, such as hard sharing, hierarchical sharing, and soft sharing. How choosing a suitable sharing mechanism depends on the relations among the tasks, which is not easy since it is difficult to understand the underlying shared factors among these tasks. In this paper, we propose a novel parameter sharing mechanism, named Sparse Sharing. Given multiple tasks, our approach automatically finds a sparse sharing structure. We start with an over-parameterized base network, from which each task extracts a subnetwork. The subnetworks of multiple tasks are partially overlapped and trained in parallel. We show that both hard sharing and hierarchical sharing can be formulated as particular instances of the sparse sharing framework. We conduct extensive experiments on three sequence labeling tasks. Compared with single-task models and three typical multi-task learning baselines, our proposed approach achieves consistent improvement while requiring fewer parameters.

APGN: Adversarial and Parameter Generation Networks for Multi-Source Cross-Domain Dependency Parsing

“…Domain adaptation has been extensively studied in many research areas, including machine learning (Wang et al, 2017;Kim et al, 2017), computer vision (Ganin and Lempitsky, 2015;Rozantsev et al, 2019) and natural language processing (Kim et al, 2016;Sun et al, 2020). Here, we first simply review single-source domain adaptation researches, and then give more detailed illustration about the studies of multi-source domain adaptation.…”

A Multi-task Learning Model for Daily Activity Forecast in Smart Home

“…According to the needs of each task, CNNs stochastic filter groups grouped the convolution kernel of each convolution layer [44]. There are some other networks such as branched multi-task networks [45], sluice networks [46] and learning sparse sharing [47] to address multiple task sharing issues, but it was difficult to train them due to the high complexity of the model. There are also low supervision [48] and self-supervised learning [49] which are used to do part-of-speech tagging or other issues in the NLP field.…”

FLAT: Chinese NER Using Flat-Lattice Transformer

“…Named entity recognition (NER) plays an indispensable role in many downstream natural language processing (NLP) tasks (Chen et al, 2015;Diefenbach et al, 2018). Compared with English NER (Lample et al, 2016;Yang et al, 2017;Liu et al, 2017;Sun et al, 2020), Chinese NER is more difficult since it usually involves word segmentation.…”