Multi-Task Learning in Natural Language Processing: An Overview

Abstract

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on multiple related tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including the parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Collectively encoding protein properties enriches protein language models

“…Generally, an MTL model can be trained by linearly combining loss functions from different tasks into a single total loss function [15]. In this way, the model can learn a shared representation for all tasks by stochastic gradient descent (SGD) with back-propagation [15,43].…”

“…Generally, an MTL model can be trained by linearly combining loss functions from different tasks into a single total loss function [15]. In this way, the model can learn a shared representation for all tasks by stochastic gradient descent (SGD) with back-propagation [15,43]. Ordinarily, assuming that there are M tasks in all, the global loss function can be defined as where L i represents task-specific loss function, and w i denotes weights assigned for each L i .…”

“…Multi-task learning (MTL), which is able to leverage useful information of related tasks to achieve simultaneous strong performance on multiple associated tasks [15], has led to great success in many machine learning applications like NLP [15,16]. As for the protein sequence domain, MTL has been widely applied for functional studies, like protein-protein interaction and protein targets [17][18][19][20].…”

A domain-adaptive pre-training approach for language bias detection in news

“…Spinde et al [41] train DistilBERT [34] on combinations of biasrelated datasets using a Multi-task Learning (MTL) [6,54] approach. Their best-performing MTL model achieves 0.776 F1 score on a subset of BABE.…”

Multi-Task Distributed Learning using Vision Transformer with Random Patch Permutation

“…Specifically, MTL with transformer-based models has emerged as a popular approach to improve the performances of the closely related task in NLP [15], [16]. In this approach, a shared transformer learns several related tasks simultaneously, like sentence classification and word prediction, and the tasksspecific module yields the outcome for each task.…”