MVP: Multi-task Supervised Pre-training for Natural Language Generation

Abstract

Pre-trained language models (PLMs) have achieved notable success in natural language generation (NLG) tasks. Up to now, most of the PLMs are pre-trained in an unsupervised manner using large-scale general corpus. In the meanwhile, an increasing number of models pre-trained with less labeled data showcase superior performance compared to unsupervised models. Motivated by the success of supervised pre-training, we propose Multi-task superVised Pre-training (MVP) for natural language generation. For pre-training the text generation model MVP, we collect a labeled pre-training corpus from 45 datasets over seven generation tasks. For each task, we further pre-train specific soft prompts to stimulate the model capacity in performing a specific task. Extensive experiments have demonstrated the effectiveness of our supervised pre-training in a number of NLG tasks, and our general methods achieve state-of-the-art performance on 12 of 17 datasets.

TabGenie: A Toolkit for Table-to-Text Generation

“…Along with that, demands for a unified input data format have recently been raised with multi-task training for large language models (LLMs) (Sanh et al, 2021;Scao et al, 2022;Ouyang et al, 2022, inter alia). Some works have used simple data linearization techniques for converting structured data to a textual format, in order to align it with the format used for other tasks (Xie et al, 2022;Tang et al, 2022). However, linearizations are using custom preprocessing code, leading to discrepancies between individual works.…”

TabGenie: A Toolkit for Table-to-Text Generation

“…Along with that, demands for a unified input data format have recently been raised with multi-task training for large language models (LLMs) (Sanh et al, 2022;Scao et al, 2022;Ouyang et al, 2022, inter alia). Some works have used simple data linearization techniques for converting structured data to a textual format, in order to align it with the format used for other tasks (Xie et al, 2022;Tang et al, 2022). However, linearizations are using custom preprocessing code, leading to discrepancies between individual works.…”