Discrete and Soft Prompting for Multilingual Models

Zhao, Muyu; Schütze, Hinrich

doi:10.18653/v1/2021.emnlp-main.672

Cited by 23 publications

(25 citation statements)

References 21 publications

(40 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As suggested by Perez et al (2021) and Zhao & Schütze (2021), we randomly select the same number of samples from training and validation set, making it a reasonable fewshot scenario. Checkpoints are selected via early stopping on the selected validation set, and the stopping metric is the default metric for each task.…”

Section: Few-shot Domain Transfermentioning

confidence: 99%

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Zhang¹,

Guo²,

Meng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The workflow of pretraining and fine-tuning has emerged as a popular paradigm for solving various NLP and V&L (Vision-and-Language) downstream tasks. With the capacity of pretrained models growing rapidly, how to perform parameterefficient fine-tuning has become fairly important for quick transfer learning and deployment. In this paper, we design a novel unified parameterefficient transfer learning framework that works effectively on both pure language and V&L tasks. In particular, we use a shared hypernetwork that takes trainable hyper-embeddings as input, and outputs weights for fine-tuning different small modules in a pretrained language model, such as tuning the parameters inserted into multi-head attention blocks (i.e., prefix-tuning) and feedforward blocks (i.e., adapter-tuning). We define a set of embeddings (e.g., layer, block, task and visual embeddings) as the key components to calculate hyper-embeddings, which thus can support both pure language and V&L tasks. Our proposed framework adds fewer trainable parameters in multi-task learning while achieves superior performances and transfer ability compared to state-of-the-art methods. Empirical results on the GLUE benchmark and multiple V&L tasks confirm the effectiveness of our framework on both textual and visual modalities. 1 * Equal contribution 1 Work is done at the internship of Noah's Ark Lab, Huawei Technologies.

show abstract

Section: Few-shot Domain Transfermentioning

confidence: 99%

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Zhang¹,

Guo²,

Meng³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other work exploring cross-lingual transfer learning and parameter-efficient methods includes Zhao and Schütze (2021). They find that prompts can effectively be used in cross-lingual settings, but their work is constrained to classification while ours focuses on generative tasks.…”

Section: Related Workmentioning

confidence: 99%

Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation

Vu¹,

Barua²,

Lester³

et al. 2022

Preprint

View full text Add to dashboard Cite

In this paper, we explore the challenging problem of performing a generative task (i.e., summarization) in a target language when labeled data is only available in English. We assume a strict setting with no access to parallel data or machine translation. Prior work has shown, and we confirm, that standard transfer learning techniques struggle in this setting, as a generative multilingual model finetuned purely on English catastrophically forgets how to generate non-English. Given the recent rise of parameter-efficient adaptation techniques (e.g., prompt tuning), we conduct the first investigation into how well these methods can overcome catastrophic forgetting to enable zero-shot cross-lingual generation. We find that parameter-efficient adaptation provides gains over standard fine-tuning when transferring between less-related languages, e.g., from English to Thai. However, a significant gap still remains between these methods and fully-supervised baselines. To improve cross-lingual transfer further, we explore three approaches: (1) mixing in unlabeled multilingual data, (2) pre-training prompts on target language data, and (3) explicitly factoring prompts into recombinable language and task components. Our methods can provide further quality gains, suggesting that robust zero-shot cross-lingual generation is within reach.

show abstract

“…Prompting Methods Prompting is a technique that aims to make better use of pre-trained knowledge by reformulating tasks at hand (Liu et al, 2021b). The advent of prompting allows us to do more with one system, i.e., unifying signals cross tasks (Sanh et al, 2021), languages (Zhao and Schütze, 2021), even modality . In this paper, we continue to expand what one system can do by proposing multilingual multitask learning with prompting methods, connecting geographically diverse languages and linguistically different tasks.…”

Section: Multi-lingual Learning With Pretrained Modelsmentioning

confidence: 99%

“…Although prompting methods have proven effective in many NLP scenarios, its effectiveness comes at the cost of prompt engineering (Liu et al, 2021b), as there are usually various factors that influence the prompt design process. Existing works have studied the manual prompt (Schick and Schütze, 2021), soft (trainable) prompt (Lester et al, 2021), and mix prompt (mixing the manual and soft prompt) (Gu et al, 2021;Zhao and Schütze, 2021). The situation becomes more complicated in the multilingual situation, and in this paper, we care about the languages and uniformity of prompt templates design.…”

Section: Prompt Designmentioning

confidence: 99%

See 1 more Smart Citation

Polyglot Prompt: Multilingual Multitask PrompTraining

Fu¹,

Ng²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

This paper aims for a potential architectural breakthrough for multilingual learning and asks: could different tasks from different languages be modeled in a monolithic framework (without any task/language-specific module)?The benefit of achieving this is not only that systems trained on low resources scenario can be assisted by more other languages and tasks, but opening new doors for future multilingual research. We approach this goal by developing a learning framework Polyglot Prompt, where prompting methods are introduced to learn a unified semantic space for different languages and tasks after proper multilingual prompt engineering. Experimentally, we perform a comprehensive evaluation on 6 tasks (topic classification, sentiment classification, named entity recognition, question answering, natural language inference, summarization), 24 datasets, and 49 languages, which shows the efficacy of multilingual multitask prompting training and suggests several interesting observations. e.g., English prompts are polyglots since directly applying them to task samples in other languages could result in a better improvement. We also present an interpretable multilingual evaluation methodology and show how the proposed framework, multilingual multitask prompt training, works. We release all datasets prompted in the best setting 1 and will release our code soon.

show abstract

Discrete and Soft Prompting for Multilingual Models

Cited by 23 publications

References 21 publications

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation

Polyglot Prompt: Multilingual Multitask PrompTraining

Contact Info

Product

Resources

About