A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters

Abstract

Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a largescale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of success and failure cases of few-shot transfer, which highlights the role of lexical features. Additionally, we show that a straightforward full model finetuning approach is quite effective for few-shot transfer, outperforming several state-of-the-art few-shot approaches. As a step towards standardizing few-shot crosslingual experimental designs, we make our sampled few shots publicly available. 1 * Equal contribution.

Discrete and Soft Prompting for Multilingual Models

“…Like GPT3, discrete prompting uses natural language to describe NLU tasks. Schick and Schütze (2021), Tam et al (2021), Le Scao and Rush (2021) use human-designed prompts. Gao et al (2020) leverage T5 (Raffel et al, 2020) to generate prompts.…”

“…Qin and Eisner (2021) and Zhong et al (2021) learn soft prompts eliciting more knowledge (Petroni et al, 2019) from PLMs than discrete prompts. Similar to soft prompting but with the PLM being frozen, Li and Liang (2021) propose prefix-tuning to encourage PLMs to solve generation tasks with high parameter-efficiency (Houlsby et al, 2019;Zhao et al, 2020). Lester et al (2021) demonstrate that soft prompting benefits from scaling up the number of PLM parameters.…”

“…Discrete prompting (DP). Following Schick and Schütze (2021), Le Scao and Rush (2021), we reformulate the NLI examples (cf. example in §1) into cloze-style questions using a human-designed prompt.…”

“…Soft prompting (SP; Li and Liang (2021); Qin and Eisner (2021); Zhong et al (2021); Liu et al (2021)) leverages prompts containing "pseudo tokens" that are not part of the PLM vocabulary. In this work, we ask a PLM to fill in the blank ( ) in sentence:…”

“…where each <v i >, i ∈ {1, 2, 3, 4} is associated with a randomly initialized trainable vector (in the PLM's lowest embedding layer) v i ∈ R d , where d is the hidden dimension size of the embedding layer. Directly using v i yields sub-optimal task performance: Li and Liang (2021) reparameterize v i with another trainable matrix and then feed it forward through an MLP. Here, we adopt Liu et al ( 2021)'s approach.…”