Contributions of Transformer Attention Heads in Multi- and Cross-lingual Tasks

Abstract

This paper studies the relative importance of attention heads in Transformer-based models to aid their interpretability in cross-lingual and multi-lingual tasks. Prior research has found that only a few attention heads are important in each mono-lingual Natural Language Processing (NLP) task and pruning the remaining heads leads to comparable or improved performance of the model. However, the impact of pruning attention heads is not yet clear in cross-lingual and multi-lingual tasks. Through extensive experiments, we show that (1) pruning a number of attention heads in a multilingual Transformer-based model has, in general, positive effects on its performance in cross-lingual and multi-lingual tasks and (2) the attention heads to be pruned can be ranked using gradients and identified with a few trial experiments. Our experiments focus on sequence labeling tasks, with potential applicability on other cross-lingual and multi-lingual tasks. For comprehensiveness, we examine two pre-trained multi-lingual models, namely multi-lingual BERT (mBERT) and XLM-R, on three tasks across 9 languages each. We also discuss the validity of our findings and their extensibility to truly resource-scarce languages and other task settings. * Equal contribution. † Work done when interning at the Minds, Machines, and Society Lab at Dartmouth College. 1 We regard single-source machine translation as a monolingual task since the inputs to the models are mono-lingual.

Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

“…To identify and remove interference, we need a metric that can separate harmful, unimportant, and beneficial attention heads. Prior work (Michel et al, 2019;Ma et al, 2021) utilized the magnitude of gradients as an importance metric. However, this metric measures the sensitivity of the loss function to the masking of a particular head.…”

Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers

“…To identify and remove interference, we need a metric which can distinguish harmful, unimportant, and beneficial attention heads. Prior work (Michel et al, 2019;Ma et al, 2021) utilized the magnitude of gradients as an importance metric. However, this metric measures the sensitivity of the loss function to the masking of a particular head regardless of the direction of that sensitivity.…”

“…These works adapt a model for the target language by training only Adapters Pfeiffer et al, 2020;Ansell et al, 2021), prompts (Zhao and Schütze, 2021), or subsets of model parameters (Ansell et al, 2022). Ma et al (2021) previously investigated pruning in multilingual models using gradient-based importance metrics to study variability across attention heads. However, they used a process of iterative pruning and language-specific finetuning.…”