Attention Fusion: a light yet efficient late fusion mechanism for task adaptation in NLU

Abstract

Fine-tuning a pre-trained language model using annotated data has become the de-facto standard for adapting general-purpose pretrained models like BERT to downstream tasks. However, given the trend of larger pretrained models, fine-tuning these models for each downstream task is parameter-inefficient and computationally-expensive deeming this approach sub-optimal for adoption by NLU systems. In recent years, various approaches have been proposed for parameter efficient task adaptation such as Adaptor, Bitfit, Prompt tuning, Prefix tuning etc. However, most of these efforts propose to insert task specific parameters in-between or inside intermediate layers of the pre-trained encoder resulting in higher computational cost due to backpropagation of errors to all layers. To mitigate this issue, we propose a light but efficient, attention based fusion module which computes task-attuned token representations by aggregating intermediate layer representations from a pre-trained network. Our proposed fusion module trains only 0.0009% of total parameters and achieves competitive performance to the standard fine-tuning approach on various tasks. It is also decoupled from the pre-trained network making it efficient during computation and scalable during deployment. Last but not the least, we demonstrate that our proposed attention-fusion mechanism can transfer effectively to different languages for further re-use and expansion.

Efficient Structured Prediction with Transformer Encoders

“…sentiment analysis (Horne et al, 2020;Xiao et al, 2021), morphosyntactic parsing (Kondratyuk and Straka, 2019), gender debiasing and coreference resolution (Abzaliev, 2019), and cross-lingual transfer learning (Chen et al, 2022)). Building on this approach, Cao et al (2022) expand the task-oriented layer aggregation to encompass both token and task aspects. This extension is achieved through the introduction of an attention fusion model that leverages the local features of a token.…”

“…This extension is achieved through the introduction of an attention fusion model that leverages the local features of a token. We extend Cao et al (2022)'s method by introducing an attention mechanism that incorporate global views of intermediate representations.…”

“…• Fusion: returns a linear aggregation of the intermediate representations of a frozen model for each token and task, i.e., A (𝐵) = 𝑙 𝑖=1 (𝐴 ⊙ 𝐵) 𝑖,:,: where 𝐴 is an 𝑙 × 𝑛 attention matrix and 𝐴 ⊙ 𝐵 is the Hadamard product between 𝐴 and every feature view of 𝐵. The attention matrix 𝐴 is based on a 𝑑-dimensional taskspecific attention vector 𝑄 learned during training, i.e., 𝐴 𝑖,𝑗 = exp(𝑄×𝐵 𝑖,𝑗,: ) 𝑙 𝑘=1 exp(𝑄×𝐵 𝑘,𝑗,: ) (Cao et al, 2022).…”

“…We are not the first to propose the intermediate layer aggregation at the token level. Cao et al (2022) have also explored a similar approach. They train the token-layer attention weights based on a task-specific query vector used to measure the similarity between intermediate representations locally.…”