Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators

Abstract

This paper presents a novel pre-trained language models (PLM) compression approach based on the matrix product operator (short as MPO) from quantum many-body physics. It can decompose an original matrix into central tensors (containing the core information) and auxiliary tensors (with only a small proportion of parameters). With the decomposed MPO structure, we propose a novel fine-tuning strategy by only updating the parameters from the auxiliary tensors, and design an optimization algorithm for MPO-based approximation over stacked network architectures. Our approach can be applied to the original or the compressed PLMs in a general way, which derives a lighter network and significantly reduces the parameters to be fine-tuned. Extensive experiments have demonstrated the effectiveness of the proposed approach in model compression, especially the reduction in finetuning parameters (91% reduction on average). The code to reproduce the results of this paper can be found at https://github.com/ RUCAIBox/MPOP.

Variational Tensor Neural Networks for Deep Learning

“…An example of such clever data compression schemes based on TN and MPO decomposition has already been introduced in the previous subsection. Using MPO as an efficient representation for weight matrices of a NN was originally suggested in the ML community under the name tensor trains [44] and later reintroduced in other contexts for systematic compression of fully connected NN models [34,45], for solving partial differential equations with NNs [31] and for language models [46] and speech processing [47].…”

Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing

“…tensor-train operators (Oseledets, 2011)) were proposed for a more effective representation of the linear structure of neural networks , which was used to compress deep neural networks (Novikov et al, 2015), convolutional neural networks (Garipov et al, 2016;Yu et al, 2017), and LSTM (Gao et al, 2020b;Sun et al, 2020a). Based on MPO decomposition, recent studies designed lightweight finetuning and compression methods for PLMs (Liu et al, 2021), developed parameter-efficient MoE architecture (Gao et al, 2022), over-parametrization PLMs and empirical study the emergency ability in quantized large language models . Unlike these works, our work aims to develop a very deep PLM with lightweight architecture and stable training.…”

“…Second, it should not affect the capacity to capture layer-specific variations. To achieve this, we utilize the MPO decomposition (Liu et al, 2021) to develop a parameterefficient architecture by sharing informative components across layers and keeping layer-specific supplementary components (Section 3.2). As another potential issue, it is difficult to optimize deep PLMs due to unstable training (Wang et al, 2022b), especially when weight sharing (Lan et al, 2019) is involved.…”

“…Following Gao et al (2022), we further simplify the decomposition results of a matrix as a central tensor C (the middle tensor) and auxiliary tensors {A i } n−1 i=1 (the rest tensor). As a major merit, such a decomposition can effectively reorganize and aggregate the information of the matrix (Liu et al, 2021): central tensor C can encode the essential information of the original matrix, while auxiliary tensors {A i } n−1 i=1 serve as its complement to exactly reconstruct the matrix.…”

“…They only contain a very small proportion of parameters, which does not significantly increase the model size. While, another merit of MPO decomposition is that these tensors are highly correlated via bond dimensions, and a small perturbation on an auxiliary tensor can reflect the whole matrix (Liu et al, 2021). If the downstream task requires more layer specificity, we can further incorporate low-rank adapters (Hu et al, 2021) for layer-specific adaptation.…”