Gradient-Free Structured Pruning with Unlabeled Data

Medicine-Engineering Interdisciplinary Research Based on Bibliometric Analysis: A Case Study on Medicine-Engineering Institutional Cooperation of Shanghai Jiao Tong University

“…However, as the size and complexity of LLMs rapidly increase [29,30,31], this conventional approach becomes impractical and costly, prompting the need for retraining-free compression techniques. Recent developments in this area have primarily centered around quantization [32,33,34] and have expanded to include pruning methods [13,15,14] that eliminate the need for retraining. In this paper, our work targets enhancing the performance of the retraining-free pruning paradigm, which can reduce the model size, lower the memory consumption, accelerate the inference, and be orthogonal and compatible with quantization for further compression simultaneously.…”

“…In the context of network pruning, retraining-free approaches such as those proposed by [35] seek to mitigate output distortion instead of retraining to maintain as much of the model's original capability as possible. Mask-Tuning, introduced by [13] and adopted by KCM [15], involves rescaling the mask as a reconstruction technique. While it tests the limits of encoder-based models, it struggles to maintain performance at high pruning ratios.…”

“…is a calculation function of computational complexity. In the retraining-free context, L(M) can also be replaced with feature map loss or other metrics [14,15]. Layer-wise Pruning.…”

“…To validate the effectiveness of our reconstruction algorithm, we compare the performance reconstructed with various state-of-the-art structured pruning approaches. These include retraining-based method like LLM-Pruner [48], and retraining-free algorithms, such as Mask-Tuning [13], KCM [15], and FLAP [14]. The baseline reconstruction methods include Mask-Tuning [13] and Bias Compensation [14].…”

“…Additionally, existing retraining-free methods primarily focus on developing better criteria for determining the pruned architecture, with proposed reconstruction techniques often lacking generalizability. As illustrated in Figure 1a, we applied different algorithms [13,14] to reconstruct models pruned using manifold criteria [13,15,16,17] and compared the accuracy drop. Our results reveal that existing reconstruction approaches exhibit limited and unstable performance, particularly for retraining-based criteria.…”