FTR-NAS: Fault-Tolerant Recurrent Neural Architecture Search

Hu, Kai; Ding, Dong; Tian, Suqing; Gong, Rui; Luo, Li; Wang, Lei

doi:10.1007/978-3-030-63823-8_67

Cited by 2 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, we can observe that the PET methods exhibit noticeable performance differences (standard deviation This phenomenon is intuitive and demonstrates the critical impact of design differences (the position and quantity of parameters in the tunable module) on the performance of PET methods. This finding has been consistently found in numerous prior works (Ding et al, 2023;Hu et al, 2022c). However, we find that as the model scaling increases (from BERT SMALL to BERT LARGE in the subfigure [a]; from BLOOM 560M to BLOOM 7.1B in the sub-figure [b]; from T5 SMALL to T5 XXL in the sub-figure [c]), the performance discrepancies among PET methods diminish across all types of models, as evidenced by the decreasing standard deviation (S.D.)…”

Section: Model Scaling Impact On Pet Methodssupporting

confidence: 87%

“…Unified View of PET Methods Positions of Tunable Modules θ = {W 1 , W 2 , ..., W p } Prompt (Lester et al, 2021) h out = f (h in ) + ∆h W will be concatenated to input hidden states Adapter (Houlsby et al, 2019a) W will be plugged between SelfAttn./FFN. layers LoRA (Hu et al, 2022a) W will be plugged into SelfAttn layers BitFit (Ben Zaken et al, 2022) W will be add into Bias terms Hu et al (2022c). Each PET method has p tunable weights W in designed positions.…”

Section: Pet Methodsmentioning

confidence: 99%

“…Another line of PET method selects the existing parameters in a PLM (Ben Zaken et al, 2022;Guo et al, 2021) as the tunable module to optimize. To further enhance the performance of PET methods, some works propose automatic selection strategies (Hu et al, 2022c;Lawton et al, 2023;Zhou et al, 2023) for tunable parameters.…”

Section: Related Workmentioning

confidence: 99%

“…These modules are composed of tunable parameters W that modify the original layers and the corresponding transformations in PLMs. To make comparisons, we follow the unified view (He et al, 2022a;Hu et al, 2022c) to re-frame the transformations of all PET methods as the modifications ∆h of specific hidden states in the corresponding PLM's layers as follows:…”

Section: Parameter Efficient Tuning (Pet)mentioning

confidence: 99%

See 3 more Smart Citations

Exploring the Impact of Model Scaling on Parameter-Efficient Tuning

Su,

Chan,

Cheng

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Parameter-efficient tuning (PET) methods can effectively drive extremely large pre-trained language models (PLMs) by training only minimal parameters. Different PET methods utilize different manually designed tunable modules. In small PLMs, there are usually noticeable performance differences among PET methods. Nevertheless, as the model scale increases, the performance differences become marginal. Hence, we hypothesize that model scaling mitigates the impact of design differences on PET methods. To investigate this hypothesis, we introduce a more flexible PET method called Arbitrary PET (APET) method. The APET method is compatible with a tunable module, which consists of any number of parameters distributed in arbitrary positions. Then, we utilize it and conduct experiments on 11 NLP tasks across 3 representative PLMs. Our investigations reveal that model scaling (1) mitigates the effects of the positions of tunable parameters on performance, and (2) enables tuning methods to achieve performance comparable to full-parameter fine-tuning by optimizing fewer tunable parameters. Intriguingly, we also observe that tuning methods optimize the similar number of tunable parameters to exceed random guess performance on different tasks. We collectively discuss this phenomenon and the two aforementioned findings from an optimization perspective to understand the underlying mechanisms. These conclusions enhance our understanding of the impact of model scaling on PET and assist in designing more effective and efficient PET methods for PLMs of different scales. The source code can be obtained from this GitHub repository: https:// github.com/yushengsu-thu/PET_Scaling.

show abstract

Section: Model Scaling Impact On Pet Methodssupporting

confidence: 87%