Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation

Hou, Zejiang; Salazar, Julián; Polovets, George

doi:10.1162/tacl_a_00517

Cited by 12 publications

(12 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It exploits a low-dimensional manifold that can approximately represent the https://doi.org/10.1038/s42256-023-00626-4 whole model parameters, and the optimization trajectory follows this manifold. Some delta-tuning methods can be categorized into this approach, for example, LoRA 15 , BitFit 14 and diff pruning 44 . The other approach seeks a surrogate of the original objective function in a small functional subspace and uses the minimizer of the surrogate function as the approximate final solution.…”

Section: Optimization Perspectivementioning

confidence: 99%

“…BitFit 14 updates the bias terms in PLMs while freezing the remaining modules. Low rank adaptation (LoRA) 15 decomposes attention weight update into low-rank matrices to reduce the number of trainable parameters. The delta-tuning methods enable efficient tuning and practical usage for large pre-trained models and often achieve comparable results to the standard fine-tuning.…”

mentioning

confidence: 99%

“…For example, the vanilla fine-tuning of GPT-3 needs to update about 175,255 million parameters, which is almost infeasible in both industry and academia. However, if we tune only the injected low-rank decomposition matrices in each transformer layer 15 , only 37.7 million parameters will be involved in backpropagation. Delta-tuning not only provides a promising way to adapt large PLMs but also sheds light on the mechanisms behind We experiment all methods on T5 BASE , with the best performance highlighted in bold, and also report the performance of PT on T5 LARGE .…”

mentioning

confidence: 99%

See 2 more Smart Citations

Parameter-efficient fine-tuning of large-scale pre-trained language models

et al. 2023

View full text Add to dashboard Cite

With the prevalence of pre-trained language models (PLMs) and the pre-training–fine-tuning paradigm, it has been continuously shown that larger models tend to yield better performance. However, as PLMs scale up, fine-tuning and storing all the parameters is prohibitively costly and eventually becomes practically infeasible. This necessitates a new branch of research focusing on the parameter-efficient adaptation of PLMs, which optimizes a small portion of the model parameters while keeping the rest fixed, drastically cutting down computation and storage costs. In general, it demonstrates that large-scale models could be effectively stimulated by the optimization of a few parameters. Despite the various designs, here we discuss and analyse the approaches under a more consistent and accessible term ‘delta-tuning’, where ‘delta’ a mathematical notation often used to denote changes, is borrowed to refer to the portion of parameters that are ‘changed’ during training. We formally describe the problem and propose a unified categorization criterion for existing delta-tuning methods to explore their correlations and differences. We also discuss the theoretical principles underlying the effectiveness of delta-tuning and interpret them from the perspectives of optimization and optimal control. Furthermore, we provide a holistic empirical study on over 100 natural language processing tasks and investigate various aspects of delta-tuning. With comprehensive study and analysis, our research demonstrates the theoretical and practical properties of delta-tuning in the adaptation of PLMs.

show abstract

Section: Optimization Perspectivementioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Parameter-efficient fine-tuning of large-scale pre-trained language models

et al. 2023

View full text Add to dashboard Cite

show abstract

“…However, these foundational models have learned robust semantic representations of text. They can be fine-tuned for specific tasks with relatively few labeled examples (even as few as 100 or less) using techniques such as transfer learning 21 or meta-learning, 22 , 23 and still achieve state-of-the-art performance.…”

Section: Proposed Alternativementioning

confidence: 99%

Could an artificial intelligence approach to prior authorization be more human?

Lenert

Lane

Wehbe

2023

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Prior authorization (PA) may be a necessary evil within the healthcare system, contributing to physician burnout and delaying necessary care, but also allowing payers to prevent wasting resources on redundant, expensive, and/or ineffective care. PA has become an “informatics issue” with the rise of automated methods for PA review, championed in the Health Level 7 International’s (HL7’s) DaVinci Project. DaVinci proposes using rule-based methods to automate PA, a time-tested strategy with known limitations. This article proposes an alternative that may be more human-centric, using artificial intelligence (AI) methods for the computation of authorization decisions. We believe that by combining modern approaches for accessing and exchanging existing electronic health data with AI methods tailored to reflect the judgments of expert panels that include patient representatives, and refined with “few shot” learning approaches to prevent bias, we could create a just and efficient process that serves the interests of society as a whole. Efficient simulation of human appropriateness assessments from existing data using AI methods could eliminate burdens and bottlenecks while preserving PA’s benefits as a tool to limit inappropriate care.

show abstract

“…By tapping into the abundance of unlabeled data, our model gains a robust understanding of diverse molecular structures. Remarkably, we are able to selectively fine-tune specific aspects of the model in a low-rank manner 57 , particularly when dealing with intricate molecular geometry configurations such as binding scenarios or conformers exhibiting high energy states. An additional rationale for favoring diffusion models in molecule optimization lies in their global optimization approach.…”

Section: Discussionmentioning

confidence: 99%

Huawei Technologies

Zhang¹,

Rohlfer²

The Source of Innovation in China

View full text Add to dashboard Cite

Event-based sensors, with their high temporal resolution (1µs) and dynamical range (120dB), have the potential to be deployed in high-speed platforms such as vehicles and drones. However, the highly sparse and fluctuating nature of events poses challenges for conventional object detection techniques based on Artificial Neural Networks (ANNs). In contrast, Spiking Neural Networks (SNNs) are well-suited for representing event-based data due to their inherent temporal dynamics. In particular, we demonstrate that the membrane potential dynamics can modulate network activity upon fluctuating events and strengthen features of sparse input. In addition, the spike-triggered adaptive threshold can stabilize training which further improves network performance. Based on this, we develop an efficient spiking feature pyramid network for event-based object detection. Our proposed SNN outperforms previous SNNs and sophisticated ANNs with attention mechanisms, achieving a mean average precision (map50) of 47.7% on the Gen1 benchmark dataset. This result significantly surpasses the previous best SNN by 9.7% and demonstrates the potential of SNNs for event-based vision. Our model has a concise architecture while maintaining high accuracy and much lower computation cost as a result of sparse computation. Our code will be publicly available.

show abstract

Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation

Cited by 12 publications

References 28 publications

Parameter-efficient fine-tuning of large-scale pre-trained language models

Parameter-efficient fine-tuning of large-scale pre-trained language models

Could an artificial intelligence approach to prior authorization be more human?

Huawei Technologies

Contact Info

Product

Resources

About