LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning

Sung, Yi-Lin; Cho, Jaemin; Bansal, Mohit

doi:10.48550/arxiv.2206.06522

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We present the local non-overlapping [6] and overlapping [67] self-attention for outdoor and indoor cases respectively, due to different illuminations and surroundings. Secondly, CasMTR enjoys flexible training by a novel Parameter and Memory-efficient Tuning method (PMT), which is originally derived for NLP tasks [44]. Essentially, PMT can incrementally finetune CasMTR based on off-the-shelf matching models with reliable coarse matching initialization and fast convergence.…”

Section: Methodsmentioning

confidence: 99%

Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints

Cao¹,

Fu²

2023

Preprint

View full text Add to dashboard Cite

Learning robust local image feature matching is a fundamental low-level vision task, which has been widely explored in the past few years. Recently, detector-free local feature matchers based on transformers have shown promising results, which largely outperform pure Convolutional Neural Network (CNN) based ones. But correlations produced by transformer-based methods are spatially limited to the center of source views' coarse patches, because of the costly attention learning. In this work, we rethink this issue and find that such matching formulation degrades pose estimation, especially for low-resolution images. So we propose a transformer-based cascade matching model -Cascade feature Matching TRansformer (CasMTR), to efficiently learn dense feature correlations, which allows us to choose more reliable matching pairs for the relative pose estimation. Instead of re-training a new detector, we use a simple yet effective Non-Maximum Suppression (NMS) post-process to filter keypoints through the confidence map, and largely improve the matching precision. CasMTR achieves state-of-the-art performance in indoor and outdoor pose estimation as well as visual localization. Moreover, thorough ablations show the efficacy of the proposed components and techniques. † Corresponding author.Under review.

show abstract

Section: Methodsmentioning

confidence: 99%

Improving Transformer-based Image Matching by Cascaded Capturing Spatially Informative Keypoints

Cao¹,

Fu²

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Zhang et al [70] search for the optimal configurations to combine multiple VPET approaches following once-for-all scheme [7,61]. Since the additional parameters require extra computations compared to full fine-tuning, a few recent works [53,55] design specific architectures to avoid storing the intermediate activations, thereby alleviating the fine-tuning memory cost. However, it is noteworthy that enhancing training efficiency is not the primary objective of our work.…”

Section: Related Workmentioning

confidence: 99%

Sensitivity-Aware Visual Parameter-Efficient Tuning

He¹,

Cai²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

Visual Parameter-Efficient Tuning (VPET) has become a powerful alternative for full fine-tuning so as to adapt pre-trained vision models to downstream tasks, which only tunes a small number of parameters while freezing the vast majority ones to ease storage burden and optimization difficulty. However, existing VPET methods introduce trainable parameters to the same positions across different tasks depending solely on human heuristics and neglect the domain gaps. To this end, we study where to introduce and how to allocate trainable parameters by proposing a novel Sensitivity-aware visual Parameter-efficient Tuning (SPT) scheme, which adaptively allocates trainable parameters to task-specific important positions given a desired tunable parameter budget. Specifically, our SPT first quickly identifies the sensitive parameters that require tuning for a given task in a data-dependent way. Next, our SPT further boosts the representational capability for the weight matrices whose number of sensitive parameters exceeds a pre-defined threshold by utilizing any of the existing structured tuning methods, e.g., LoRA [27] or Adapter [26], to replace directly tuning the selected sensitive parameters (unstructured tuning) under the budget. Extensive experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing VPET methods and largely boosts their performance, e.g., SPT improves Adapter with supervised pre-trained ViT-B/16 backbone by 4.2% and 1.4% mean Top-1 accuracy, reaching SOTA performance on FGVC and VTAB-1k benchmarks, respectively. Source code is at https://github.com/ ziplab/SPT.

show abstract

“…Lester et al [23] proposed to prepend a trainable Tensor into original model input. Sung et al [17] introduced a novel Ladder-Side Tuning (LST) paradigm, which only finetunes a small Transformers network incorporated beside the original model. In this architecture design, the parameters of the newly incorporated networks are only updated to save the computation cost.…”

Section: Parameter-efficient Fine-tuningmentioning

confidence: 99%

“…Different form previous studies, our work introduces a novel approach by combining an additional CNN as a complementary Encoder within SAM architecture. Our approach draws inspiration from the Ladder-Side Tuning (LST) network for Transformers [17]. Our proposed approach enables the flexible integration of an additional network while avoiding backpropagation on the entire large model (i.e., SAM Encoder), leading to faster training and reduced resource costs.…”

Section: Introductionmentioning

confidence: 99%