Locality Guidance for Improving Vision Transformers on Tiny Datasets

Li, Kehan; Yu, Runyi; Wang, Zhennan; Li, Yuan; Song, Guoli; Chen, Jie

doi:10.1007/978-3-031-20053-3_7

Cited by 33 publications

(21 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After experiments, we found that when training Multilayer Perceptron (MLP) as a Feed‐Forward Network, underwater images achieve some results, but still not to our satisfaction. Inspired by [50], we modify the MLP module to Locally‐Enhanced Feed‐Forward (LeFF) module, as shown in Figure 4. We first increase the feature dimension of each token using a 1 × 1 convolution.…”

Section: Methodsmentioning

confidence: 99%

Cformer: An underwater image enhancement hybrid network combining convolution and transformer

Deng

Zhao

et al. 2023

IET Image Processing

View full text Add to dashboard Cite

Underwater images are the most direct and effective ways to obtain underwater information. However, underwater images typically suffer from contrast reduction and colour distortion due to the absorption and scattering of water by light, which seriously limits the further development of underwater visual tasks. Recently, the convolutional neural network has been extensively applied in underwater image enhancement for its powerful local information extraction capabilities, but due to the locality of convolution operation, it cannot capture the global context well. Although the recently emerging Transformer can capture global context, it cannot model local correlations. Cformer is proposed, which is an Unet‐like hybrid network structure. First, a Depth Self‐Calibrated block is proposed to extract the local features of the image effectively. Second, a novel Cross‐Shaped Enhanced Window Transformer block is proposed. It captures long‐range pixel interactions while dramatically reducing the computational complexity of feature maps. Finally, the depth self‐calibrated block and the cross‐shaped enhanced window Transformer block are ingeniously fused to build a global–local Transformer module. Extensive ablation studies are performed on public underwater datasets to demonstrate the effectiveness of individual components in the network. The qualitative and quantitative comparisons indicate that Cformer achieves superior performance compared to other competitive models.

show abstract

Section: Methodsmentioning

confidence: 99%

Cformer: An underwater image enhancement hybrid network combining convolution and transformer

Deng

Zhao

et al. 2023

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…As a result of this issue, these models are in need of a large amount of sample data set and quadratic computational cost. In addition, the receptive eld of the ViT models are constant and capture the global dependencies of the input data set [42,43].…”

Section: Crop-netmentioning

confidence: 99%

Crop-Net: A Novel Deep Learning Framework for Crop Classification using Time-series Sentinel-1 Imagery by Google Earth Engine

Seydi

Arefi

Hasanlou

2023

Preprint

View full text Add to dashboard Cite

Agricultural land management relies heavily on accurate and timely estimation of uncultivated land. Geographical heterogeneity limits the ability of the model to map crops at large scales. This is because the spectral profile of a crop varies spatially. In addition, the generation of robust deep features from remotely sensed SAR data sets is limited by the conventional deep learning models (lacks a mechanism for informative representation). To address these issues, this study proposes a novel dual-stream framework by combining convolutional neural network (CNN) and nested hierarchical transformer (NesT). Based on a hierarchical transformer structure and convolutional layers with spatial/spectral attention modules, the proposed deep learning framework, called Crop-Net, was designed. Time-series Sentinel-1 SAR data were used to evaluate the performance of the proposed model. Sample datasets were also collected by field survey in ten classes including non-crop classes (i.e. water, built-up and barren) and agricultural crop classes (i.e. arboretum, alfalfa, agricultural-vegetable, broad-bean, barley, canola and wheat). The effectiveness of the Crop-Net model was compared with other advanced machine learning and deep learning frameworks. The proposed Crop-Net model is shown to outperform other models through numerical analysis and visual interpretation of crop classification results. It provides accuracy of more than 98.6 (%) and 0.983 in terms of overall accuracy and kappa coefficient, respectively.

show abstract

“…Training ViT on a small dataset from scratch. Very few works have investigated how to train ViT on small datasets [33][34][35][36][37][38]. Compared with CNN, ViT lacks the former's unique inductive bias, and more data are required to force ViT to learn this prior.…”

Section: Type-bmentioning

confidence: 99%

“…Cao et al [37] proposed a method called parametric instance discrimination to construct the contrastive loss to improve the feature extraction performance of ViT trained on small datasets. Li et al [38] applied a CNN-based teacher model to guide ViT to improve the ability about capturing local information.…”

Section: Type-bmentioning

confidence: 99%

GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples

Gao¹,

Xu²,

Zhang³

et al. 2023

Preprint

View full text Add to dashboard Cite

<p>Vision Transformer (ViT) has performed remarkably in various computer vision tasks. Nonetheless, affected by the massive amount of parameters, ViT usually suffers from serious overfitting problems with a relatively limited number of training samples. In addition, ViT generally demands heavy computing resources, which limit its deployment on resource-constrained devices. As a type of model-compression method, model binarization is potentially a good choice to solve the above problems. Compared with the full-precision one, the model with the binarization method replaces complex tensor multiplication with simple bit-wise binary operations and represents full-precision model parameters and activations with only 1-bit ones, which potentially solves the problem of model size and computational complexity, respectively. In this paper, we investigate a binarized ViT model. Empirically, we observe that the existing binarization technology designed for Convolutional Neural Networks (CNN) cannot migrate well to a ViT's binarization task. We also find that the decline of the accuracy of the binary ViT model is mainly due to the information loss of the \textbf{Attention} module and the \textbf{Value} vector. Therefore, we propose a novel model binarization technique, called \textbf{G}roup \textbf{S}uperposition \textbf{B}inarization (\textbf{GSB}), to deal with these issues. Furthermore, in order to further improve the performance of the binarization model, we have investigated the gradient calculation procedure in the binarization process and derived more proper gradient calculation equations for GSB to reduce the influence of gradient mismatch. Then, the knowledge distillation technique is introduced to alleviate the performance degradation caused by model binarization. Analytically, model binarization can limit the parameter’s search space during parameter updates while training a model. Therefore, the binarization process can actually play an implicit regularization role and help solve the problem of overfitting in the case of insufficient training data. Experiments on three datasets with limited numbers of training samples demonstrate that the proposed GSB model achieves state-of-the-art performance among the binary quantization schemes and exceeds its full-precision counterpart on some indicators. Code and models are available at https://github.com/IMRL/GSB-Vision-Transformer.</p>

show abstract

Locality Guidance for Improving Vision Transformers on Tiny Datasets

Cited by 33 publications

References 36 publications

Cformer: An underwater image enhancement hybrid network combining convolution and transformer

Cformer: An underwater image enhancement hybrid network combining convolution and transformer

Crop-Net: A Novel Deep Learning Framework for Crop Classification using Time-series Sentinel-1 Imagery by Google Earth Engine

GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samples

Contact Info

Product

Resources

About