Transformers Meet Small Datasets

Abstract

The research and application areas of transformers have been extensively enlarged due to the success of vision transformers (ViTs). However, due to the lack of local content acquisition capabilities, the pure transformer architectures cannot be trained directly on small datasets. In this work, we first propose a new hybrid model by combining the transformer and convolution neural network (CNN). The proposed model improves the classification ability on small datasets. This is accomplished by introducing more convolution operations in the transformer's two core sections: 1) Instead of the original multi-head attention mechanism, we design a convolutional parameter sharing multi-head attention (CPSA) block that incorporates the convolutional parameter sharing projection in the attention mechanism; 2) the feed-forward network in each transformer encoder block is replaced with a local feed-forward network (LFFN) block that introduces a sandglass block with more depth-wise convolutions to provide more locality to the transformers. We achieve state-of-the-art results when training from scratch on 4 small datasets as compared with the transformers and CNNs without extensive computing resources and auxiliary training. The proposed strategy opens up new paths for the application of transformers on small datasets.

Metasurface-empowered snapshot hyperspectral imaging with convex/deep (CODE) small-data learning theory

“…After the last T-Block that focuses more on spectral attention, we further enhance the spatial features using spatialspectral domain learning (SDL) module 37 , whose output is the desired deblurred multispectral image e Y = f À1 B ðY Þ. It is quite interesting to notice that numerous recent articles have proposed and successfully demonstrated the training of the Transformer with just small data [38][39][40][41] . The CODE addresses the challenge of small data learning using a completely different philosophy.…”

“…The CODE addresses the challenge of small data learning using a completely different philosophy. Simply speaking, typical techniques [38][39][40][41] have to force the deep network to return a good deep solution (as the final solution), while CODE just accepts the weak DE solution. CODE assumes that though the small scale of data results in such a weak solution, the solution itself still contains useful information.…”