A novel hybrid transformer-CNN architecture for environmental microorganism classification

Abstract

The success of vision transformers (ViTs) has given rise to their application in classification tasks of small environmental microorganism (EM) datasets. However, due to the lack of multi-scale feature maps and local feature extraction capabilities, the pure transformer architecture cannot achieve good results on small EM datasets. In this work, a novel hybrid model is proposed by combining the transformer with a convolution neural network (CNN). Compared to traditional ViTs and CNNs, the proposed model achieves state-of-the-art performance when trained on small EM datasets. This is accomplished in two ways. 1) Instead of the original fixed-size feature maps of the transformer-based designs, a hierarchical structure is adopted to obtain multi-scale feature maps. 2) Two new blocks are introduced to the transformer’s two core sections, namely the convolutional parameter sharing multi-head attention block and the local feed-forward network block. The ways allow the model to extract more local features compared to traditional transformers. In particular, for classification on the sixth version of the EM dataset (EMDS-6), the proposed model outperforms the baseline Xception by 6.7 percentage points, while being 60 times smaller in parameter size. In addition, the proposed model also generalizes well on the WHOI dataset (accuracy of 99%) and constitutes a fresh approach to the use of transformers for visual classification tasks based on small EM datasets.

Hybrid quantum-classical convolutional neural network for phytoplankton classification

“…It is worth noting that the visual transformer has recently achieved remarkable performance in image processing (Dosovitskiy et al, 2020) by identifying long-range dependencies and obtaining global information. Its success has led to its application in classifying plankton datasets (Baek et al, 2022;Dagtekin and Dethlefs, 2022;Kyathanahally et al, 2022;Shao et al, 2022). In the future, it will be intriguing to develop quantum visual transformer models based on the quantum selfattention mechanism (Li et al, 2022;Shi et al, 2023;Zhao et al, 2022), and explore their potential for phytoplankton classification.…”