TransU2Net: A Hybrid Network for Retinal Layer Segmentation in OCT

Abstract

The retina, a transparent membrane at the eye’s back, is crucial for capturing light signals. OCT provides a non-invasive method for high-resolution imaging of these layers. Its complex, uneven structure, along with OCT imaging low contrast and noise, complicates segmentation for clinical or research applications. Different eye diseases variably affect retinal layers. This paper introduces TransU2Net, a hybrid network for retinal layer segmentation in OCT images. TransU2Net combines U2-Net and Transformer architectures, utilizing U2Net_Half Encoder for feature extraction and a Transformer module to capture global and local features. Additionally, the model incorporates a Convolutional Block Attention Module (CBAM) in its decoder to enhance its performance. We conducted ablation experiments on our collected dataset to demonstrate the respective roles of U2Net_Half_Encoder and CBAM in the encoding and decoding processes. Subsequently, we compared the TransU2Net with other state-of-the-art (SOTA) models. On the private dataset, the average Dice and IoU achieved 91.64% and 84.80%. On the Duke OCT dataset, the average Dice achieved 84.80%, surpassing other SOTA models in both datasets.