Tumor co-segmentation in PET/CT using multi-modality fully convolutional neural network

Abstract

Automatic tumor segmentation from medical images is an important step for computer-aided cancer diagnosis and treatment. Recently, deep learning has been successfully applied to this task, leading to state-of-the-art performance. However, most of existing deep learning segmentation methods only work for a single imaging modality. PET/CT scanner is nowadays widely used in the clinic, and is able to provide both metabolic information and anatomical information through integrating PET and CT into the same utility. In this study, we proposed a novel multi-modality segmentation method based on a 3D fully convolutional neural network (FCN), which is capable of taking account of both PET and CT information simultaneously for tumor segmentation. The network started with a multi-task training module, in which two parallel sub-segmentation architectures constructed using deep convolutional neural networks (CNNs) were designed to automatically extract feature maps from PET and CT respectively. A feature fusion module was subsequently designed based on cascaded convolutional blocks, which re-extracted features from PET/CT feature maps using a weighted cross entropy minimization strategy. The tumor mask was obtained as the output at the end of the network using a softmax function. The effectiveness of the proposed method was validated on a clinic PET/CT dataset of 84 patients with lung cancer. The results demonstrated that the proposed network was effective, fast and robust and achieved significantly performance gain over CNN-based methods and traditional methods using PET or CT only, two V-net based co-segmentation methods, two variational co-segmentation methods based on fuzzy set theory and a deep learning co-segmentation method using W-net.

PET segmentation of bulky tumors: Strategies and workflows to improve inter-observer variability

“…for the use of MATV as prognostic factor in lymphoma patients or to use metabolic information to measure treatment response [17]. Furthermore, the strategies can, for example, also be used for the fast generation of reliable training sets for Convolutional Neural Networks (CNN) which are used more and more frequently for segmentation tasks [18][19][20]. The aim of this study was to investigate the potential improvements in the inter-observer variability of tumor segmentation results using these new workflows compared with more standard segmentation approaches, while allowing for the generation of plausible and reliable segmentations.…”

Applications of artificial intelligence and deep learning in molecular imaging and radiotherapy

“…Automatic nasopharyngeal carcinoma tumor segmentation from 18 F-FDG PET/CT scans using a U-Net architecture proved the feasibility of this task (dice coefficient = 0.87) using AI-based algorithms (Zhao et al 2019a). Similar studies on head and neck (Huang et al 2018), as well as lung cancers (Zhao et al 2018) exhibited promising results using convolutional neural networks for automated tumor segmentation from PET/CT images. Nevertheless, fully automated lesion delineation from PET, CT, and MR images or any combination of these images still remains a major challenge owing to the large variability of lesion shape and uptake associated with various malignant diseases.…”

Artificial Intelligence in Nuclear Medicine

“…Zhao et al showed, for a small group of 30 patients, that the automatic segmentation of such tumors on 18 F-FDG PET/CT data was, in principle, possible using the U-Net architecture (mean Dice score of 87.47%) (44). Other groups applied similar approaches to head and neck cancer (45) and lung cancer (46,47). Still, fully automated tumor segmentation remains a challenge, probably because of the extremely diverse appearance of these diseases.…”