CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection

Abstract

An increasing number of public datasets have shown a marked clinical impact on assessing anatomical structures. However, each of the datasets is small, partially labeled, and rarely investigates severe tumor subjects. Moreover, current models are limited to segmenting specific organs/tumors, which can not be extended to novel domains and classes. To tackle these limitations, we introduce embedding learned from Contrastive Language-Image Pretraining (CLIP) to segmentation models, dubbed the CLIP-Driven Universal Model. The Universal Model allows better segmentation of 25 organs and 6 types of tumors by exploiting the semantic relationship among abdominal structures. The model is developed from an assembly of 14 datasets with 3,410 CT scans and evaluated on 6,162 external CT scans from 3 datasets. We achieve the state-of-theart results on Beyond The Cranial Vault (BTCV). Compared with dataset-specific models, the Universal Model is computationally more efficient (6× faster), generalized better to CT scans from varying sites, and shows stronger transfer learning performance on novel tasks. The design of CLIP embedding enables the Universal Model to be easily extended to new classes without catastrophically forgetting the previously learned classes.

Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation

“…Presently, tasks like vision-language have demonstrated zero-shot learning abilities through text-guided approaches (Zhou et al 2021a). Therefore, the use of different modal images, clinical report data and omics information to assist multi-organ segmentation is still worthy of further exploration (Liu et al 2023c).…”

CXR-IRGen: An Integrated Vision and Language Model for the Generation of Clinically Accurate Chest X-Ray Image-Report Pairs

“…However, the large number of CXR images increases the workload and diagnosis time, posing a challenge for radiologists. Deep learning techniques provide huge support to this issue by demonstrating promising performance in AI-assisted medical applications, including segmentation and diagnosis [26,38]. Nonetheless, the availability of high-quality medical data is still limited due to privacy protocols and imbalanced data distribution, which further constrains the deployment of deep learning models in the medical field [19,27,40].…”