Compound Figure Separation of Biomedical Images with Side Loss

Abstract

With the rapid development of self-supervised learning (e.g., contrastive learning), the importance of having large-scale images (even without annotations) for training a more generalizable AI model has been widely recognized in medical image analysis. However, collecting large-scale task-specific unannotated data at scale can be challenging for individual labs. Existing online resources, such as digital books, publications, and search engines, provide a new resource for obtaining large-scale images. However, published images in healthcare (e.g., radiology and pathology) consist of a considerable amount of compound figures with subplots. In order to extract and separate compound figures into usable individual images for downstream learning, we propose a simple compound figure separation (SimCFS) framework without using the traditionally required detection bounding box annotations, with a new loss function and a hard case simulation. Our technical contribution is four-fold: (1) we introduce a simulation-based training framework that minimizes the need for resource extensive bounding box annotations; (2) we propose a new side loss that is optimized for compound figure separation; (3) we propose an intra-class image augmentation method to simulate hard cases; and (4) to the best of our knowledge, this is the first study that evaluates the efficacy of leveraging self-supervised learning with compound image separation. From the results, the proposed SimCFS achieved state-of-the-art performance on the ImageCLEF 2016 Compound Figure Separation Database. The pretrained self-supervised learning model using large-scale mined figures improved the accuracy of downstream image classification tasks with a contrastive learning algorithm. The source code of SimCFS is made publicly available at https://github.com/hrlblab/ImageSeperation .

Glo-In-One: Holistic Glomerular Detection, Segmentation, and Lesion Characterization with Large-scale Web Image Mining

“…Glomerular Classification 1) Data: Model Pretrain.In this study, we collected over 10,000 compound figures (each figure might contain multiple subplots) through the NIH Open-I search engine with the keywords "glomerular OR glomeruli OR glomerulus". Then, our compound figure separation method [30] was employed to separate compound images into individual images, which were further categorized to different modalities (e.g., light microscopy, florescent microscopy, and electron microscopy) as well as different stain types within the light microscopy. To curate all images, an automatic deep learning-based curator (detector) was trained using only a smaller scale annotated images dataset [75].…”

“…Briefly, we collect 10,000 compound figures with the keywords "glomerular OR glomeruli OR glomerulus" through the NIH Open-i search engine. The details of web image mining are provided in [30].…”

“…We employ a SimSiam [29] network as the baseline method of self-supervised contrastive learning. By deploying our previously developed compound figure separation approach [30], we provide 30,000 unannotated glomerular images via web image mining to train the SimSiam network.…”

Glo-In-One: holistic glomerular detection, segmentation, and lesion characterization with large-scale web image mining

“…The details of web image mining are provided in Ref. 30. However, the images from online resources are typically in compound figures (with multiple subplots), which cannot be directly used for self-supervised learning.…”

“…However, the images from online resources are typically in compound figures (with multiple subplots), which cannot be directly used for self-supervised learning. Thus, we employ our previously developed compound image separation approach 30 to detect, separate, and curate subplots to individual images for downstream learning tasks. Using the compound figure separation approach, we acquired over 30,000 unannotated glomerular images via large web image mining.…”