Semantic description of medical image findings: structured learning approach

Abstract

Computer Aided Diagnosis (CADx) systems are designed to assist doctors in medical image interpretation. However, a CADx is often thought of as a "black box" whose diagnostic decision is not intelligible to a radiologist. Therefore, a system that uses semantic image interpretation, and mimics human image analysis, has clear benefits. In this paper, we propose a system which automatically generates textual description of medical image findings, such as lesions.Having found a lesion, a radiologist examines its visual appearance characteristics to make a final diagnosis. The visual appearance is usually described in terms of semantic descriptors such as: shape, orientation, margin, boundary type, contrast enhancement, localization, mass effect on surrounding tissues, and others.The estimation of semantic descriptor values requires explicit or implicit representation by a diverse set of image measurements that describe each one of the semantic descriptors quantitatively. We use various image measurements to calculate the informative features, such as histograms of pixel values, shape and texture descriptors and others.We pose the problem of semantic description of a lesion as learning to map a set of image based informative features to a set of semantic descriptor values. A lesion is described by a set of J semantic descriptors. Semantic description of the i-th lesion is an assignment where each j-th semantic descriptor can have one of the possible discrete values corresponding to the radiological lexicon. Following the standard practice in structured learning, the energy function of the above assignment is a sum of unary and pair-wise terms [1]:

Breast tumor segmentation and shape classification in mammograms using generative adversarial and convolutional neural network

“…The The classification results show that our second stage properly infers the tumor shape from the binary mask of the breast tumor, which was obtained from the first stage (cGAN segmentation). Hence, we have empirically shown that our CNN is focusing its learning on the morphological structure of the breast tumor, while the rest of approaches ( [10], [11], [45], [46]) rely on the original pixel variations of the input mammogram to make the same inference. Moreover, in [32] they used a hybrid strategy in which they include the pixel variability within the mask of breast tumor region to retain the intensity and texture information.…”

“…For a quantitative comparison, we compared our model with three state-of-the-art tumor shape classification methods [12,10,11]. The three methods were evaluated on the DDSM dataset.…”

“…In the second stage, the binary mask is classified to a shape type (irregular, lobular, oval and round). Unlike traditional object classifiers [10], [11] that use texture, intensity or edge information, our method is forced to learn only morphological features from the binary masks. To be more specific, we present a thorough improvement of our previous work [12].…”

“…Shape classification overall accuracy with the DDSM dataset resulting from[10,11,12] and our model. Best result is marked in bold.…”

Radiomics in immuno-oncology

“…Quantitative features are derived by applying advanced mathematical algorithms to the images (as in the case of radiomics). 2,21,22 Semantic and quantitative features can be used in statistical or artificial intelligence (AI) models to predict specific clinical endpoints. Individual radiomic features can be combined to form imaging 'signatures' or 'phenotypes'.…”