Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval

Abstract

In medical imaging, the characteristics purely derived from a disease should reflect the extent to which abnormal findings deviate from the normal features. Indeed, physicians often need corresponding images without abnormal findings of interest or, conversely, images that contain similar abnormal findings regardless of normal anatomical context. This is called comparative diagnostic reading of medical images, which is essential for a correct diagnosis. To support comparative diagnostic reading, content-based image retrieval (CBIR), which can selectively utilize normal and abnormal features in medical images as two separable semantic components, will be useful. Therefore, we propose a neural network architecture to decompose the semantic components of medical images into two latent codes: normal anatomy code and abnormal anatomy code. The normal anatomy code represents normal anatomies that should have existed if the sample is healthy, whereas the abnormal anatomy code attributes to abnormal changes that reflect deviation from the normal baseline. These latent codes are discretized through vector quantization to enable binary hashing, which can reduce the computational burden at the time of similarity search. By calculating the similarity based on either normal or abnormal anatomy codes or the combination of the two codes, our algorithm can retrieve images according to the selected semantic component from a dataset consisting of brain magnetic resonance images of gliomas. Our CBIR system qualitatively and quantitatively achieves remarkable results.

Implicit Feature Decoupling with Depthwise Quantization

“…Kobayashi et al [25] train a quantizer end-to-end with a DNN. They use multiple codebooks and train each codebook independently for a different supervised task.…”

Learning Disentangled Representations in the Imaging Domain

“…To address these problems, medical image synthesis is considered for augmenting and balancing these datasets. Besides, some images are impossible to acquire: doctors might wish to have an image of patient when the patient was healthy in order to perform a comparative diagnosis [82,148] (these hypothetical image estimations are also called counterfactuals).…”

“…Other works. Other applications include metal artifacts reduction [153], speckle noise reduction [154], multimodal synthesis [155,156,157], disease decomposition [82,148,158,159], explainability [160], controllable synthesis [126,161,162,163,164], and image harmonisation [165,166].…”

Deep learning application on head CT images

“…I attempted to extend the idea of VQVAE into disentanglement of the factors for medical imaging [106], that is to train the latent codes to represent the abnormalities in the medical image according to the image semantic. As it is commonly perceived, there is only a finite number of intensity levels or Hounsfield unit that represent an image while the usual practice of radiologists during scan inspection is to group a range of certain intensity values into a substance such as water, fat and bone.…”

“…Sun et al [94] does not dedicate a Segmentor such as Chapter 3. ICH disease decomposition on CT images 36 in [93] and above-mentioned works [106,121] to train for abnormality segmentation, the generator will learn to remove the abnormal region in the image to fool the discriminator. Since the model in [94] is designed such that true segmentation mask is not needed during test time, the target mask is only used in masked reconstruction loss to guide the generator to prevent one to many mapping by leaving the healthy region in the image unchanged.…”