Content-Based Image Retrieval Using Spatial Layout Information in Brain Tumor T1-Weighted Contrast-Enhanced MR Images

Abstract

This study aims to develop content-based image retrieval (CBIR) system for the retrieval of T1-weighted contrast-enhanced MR (CE-MR) images of brain tumors. When a tumor region is fed to the CBIR system as a query, the system attempts to retrieve tumors of the same pathological category. The bag-of-visual-words (BoVW) model with partition learning is incorporated into the system to extract informative features for representing the image contents. Furthermore, a distance metric learning algorithm called the Rank Error-based Metric Learning (REML) is proposed to reduce the semantic gap between low-level visual features and high-level semantic concepts. The effectiveness of the proposed method is evaluated on a brain T1-weighted CE-MR dataset with three types of brain tumors (i.e., meningioma, glioma, and pituitary tumor). Using the BoVW model with partition learning, the mean average precision (mAP) of retrieval increases beyond 4.6% with the learned distance metrics compared with the spatial pyramid BoVW method. The distance metric learned by REML significantly outperforms three other existing distance metric learning methods in terms of mAP. The mAP of the CBIR system is as high as 91.8% using the proposed method, and the precision can reach 93.1% when the top 10 images are returned by the system. These preliminary results demonstrate that the proposed method is effective and feasible for the retrieval of brain tumors in T1-weighted CE-MR Images.

Retrieval of Brain Tumors by Adaptive Spatial Pooling and Fisher Vector Representation

“…Following the experimental setup in Refs. [ 1 , 3 ], we randomly split the 233 patients into 5 subsets of roughly equal size. Partitioning according to the patient ensures that slices from the same patient will not simultaneously appear in the training set and test set.…”

“…To demonstrate the power of the proposed method, we compare it with three other brain tumor retrieval methods [ 1 – 3 ]. A brief overview of the three methods can be found in Section 1.…”

“…The power of traditional rigid distance functions, such as the Euclidean distance and cosine similarity, is limited because of the complexity of the image content and the sematic gap between low-level visual features and high-level human interpretation [ 28 , 29 ]. To alleviate this problem, numerous distance metric learning algorithms can be applied [ 1 , 3 , 30 – 33 ]. The primary idea of distance metric learning is to find an optimal metric that keeps intraclass samples close while separating interclass samples as much as possible.…”

“…Compared with text-based image retrieval, the CBIR can search query images from a database according to their visual content. Recent years have witnessed a growing interest in CBIR for various medical images, such as MRI images [ 1 – 3 ], CT images [ 4 ], mammograms [ 5 ], and pathology images [ 6 ]. In this paper, we focus on developing a CBIR system for retrieving MRI images of brain tumors to assist radiologists in the diagnosis of brain tumors.…”

“…Compared with the work by Yang [ 1 ], the retrieval performance is improved. In another work by Huang [ 3 ], a bounding box covering the brain tumor was used as the ROI, and a learned region partition method was applied. The raw image patches were used as local features and pooled per subregion with a BoW model.…”