Evaluation Challenges for Bridging Semantic Gap

Abstract

Evaluating the success of prediction and retrieval systems depends upon a reliable reference standard, a ground truth. The ideal gold standard is expected to result from the marking, labeling, and rating of domain experts of the image of interest. However experts often disagree and this lack of agreement challenges the development and evaluation of image-based feature prediction of expert-defined "truth." This paper addresses the success and limitations in bridging the semantic gap between CT-based pulmonary nodule image features and the ratings of diagnostic characteristics recorded by expert pulmonary radiologists. The prediction of diagnostic characteristics promises to automatically annotate medical images with medically meaningful descriptions directly usable for indexing and retrieving in content-based image retrieval (CBIR) and assisting in computer aided diagnosis (CADx). Successful results in predicting texture characteristics will be contrasted with less successful results for boundary shapes. The two primary differences in agreement between radiologists will be discussed; the first concerns agreement about the existence of a nodule, while the second considers the variability in diagnostic ratings among radiologists who agree on the presence of a nodule.

Toward Understanding the Size Dependence of Shape Features for Predicting Spiculation in Lung Nodules for Computer-Aided Diagnosis

“…In this context, unlike studies that have as their goal demonstrating the complete performance of a CAD system, we are looking into gaining fundamental insight into the shape feature calculations. Rather than investigating the best combination between a set of image features and a classifier to obtain optimal classification performance, we propose to analyze the effects of geometric measurements (e.g., radii or area measurements) on basic shape features that have simple geometric interpretations and were used in several LIDC studies [6][7][8][9][10][11]. This is in particular important because early digital imaging research established that the accuracy of geometric measurements in the digital plane depends on resolution.…”

“…Table 4 presents the formulas for the eight shape features: solidity, roughness, second moment, radial distance standard deviation, circularity, compactness, extent, and eccentricity. These shape features were chosen because they were used in previous CAD studies based on LIDC data [6][7][8][9][10][11], because they are specifically related to distinguishing levels of shape complexity (vs the harder problem of distinguishing between shapes in general), and because they have a simple geometric interpretation. The simple geometric interpretation allows for analysis of the effects of image resolution on geometric measurements that comprise the shape features and the effects of noise associated with uncertainty in statistical calculations.…”

“…While a naïve assumption is that shape features would correlate with shape semantic characteristics, such as spiculation, in early studies using the LIDC dataset, the shape of the nodule segmentation was not highly correlated with semantic ratings of radiologists [6][7][8][9][10][11]. The reasons for this semantic gap have been attributed to uncertainty in the location of the boundary as captured by radiologist annotation variability and uncertainty in the semantic rating, since different raters may provide different ratings for spiculation for the same nodule.…”

“…New shape features that included Fourier-based descriptors, a variant of the radial gradient index, and a radial normal index were used to predict semantic ratings, including spiculation [11,25]. The authors reported no improvement in predicting semantic ratings, also citing the extent of radiologist disagreement as the primary cause of the poor classification performance.…”

“…Since high inter-observer variation was the primary cause of poor classifier performance cited in several of the studies [6,7,10,11,24,25,28], we propose a new approach of first filtering out inter-observer variation in order to reveal fundamental relationships between shape features and the semantic concept of spiculation. In this context, unlike studies that have as their goal demonstrating the complete performance of a CAD system, we are looking into gaining fundamental insight into the shape feature calculations.…”