A shape-dependent variability metric for evaluating panel segmentations with a case study on LIDC

Abstract

The segmentation of medical images is challenging because a ground truth is often not available. Computer-Aided Detection (CAD) systems are dependent on ground truth as a means of comparison; however, in many cases the ground truth is derived from only experts' opinions. When the experts disagree, it becomes impossible to discern one ground truth. In this paper, we propose an algorithm to measure the disagreement among radiologist's delineated boundaries. The algorithm accounts for both the overlap and shape of the boundaries in determining the variability of a panel segmentation. After calculating the variability of 3788 thoracic computed tomography (CT) slices in the Lung Image Database Consortium (LIDC), we found that the radiologists have a high consensus in a majority of lung nodule segmentations. However, our algorithm identified a number of segmentations that the radiologists significantly disagreed on. Our proposed method of measuring disagreement can assist others in determining the reliability of panel segmentations. We also demonstrate that it is superior to simply using overlap, which is currently one of the most common ways of measuring segmentation agreement. The variability metric presented has applications to panel segmentations, and also has potential uses in CAD systems.

Assessing diagnostic complexity: An image feature-based strategy to reduce annotation costs

“…The union of the most important features from the two data distributions (balanced and unbalanced) resulted in a set of 21 features that included 8 uncorrelated features. Then three classification models were created on all features (42), most important features (21), and most important uncorrelated features (8). Table 6 shows the results for the combination of trees parameters that resulted in the highest average accuracy for balanced and unbalanced datasets, and for each feature set.…”

“…The significance of relating image content to the human perception and cognition was also recently shown in [7] where certain image features where found to be important in predicting diagnostic error for mass interpretation in mammograms. Furthermore, segmentation of lesions can also benefit from the proposed approach given that in many cases the ground truth is derived from only experts' delineated boundaries [8]. Our approach can help determine how many outlines to aggregate (for example, using a p-map approach as defined in [9]) until a reliable region of interest is identified for the lesion.…”

Extracting complex lesion phenotypes in Zea mays

“…Manual segmentation may be feasible for small numbers of images with relatively few instances of the phenotypes, and for phenotypes that fall into very distinct classes. Usually, experts have time to segment only a few images, limiting the sample size and therefore the statistical power of an experiment; they disagree in their segmentations; and how well a segmenter performs can vary depending on the metric used to measure them [9][10][11][12]. Using manual segmentation in high throughput situations requires many more people, further complicating inter-and intra-observer consistency [8,13].…”