Consensus Versus Disagreement in Imaging Research: a Case Study Using the LIDC Database

Abstract

Traditionally, image studies evaluating the effectiveness of computer-aided diagnosis (CAD) use a single label from a medical expert compared with a single label produced by CAD. The purpose of this research is to present a CAD system based on Belief Decision Tree classification algorithm, capable of learning from probabilistic input (based on intra-reader variability) and providing probabilistic output. We compared our approach against a traditional decision tree approach with respect to a traditional performance metric (accuracy) and a probabilistic one (area under the distance-threshold curve-AuC dt ). The probabilistic classification technique showed notable performance improvement in comparison with the traditional one with respect to both evaluation metrics. Specifically, when applying crossvalidation technique on the training subset of instances, boosts of 28.26% and 30.28% were noted for the probabilistic approach with respect to accuracy and AuC dt , respectively. Furthermore, on the validation subset of instances, boosts of 20.64% and 23.21% were noted again for the probabilistic approach with respect to the same two metrics. In addition, we compared our CAD system results with diagnostic data available for a small subset of the Lung Image Database Consortium database. We discovered that when our CAD system errs, it generally does so with low confidence. Predictions produced by the system also agree with diagnoses of truly benign nodules more often than radiologists, offering the possibility of reducing the false positives.

Assessing diagnostic complexity: An image feature-based strategy to reduce annotation costs

“…In the computer-aided diagnosis (CAD) literature, consensus image interpretation is used as a standard of reference to provide the target class label to which the CAD method is compared [26]. For diagnosis of lung nodules, the application domain of this paper, most of the CAD systems use traditional classification techniques such as linear discriminant analysis [27][28][29][30][31][32][33], decision trees [34,35], and neural networks [36][37][38][39] to learn the class label from nodules' appearance, size, and shape image features.…”

Towards Achieving Diagnostic Consensus in Medical Image Interpretation

“…Consensus image interpretation is the standard approach taken to generate the final annotation when utilizing computeraided diagnosis (CADx) [19]. The consensus interpretation in image annotation is the agreement reached by two or more radiologists [20].…”

A method for evaluating the performance of computer-aided detection of pulmonary nodules in lung cancer CT screening: detection limit for nodule size and density

“…Performance evaluations of CAD systems for the detection of lung nodules have been carried out using clinical images including actual nodules. [2][3][4] When a CAD system is introduced in an institution providing CT screening, its performance should be evaluated by image data obtained under the same scan and image reconstruction conditions as those used at that site, because of the dependence of CAD performance on scan/ reconstruction conditions. [5][6][7] However, it would be difficult for end users of a CAD system to archive a large database of CT screenings with sufficient numbers of nodules at each screening site.…”