Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density

Chang, Ken; Beers, Andrew; Brink, Laura; Patel, Jay; Singh, Praveer; Arun, Nishanth; Hoebel, Katharina; Gaw, Nathan; Shah, Meesam; Pisano, Etta D.; Tilkin, Mike; Coombs, Laura P.; Dreyer, Keith J.; Allen, Bibb; Agarwal, Sheela; Kalpathy–Cramer, Jayashree

doi:10.1016/j.jacr.2020.05.015

Cited by 48 publications

(34 citation statements)

References 43 publications

(50 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning has brought about many promising applications within medical imaging with recent studies showing its potential for key clinical assessments within dermatology, ophthalmology, pathology, oncology, cardiology, and radiology [1,2,3,4,5,6]. One major class of deep neural networks is convolutional neural networks (CNNs), which take raw pixel values as input, and transform them into the output of interest (such as diagnosis of disease [40] or expected clinical outcome [41]) after passing through many layers of non-linear transforms that provide the necessary capacity for higher levels of abstraction.…”

mentioning

confidence: 99%

Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Arun

Gaw

Singh

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Saliency maps have become a widely used method to make deep learning models more interpretable by providing post-hoc explanations of classifiers through identification of the most pertinent areas of the input medical image. They are increasingly being used in medical imaging to provide clinically plausible explanations for the decisions the neural network makes. However, the utility and robustness of these visualization maps has not yet been rigorously examined in the context of medical imaging. We posit that trustworthiness in this context requires 1) localization utility, 2) sensitivity to model weight randomization, 3) repeatability, and 4) reproducibility. Using the localization information available in two large public radiology datasets, we quantify the performance of eight commonly used saliency map approaches for the above criteria using area under the precision-recall curves (AUPRC) and structural similarity index (SSIM), comparing their performance to various baseline measures. Using our framework to quantify the trustworthiness of saliency maps, we show that all eight saliency map techniques fail at least one of the criteria and are, in most cases, less trustworthy when compared to the baselines. We suggest that their usage in the high-risk domain of medical imaging warrants additional scrutiny and recommend that detection or segmentation models be used if localization is the desired output of the network.

show abstract

mentioning

confidence: 99%

Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Arun

Gaw

Singh

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Today, abundant deep learning algorithms are being developed, and their development is now becoming easier. However, it is still difficult to develop a relatively broad, general, robust, and unbiased model [5][6][7]. Chang et al [7] recently compared the performances of algorithms trained with different formats of mammographs.…”

Section: Discussionmentioning

confidence: 99%

“…However, it is still difficult to develop a relatively broad, general, robust, and unbiased model [5][6][7]. Chang et al [7] recently compared the performances of algorithms trained with different formats of mammographs. They obtained radiographs from various clinics and divided them into three different formats according to the imaging system or the version of the imaging system used.…”

Section: Discussionmentioning

confidence: 99%

“…The major issues regarding its use are increasing the consistency and agreement of repeat analyses and enhancing software performance. One potential source of bias is the use of well-controlled cases for training and testing algorithms [5][6][7].…”

Section: Introductionmentioning

confidence: 99%

“…Various automatic diagnostic systems for panoramic radiography have been reported as well. These studies have been conducted using images obtained with only one type of panoramic radiography device [7][8][9][10][11]. Considering that hundreds of different models for panoramic radiography are used worldwide, non-biased and robust diagnostic model development is assumed to be as difficult as those of mammography and chest radiography.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep learning neural networks to differentiate Stafne’s bone cavity from pathological radiolucent lesions of the mandible in heterogeneous panoramic radiography

et al. 2021

View full text Add to dashboard Cite

This study aimed to develop a high-performance deep learning algorithm to differentiate Stafne’s bone cavity (SBC) from cysts and tumors of the jaw based on images acquired from various panoramic radiographic systems. Data sets included 176 Stafne’s bone cavities and 282 odontogenic cysts and tumors of the mandible (98 dentigerous cysts, 91 odontogenic keratocysts, and 93 ameloblastomas) that required surgical removal. Panoramic radiographs were obtained using three different imaging systems. The trained model showed 99.25% accuracy, 98.08% sensitivity, and 100% specificity for SBC classification and resulted in one misclassified SBC case. The algorithm was approved to recognize the typical imaging features of SBC in panoramic radiography regardless of the imaging system when traced back with Grad-Cam and Guided Grad-Cam methods. The deep learning model for SBC differentiating from odontogenic cysts and tumors showed high performance with images obtained from multiple panoramic systems. The present algorithm is expected to be a useful tool for clinicians, as it diagnoses SBCs in panoramic radiography to prevent unnecessary examinations for patients. Additionally, it would provide support for clinicians to determine further examinations or referrals to surgeons for cases where even experts are unsure of diagnosis using panoramic radiography alone.

show abstract

The development of “automated visual evaluation” for cervical cancer screening: The promise and challenges in adapting deep‐learning for clinical testing

Desai

Befano

Xue

et al. 2021

Intl Journal of Cancer

View full text Add to dashboard Cite

There is limited access to effective cervical cancer screening programs in many resource-limited settings, resulting in continued high cervical cancer burden. Human papillomavirus (HPV) testing is increasingly recognized to be the preferable primary screening approach if affordable due to superior long-term reassurance when negative and adaptability to self-sampling. Visual inspection with acetic acid (VIA) is an inexpensive but subjective and inaccurate method widely used in resource-limited settings, either for primary screening or for triage of HPV-positive individuals. A deep learning (DL)-based automated visual evaluation (AVE) of cervical images has been developed to help improve the accuracy and reproducibility of VIA as assistive technology. However, like any new clinical technology, rigorous evaluation and proof of clinical effectiveness are required before AVE is implemented widely. In the current article, we outline essential clinical and technical considerations involved in building a validated DL-based AVE tool for broad use as a clinical test.

show abstract

Multi-Institutional Assessment and Crowdsourcing Evaluation of Deep Learning for Automated Classification of Breast Density

Cited by 48 publications

References 43 publications

Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Assessing the (Un)Trustworthiness of Saliency Maps for Localizing Abnormalities in Medical Imaging

Deep learning neural networks to differentiate Stafne’s bone cavity from pathological radiolucent lesions of the mandible in heterogeneous panoramic radiography

The development of “automated visual evaluation” for cervical cancer screening: The promise and challenges in adapting deep‐learning for clinical testing

Contact Info

Product

Resources

About