Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study

Mall, Suneeta; Brennan, Patrick C.; Mello‐Thoms, Claudia

doi:10.1007/s10278-018-00174-z

Cited by 7 publications

(7 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key advantage of eye-tracking glasses is that they are not affiliated with a specific computer screen and can be used in a broader class of clinical applications. One such application studied by Mall et al [ 82 ], [ 84 ], [ 85 ], [ 86 ] is mammography image analysis with two monitors, where monitors are used for craniocaudal and mediolateral oblique views. The authors used convolutional neural networks (CNNs) to predict the level of visual attention received by different breast regions.…”

Section: Resultsmentioning

confidence: 99%

“…Physician error prediction has been of particular interest in the community. The researchers investigated the gaze and image patterns that characterize breast lesions overlooked by mammologists [ 54 ], [ 76 ], [ 77 ], [ 85 ], [ 86 ], [ 87 ], [ 88 ], navigation loss in colonoscopy [ 46 ], surgical errors that result in injury [ 52 ], and overlooked lung nodules [ 21 ]. Decision errors are closely related to physician training, as eye movements are shown to be significantly different between experts and novices in the fields of endoscopic sinus surgery [ 49 ], [ 50 ], ultrasound video reading [ 30 ], [ 31 ], and shoulder arthroscopy [ 75 ].…”

Section: Resultsmentioning

confidence: 99%

“…Radiologists make false positive (FP) and false negative (FN) diagnostic errors, where FN errors can be due to faulty search when the abnormality is never fixated; perceptual, when an abnormality is not disembedded from the background; or cognitive when the abnormality is seen but not correctly labeled. Mall et al [ 82 ], [ 84 ], [ 85 ], [ 86 ] bridged the diagnostic error nomenclature with eye tracking terminology and defined three types of areas of interest, namely foveal areas of prolonged visual dwelling with multiple fixations, peripheral areas of the short dwelling and never fixated areas. A CNN was able to accurately predict the level of interest mammography regions would receive [ 76 ].…”

Section: Resultsmentioning

confidence: 99%

See 2 more Smart Citations

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Ibragimov,

Mello-Thoms

2024

IEEE J. Biomed. Health Inform.

Self Cite

View full text Add to dashboard Cite

Machine learning (ML) has revolutionized medical image-based diagnostics. In this review, we cover a rapidly emerging field that can be potentially significantly impacted by MLeye tracking in medical imaging. The review investigates the clinical, algorithmic, and hardware properties of the existing studies. In particular, it evaluates 1) the type of eye-tracking equipment used and how the equipment aligns with study aims; 2) the software required to record and process eye-tracking data, which often requires user interface development, and controller command and voice recording; 3) the ML methodology utilized depending on the anatomy of interest, gaze data representation, and target clinical application. The review concludes with a summary of recommendations for future studies, and confirms that the inclusion of gaze data broadens the ML applicability in medical imaging from computer-aided diagnosis (CAD) to gaze-based image annotation, physicians' error detection, fatigue recognition, and other areas of potentially high research and clinical impact.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Ibragimov,

Mello-Thoms

2024

IEEE J. Biomed. Health Inform.

Self Cite

View full text Add to dashboard Cite

show abstract

“…However, they contribute to our study's decrease in FNs and the increase in TPs. Moreover, several studies in the literature have reported substantial reductions in FPs when employing fixation masks [52][53][54][55].…”

Section: Redundancy In Fixation Masksmentioning

confidence: 99%

EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion

Hsieh,

Luís,

Neves

et al. 2024

MAKE

View full text Add to dashboard Cite

Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradictory conclusions in the literature. Some authors assert that eye gaze data can enhance prediction accuracy, while others consider eye tracking irrelevant for predictive tasks. We argue that this disagreement lies in how researchers process eye-tracking data as most remain agnostic to the human component and apply the data directly to DL models without proper preprocessing. We present EyeXNet, a multimodal DL architecture that combines images and radiologists’ fixation masks to predict abnormality locations in chest X-rays. We focus on fixation maps during reporting moments as radiologists are more likely to focus on regions with abnormalities and provide more targeted regions to the predictive models. Our analysis compares radiologist fixations in both silent and reporting moments, revealing that more targeted and focused fixations occur during reporting. Our results show that integrating the fixation masks in a multimodal DL architecture outperformed the baseline model in five out of eight experiments regarding average Recall and six out of eight regarding average Precision. Incorporating fixation masks representing radiologists’ classification patterns in a multimodal DL architecture benefits lesion detection in chest X-ray (CXR) images, particularly when there is a strong correlation between fixation masks and generated proposal regions. This highlights the potential of leveraging fixation masks to enhance multimodal DL architectures for CXR image analysis. This work represents a first step towards human-centered DL, moving away from traditional data-driven and human-agnostic approaches.

show abstract

“…In contrast to traditional conventional visual image analysis, effective image features are automatically learned and extracted by DTL. Active research has been conducted on the application of DTL in terms of disease detection [ 6 ], classification [ 7 – 9 ] and evaluation of the response to different treatments [ 10 , 11 ].…”

Section: Introductionmentioning

confidence: 99%

Reducing the number of unnecessary biopsies for mammographic BI-RADS 4 lesions through a deep transfer learning method

Meng

Zhang

et al. 2023

BMC Med Imaging

View full text Add to dashboard Cite

Background In clinical practice, reducing unnecessary biopsies for mammographic BI-RADS 4 lesions is crucial. The objective of this study was to explore the potential value of deep transfer learning (DTL) based on the different fine-tuning strategies for Inception V3 to reduce the number of unnecessary biopsies that residents need to perform for mammographic BI-RADS 4 lesions. Methods A total of 1980 patients with breast lesions were included, including 1473 benign lesions (185 women with bilateral breast lesions), and 692 malignant lesions collected and confirmed by clinical pathology or biopsy. The breast mammography images were randomly divided into three subsets, a training set, testing set, and validation set 1, at a ratio of 8:1:1. We constructed a DTL model for the classification of breast lesions based on Inception V3 and attempted to improve its performance with 11 fine-tuning strategies. The mammography images from 362 patients with pathologically confirmed BI-RADS 4 breast lesions were employed as validation set 2. Two images from each lesion were tested, and trials were categorized as correct if the judgement (≥ 1 image) was correct. We used precision (Pr), recall rate (Rc), F1 score (F1), and the area under the receiver operating characteristic curve (AUROC) as the performance metrics of the DTL model with validation set 2. Results The S5 model achieved the best fit for the data. The Pr, Rc, F1 and AUROC of S5 were 0.90, 0.90, 0.90, and 0.86, respectively, for Category 4. The proportions of lesions downgraded by S5 were 90.73%, 84.76%, and 80.19% for categories 4 A, 4B, and 4 C, respectively. The overall proportion of BI-RADS 4 lesions downgraded by S5 was 85.91%. There was no significant difference between the classification results of the S5 model and pathological diagnosis (P = 0.110). Conclusion The S5 model we proposed here can be used as an effective approach for reducing the number of unnecessary biopsies that residents need to conduct for mammographic BI-RADS 4 lesions and may have other important clinical uses.

show abstract

Can a Machine Learn from Radiologists’ Visual Search Behaviour and Their Interpretation of Mammograms—a Deep-Learning Study

Cited by 7 publications

References 29 publications

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion

Reducing the number of unnecessary biopsies for mammographic BI-RADS 4 lesions through a deep transfer learning method

Contact Info

Product

Resources

About