Missed cancer and visual search of mammograms: what feature based machine-learning can tell us that deep-convolution learning cannot

Mall, Suneeta; Krupiński, Elizabeth A.; Mello‐Thoms, Claudia

doi:10.1117/12.2512539

Cited by 8 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The key advantage of eye-tracking glasses is that they are not affiliated with a specific computer screen and can be used in a broader class of clinical applications. One such application studied by Mall et al [ 82 ], [ 84 ], [ 85 ], [ 86 ] is mammography image analysis with two monitors, where monitors are used for craniocaudal and mediolateral oblique views. The authors used convolutional neural networks (CNNs) to predict the level of visual attention received by different breast regions.…”

Section: Resultsmentioning

confidence: 99%

“…Physician error prediction has been of particular interest in the community. The researchers investigated the gaze and image patterns that characterize breast lesions overlooked by mammologists [ 54 ], [ 76 ], [ 77 ], [ 85 ], [ 86 ], [ 87 ], [ 88 ], navigation loss in colonoscopy [ 46 ], surgical errors that result in injury [ 52 ], and overlooked lung nodules [ 21 ]. Decision errors are closely related to physician training, as eye movements are shown to be significantly different between experts and novices in the fields of endoscopic sinus surgery [ 49 ], [ 50 ], ultrasound video reading [ 30 ], [ 31 ], and shoulder arthroscopy [ 75 ].…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Ibragimov,

Mello-Thoms

2024

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

Machine learning (ML) has revolutionized medical image-based diagnostics. In this review, we cover a rapidly emerging field that can be potentially significantly impacted by MLeye tracking in medical imaging. The review investigates the clinical, algorithmic, and hardware properties of the existing studies. In particular, it evaluates 1) the type of eye-tracking equipment used and how the equipment aligns with study aims; 2) the software required to record and process eye-tracking data, which often requires user interface development, and controller command and voice recording; 3) the ML methodology utilized depending on the anatomy of interest, gaze data representation, and target clinical application. The review concludes with a summary of recommendations for future studies, and confirms that the inclusion of gaze data broadens the ML applicability in medical imaging from computer-aided diagnosis (CAD) to gaze-based image annotation, physicians' error detection, fatigue recognition, and other areas of potentially high research and clinical impact.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Ibragimov,

Mello-Thoms

2024

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

show abstract

“…Mall et al [29] modeled the visual search behavior of radiologists and their interpretation of mammography using CNNs. Furthermore, they [30] investigated the relationship between human visual attention and CNNs in finding lesions in mammography. Recently, Karargyris et al [14] developed a dataset with CXR, gaze, and text diagnosis reports.…”

Section: Eye Tracking In Radiologymentioning

confidence: 99%

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Ma¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. The core idea is that we infuse the visual attention information from expert radiologists to proactively guide the deep model to focus on regions with potential pathology and avoid being trapped in learning harmful shortcuts. To do so, we propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data. We mask the input image patches that are out of the radiologists' interest and add an additional residual connection in the last encoder layer of EG-ViT to maintain the correlations of all patches. The experiments on two public datasets of INbreast and SIIM-ACR demonstrate our EG-ViT model can effectively learn/transfer experts' domain knowledge and achieve much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the EG-ViT model's interpretability. In general, EG-ViT takes the advantages of both human expert's prior knowledge and the power of deep neural networks. This work opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.

show abstract

“…For example, [27] used eye gaze data to accomplish salient target segmentation, video action classification, and fine-grained image classification, respectively. In the literature, many works [53,35,36,56] have demonstrated the important role of eye gaze data in medical image analysis as well. For instance, [56] used the radiologists' eye gaze data to generate the attention map and used it to constrain the model's attention.…”

Section: Related Workmentioning

confidence: 99%

Rectify ViT Shortcut Learning by Visual Saliency

Ma¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Shortcut learning is common but harmful to deep learning models, leading to degenerated feature representations and consequently jeopardizing the model's generalizability and interpretability. However, shortcut learning in the widely used Vision Transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts, which are predominated by background related factors. For example, in the medical imaging field, eye-gaze data from radiologists is an effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions of interest. However, obtaining eye-gaze data is time-consuming, labor-intensive and sometimes even not practical. In this work, we propose a novel and effective saliency-guided vision transformer (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pre-trained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to distil the most informative image patches. In the proposed SGT, the self-attention Preprint. Under review.

show abstract

Missed cancer and visual search of mammograms: what feature based machine-learning can tell us that deep-convolution learning cannot

Cited by 8 publications

References 0 publications

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Rectify ViT Shortcut Learning by Visual Saliency

Contact Info

Product

Resources

About