Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Abstract

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. The core idea is that we infuse the visual attention information from expert radiologists to proactively guide the deep model to focus on regions with potential pathology and avoid being trapped in learning harmful shortcuts. To do so, we propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data. We mask the input image patches that are out of the radiologists' interest and add an additional residual connection in the last encoder layer of EG-ViT to maintain the correlations of all patches. The experiments on two public datasets of INbreast and SIIM-ACR demonstrate our EG-ViT model can effectively learn/transfer experts' domain knowledge and achieve much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the EG-ViT model's interpretability. In general, EG-ViT takes the advantages of both human expert's prior knowledge and the power of deep neural networks. This work opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

“…These numbers remain consistent for various algorithm types, including SVM, gradient boosting, and shallow and deep neural networks. Classification of lung pathologies using gaze as an additional data source is the second challenge that received considerable attention [62], [64], [69], [95]. The existing studies proposed to train neural networks to predict both abnormalities from chest X-rays and physician gaze maps, or to use gaze maps to train teacher networks in the student-teacher classification framework.…”

“…A summary of the papers that used machine learning in eye tracking in fundus photography, histopathology, surgical video, endoscopy, computed tomography (CT), magnetic resonance (MR) and optical coherence tomography (OCT) analysis. lesions [95], and detecting fetus anatomy [35], [36], [55]. A variation of this idea is to train ML algorithms to predict gaze data over target images together with the diagnosis of breast lesions [53], [95], lung diseases [68], [69], [70], brain tumors [44], [58], pancreatic tumors [96], diabetic retinopathy [47], and detecting fetus anatomy [29], [33], [40], which allows the algorithms to explicitly focus on anatomically important locations.…”

“…lesions [95], and detecting fetus anatomy [35], [36], [55]. A variation of this idea is to train ML algorithms to predict gaze data over target images together with the diagnosis of breast lesions [53], [95], lung diseases [68], [69], [70], brain tumors [44], [58], pancreatic tumors [96], diabetic retinopathy [47], and detecting fetus anatomy [29], [33], [40], which allows the algorithms to explicitly focus on anatomically important locations.…”

“…Following a different approach, Patra et al [34] first trained a CNN using both gaze maps and anatomy images as input and then fine-turned the network without gaze maps. Ma et al [95] used gaze maps to mask out regions in the anatomic image that did not receive the reader's attention so that the network stayed focused on the significant image regions. Finally, Ji et al [45] employed contrastive learning to match CNN activation maps and the corresponding gaze maps.…”