Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data

Abstract

Human gaze is a cost-efficient physiological data that reveals human underlying attentional patterns. The selective attention mechanism helps the cognition system focus on task-relevant visual clues by ignoring the presence of distractors. Thanks to this ability, human beings can efficiently learn from a very limited number of training samples. Inspired by this mechanism, we aim to leverage gaze for medical image analysis tasks with small training data. Our proposed framework includes a backbone encoder and a Selective Attention Network (SAN) that simulates the underlying attention. The SAN implicitly encodes information such as suspicious regions that is relevant to the medical diagnose tasks by estimating the actual human gaze. Then we design a novel Auxiliary Attention Block (AAB) to allow information from SAN to be utilized by the backbone encoder to focus on selective areas. Specifically, this block uses a modified version of a multi-head attention layer to simulate the human visual search procedure. Note that the SAN and AAB can be plugged into different backbones, and the framework can be used for multiple medical image analysis tasks when equipped with task-specific heads. Our method is demonstrated to achieve superior performance on both 3D tumor segmentation and 2D chest X-ray classification tasks. We also show that the estimated gaze probability map of the SAN is consistent with an actual gaze fixation map obtained by board-certified doctors.

The Use of Machine Learning in Eye Tracking Studies in Medical Imaging: A Review

“…Moreover, both databases follow a similar diagnostic protocol where the radiologists dictate the diagnoses during image reading. These public databases democratized the eye-tracking research for chest X-rays with eight papers published using IBM [58], [63], [64], [65], [66], or REFLACX data [68], [69], [70]. Not only does the number of patients analyzed grow, but the number of physicians who read the images also grows.…”

“…There are many commercially available eye trackers with the most prominent providers being Tobii, Argus Science, and Gazepoint. Among the studies analyzed in this review, the researchers utilized remote eye trackers, namely Tobii 1750 [17], [18], [19], Tobii x50 [20], [21], Tobii 4C [13], [14], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], Tobii EyeX [42], [43], Tobii Pro Nano [44], [45], Tobii Pro Fusion [13], [38], [39], Tobii X-60 [46], Tobii Pro Spectrum [47], [48] in addition to SMI RED [49], [50], [51], [52], [53], Mirametrix S2 [54], EyeTribe [55], [56], [57], [58], Fovio…”

“…lesions [95], and detecting fetus anatomy [35], [36], [55]. A variation of this idea is to train ML algorithms to predict gaze data over target images together with the diagnosis of breast lesions [53], [95], lung diseases [68], [69], [70], brain tumors [44], [58], pancreatic tumors [96], diabetic retinopathy [47], and detecting fetus anatomy [29], [33], [40], which allows the algorithms to explicitly focus on anatomically important locations.…”

“…The researchers, therefore, investigated several strategies for gaze-assisted CAD where gaze data is only used for CNN training and not required during CNN deployment. The simplest way to achieve this aim is to move gaze heatmaps from CNN input to CNN output [28], [33], [44], [56], [58], [62], [63], [65], [97]. During training, such a CNN both predicts the observed abnormality and generates gaze maps, explicitly learning the areas of high visual attention.…”