Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks

Bhattacharya, Nilavra; Rakshit, Somnath; Gwizdka, Jacek; Kogut, Paul

doi:10.1145/3343413.3377960

Cited by 25 publications

(12 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A potential solution could be to use higher-level features such as the thorough reading ratio, i.e., the ratio of read and skimmed text lengths (Buscher et al, 2012), or the refixation count, i.e., the number of re-visits to a certain paragraph (Feit et al, 2020). Another solution could be found in using scanpath encodings based deep learning Castner et al (2020); Bhattacharya et al (2020b). We envision the gazebased relevance detection to be a part of future adaptive UIs that leverage multiple sensors for behavioral signal processing and analysis Oviatt et al (2018); Barz et al (2020a,b).…”

Section: Discussionmentioning

confidence: 99%

“…Jacob et al (2018) investigated whether eye movements can be used to infer the interest of a reader in a currently read article. Bhattacharya et al (2020b) encoded fixations from participants' scanpaths over documents from the g-REL corpus and trained a convolutional neural network (CNN) with the perceived relevance as prediction target. This approach is limited to small texts of similar lengths.…”

Section: Relevance Estimation From Reading Behaviormentioning

confidence: 99%

See 1 more Smart Citation

Implicit Estimation of Paragraph Relevance From Eye Movements

Barz

Bhatti

Sonntag

2022

Front. Comput. Sci.

View full text Add to dashboard Cite

Eye movements were shown to be an effective source of implicit relevance feedback in constrained search and decision-making tasks. Recent research suggests that gaze-based features, extracted from scanpaths over short news articles (g-REL), can reveal the perceived relevance of read text with respect to a previously shown trigger question. In this work, we aim to confirm this finding and we investigate whether it generalizes to multi-paragraph documents from Wikipedia (Google Natural Questions) that require readers to scroll down to read the whole text. We conduct a user study (n = 24) in which participants read single- and multi-paragraph articles and rate their relevance at the paragraph level with respect to a trigger question. We model the perceived document relevance using machine learning and features from the literature as input. Our results confirm that eye movements can be used to effectively model the relevance of short news articles, in particular if we exclude difficult cases: documents which are on topic of the trigger questions but irrelevant. However, our results do not clearly show that the modeling approach generalizes to multi-paragraph document settings. We publish our dataset and our code for feature extraction under an open source license to enable future research in the field of gaze-based implicit relevance feedback.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Relevance Estimation From Reading Behaviormentioning

confidence: 99%

Implicit Estimation of Paragraph Relevance From Eye Movements

Barz

Bhatti

Sonntag

2022

Front. Comput. Sci.

View full text Add to dashboard Cite

show abstract

“…The VGG-19 architecture was chosen as it is a relatively shallow network with multiple small kernels which we hypothesised would be optimal for capturing any nuanced differences between the input images. Our hypothesis stems from results in an earlier paper which tests similarly a scanpath design on a wide range of out-of-the-box neural network models during a reading task 10 . To further validate our preferred VGG-19 method, we also report the results of two benchmark cases: a Support Vector Machine (SVM) configured for image classification, commonly used in scanpath classification tasks and a logistic regression model which is the most common method of traditional analysis to test the link between gaze data and choice behavior in games presented in normal-form.…”

Section: Model Selection Modelling Tasks Performance Metricsmentioning

confidence: 99%

“…Further, recent advances in Machine Learning (ML) techniques has led a significant increase in the accuracy of prediction when modelling gaze data 1,[6][7][8] . ML techniques have been applied to eye-tracking data to model human cognition in a variety of settings, including -but not limited to -detecting sarcasm 9 , identifying when a participant is in a state of confusion 7 , classifying the relevance of a passage text to a user 10 , and predicting where a participant will focus their attention during location-based games 11 . Further, humans are more frequently interacting with automated systems when engaging in strategic contexts which is a phenomenon that has been noticed by policy makers 12,13 .…”

Section: Introductionmentioning

confidence: 99%

Predicting Choice Behaviour in Economic Games using Gaze Data Encoded as Scanpath Images

Byrne

Reynolds

Biliotti

et al. 2022

Preprint

View full text Add to dashboard Cite

Eye movement data has been extensively utilized by researchers interested in studying decision-making within the strategic setting of economic games. In this paper, we demonstrate both a deep learning and traditional machine learning classification method which are able to accurately identify a given participant's decision strategy before they commit to an action while playing games. Our approach focuses on creating scanpath images that best capture the dynamics of a participant's gaze behaviour during a given game in a way that is meaningful to the machine learning models. Our results demonstrate a higher classification accuracy compared to traditional methods of analysis applied to the same economic game environments by as much as 18 percentage points. In a broader context, we aim to illustrate the potential for eye-tracking data to create information asymmetries in strategic environments in favour of those who collect and process the data. These information asymmetries could become especially relevant as eye-tracking is expected to become more widespread in user applications, with the seemingly imminent mass adoption of virtual reality systems, and the development of devices with the ability to record eye movement outside of a laboratory setting.

show abstract

“…This opened opportunities for adaptive user interfaces. A large body of work has focused on enhancing the query-based search of images [Faro et al 2010;Klami 2010;Klami et al 2008] or text-documents [Aula et al 2005;Bhattacharya et al 2020;Buscher et al 2008;Dumais et al 2010]. There, information about eye gaze provides feedback on the relevance of the displayed search results or text documents.…”

Section: Ui Adaptation From Gaze Behaviormentioning

confidence: 99%

Detecting Relevance during Decision-Making from Eye Movements for UI Adaptation

Feit

Vordemann

Park

et al. 2020

ACM Symposium on Eye Tracking Research and Applications

View full text Add to dashboard Cite

This paper proposes an approach to detect information relevance during decision-making from eye movements in order to enable user interface adaptation. This is a challenging task because gaze behavior varies greatly across individual users and tasks and groundtruth data is difficult to obtain. Thus, prior work has mostly focused on simpler target-search tasks or on establishing general interest, where gaze behavior is less complex. From the literature, we identify six metrics that capture different aspects of the gaze behavior during decision-making and combine them in a voting scheme. We empirically show, that this accounts for the large variations in gaze behavior and out-performs standalone metrics. Importantly, it offers an intuitive way to control the amount of detected information, which is crucial for different UI adaptation schemes to succeed. We show the applicability of our approach by developing a room-search application that changes the visual saliency of content detected as relevant. In an empirical study, we show that it detects up to 97% of relevant elements with respect to user self-reporting, which allows us to meaningfully adapt the interface, as confirmed by participants. Our approach is fast, does not need any explicit user input and can be applied independent of task and user. CCS CONCEPTS • Human-centered computing → HCI theory, concepts and models; Empirical studies in HCI.

show abstract

Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks

Cited by 25 publications

References 61 publications

Implicit Estimation of Paragraph Relevance From Eye Movements

Implicit Estimation of Paragraph Relevance From Eye Movements

Predicting Choice Behaviour in Economic Games using Gaze Data Encoded as Scanpath Images

Detecting Relevance during Decision-Making from Eye Movements for UI Adaptation

Contact Info

Product

Resources

About