Towards inferring cognitive state changes from pupil size variations in real world conditions

Medathati, N. V. Kartheek; Desai, Ruta; Hillis, James

doi:10.1145/3379155.3391319

Cited by 9 publications

(17 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Addressing the question whether pupil dilation may decode decisions on a single trial level, a few investigations using support vector machine classifiers (SVMs) have been put forward over the past years (Bednarik et al, 2012;Jangraw, Wang, Lance, Chang, & Sajda, 2014;Medathati et al, 2020). Although several articles have found random forest classifiers to be particularly useful for predictions also based on pupil dilation (Kootstra et al, 2020;Pasquali et al, 2020), no prior work has focused on predicting intentions only using pupil size.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Deriving with valid conclusions from pupil dilation usually requires highly controlled experiments, as also motor execution, such as a key press (Richer & Beatty, 1985), changes in gaze position, if not corrected for Hayes and Petrov (2016), and brightness affect pupil dilation. In the foregoing investigations most closely related to recognizing user intention, understood as the outcome of a binary decision on a stimulus' relevance from pupillometric data, changes in gaze position were not controlled for (Bednarik et al, 2012;Medathati et al, 2020). More important, however, key presses were necessary to inform about the intent to select.…”

Section: Theoretical Backgroundmentioning

confidence: 99%

“…Furthermore, the analysis and comparison of a set of potentially informative features of the pupil size signal may allow us to understand which signal components carry most information and can hence be considered worthwhile to be investigated with other types of analysis. Although the only two existing studies in this direction with a dedicated focus on pupil size (Bednarik, Vrzakova, & Hradis, 2012;Medathati, Desai, & Hillis, 2020) have Citation: Strauch, C., Hirzle, T., Van der Stigchel, S., & Bulling, A. (2021).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Decoding binary decisions under differential target probabilities from pupil dilation: A random forest approach

et al. 2021

View full text Add to dashboard Cite

Although our pupils slightly dilate when we look at an intended target, they do not when we look at irrelevant distractors. This finding suggests that it may be possible to decode the intention of an observer, understood as the outcome of implicit covert binary decisions, from the pupillary dynamics over time. However, few previous works have investigated the feasibility of this approach and the few that did, did not control for possible confounds such as motor-execution, changes in brightness, or target and distractor probability. We report on our efforts to decode intentions from pupil dilation obtained under strict experimental control on a single trial basis using a machine learning approach. The basis for our analyses are data of 69 participants who looked at letters that needed to be selected with stimulus probabilities that varied systematically in a blockwise manner (n = 19,417 trials). We confirm earlier findings that pupil dilation is indicative of intentions and show that these can be decoded with a classification performance of up to 76% area under the curve for receiver operating characteristic curves if targets are rarer than distractors. To better understand which characteristics of the pupillary signal are most informative, we finally compare relative feature importances. The first derivative of pupil size changes was found to be most relevant, allowing us to decode intention within only about 800 ms of trial onset. Taken together, our results provide credible insights into the potential of decoding intentions from pupil dilation and may soon form the basis for new applications in visual search, gaze-based interaction, or human-robot interaction.

show abstract

Section: Theoretical Backgroundmentioning

confidence: 99%

Section: Theoretical Backgroundmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Decoding binary decisions under differential target probabilities from pupil dilation: A random forest approach

et al. 2021

View full text Add to dashboard Cite

show abstract

“…We interpret the findings from reduced error rates using recurrent methods, relative to the naive, input mean estimate, to support the view that temporal memory is critical for accurate prediction of pupil diameter using eye tracking data. This accurate prediction may provide more opportunities for human-computer interaction through inferring cognitive state (Medathati et al, 2020 ). While, our videos' intrinsic characteristics (e.g., luminance, hue) may be highly correlated with video AOIs and this may extend to correlation with pupil diameter for bottom-up processes that rely on stimuli saliency, we believe this extension complements the goal of our study.…”

Section: Discussionmentioning

confidence: 99%

A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics

et al. 2021

View full text Add to dashboard Cite

There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal.

show abstract

“…Pupil study is crucial since it may benefit the evaluation of “special populations who may not be able or willing to provide a typical behavioral answer (complex motor or verbal responses) to certain research questions, such as pre-verbal infants, non-verbal adults, or children with ASD” (autistic spectrum disorder) (Sirois and Brisson, 2014 ). The close relationship between task-evoked pupil dilation and its underlying neural mechanisms enable the use of this method with participants of any age; knowledge about this relationship allows researchers to relate the neural system and cognitive studies and to interpret results of cognitive studies in terms of underlying neurophysiological processes (Eckstein et al, 2017 ; Medathati et al, 2020 ).…”

Section: Introductionmentioning

confidence: 99%

Pupil Reactions to Tactile Stimulation: A Systematic Review

2021

View full text Add to dashboard Cite

Pupil dynamics can represent an indirect measure of perception; thus, it has been broadly explored in the auditory and visual fields. Although it is crucial for experiencing the outside world, tactile perception is not well-explored. Considering that, we sought to answer the following question via a systematic review: does normal tactile perception processing modulate pupil dilation in mammals (human or not)? The review process was conducted according to PRISMA Statement. We searched on Periódicos CAPES (Brazil) for the following terms: [(touch) OR (cutaneous stimulation) OR (tactile perception) OR (somatosensory) AND (pupil OR pupillary) NOT blind NOT reflex NOT pain NOT fear NOT noxious NOT autism NOT nerve NOT (pupillary block) NOT glaucoma NOT cataract NOT aneurysm NOT syndrome NOT treatment NOT special education]. From the 6,488 papers found, 4,568 were duplicates, and nine fulfilled the inclusion criteria. All papers found a positive relationship between pupil diameter and tactile perception. We found that the pupil is a reliable indirect measure of brain states and can evaluate norepinephrine (NE)/locus coeruleus (LC) action, stimulus inhibition, arousal, cognitive processes, and affection independently of the stimuli category (visual, auditory, or tactile). We also found that the perceptual tactile processing occurs in similar ways as the other perceptual modalities. We verified that more studies should be done, mostly avoiding low sampling rate recording systems, confounders as cue signs, not automated stimulation, and concurrent stimulus and using more reliable equipment.

show abstract

Towards inferring cognitive state changes from pupil size variations in real world conditions

Cited by 9 publications

References 25 publications

Decoding binary decisions under differential target probabilities from pupil dilation: A random forest approach

Decoding binary decisions under differential target probabilities from pupil dilation: A random forest approach

A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics

Pupil Reactions to Tactile Stimulation: A Systematic Review

Contact Info

Product

Resources

About