Bridging Information-Seeking Human Gaze and Machine Reading Comprehension

Abstract

In this work, we analyze how human gaze during reading comprehension is conditioned on the given reading comprehension question, and whether this signal can be beneficial for machine reading comprehension. To this end, we collect a new eye-tracking dataset with a large number of participants engaging in a multiple choice reading comprehension task. Our analysis of this data reveals increased fixation times over parts of the text that are most relevant for answering the question. Motivated by this finding, we propose making automated reading comprehension more human-like by mimicking human information-seeking reading behavior during reading comprehension. We demonstrate that this approach leads to performance gains on multiple choice question answering in English for a state-of-the-art reading comprehension model.

Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics

“…In terms of limitations, we did not investigate the breadth or depth of influence of our method of [CLS]-based aggregate attention supervision on the model attentions across layers and heads, nor the supervision of specific layers or heads as done by Strubell et al (2018). We did not explore trade-off coefficients on the multiple losses, such as the convex combination used by Malmaud et al (2020). We used a relatively small English dataset, which limited generalizability and robustness.…”

“…Because BERT uses subword tokenization, to allow matching entries to be found in the ZuCo wordlevel data we split the ZuCo words into BERT tokens, evenly dividing values between each subword piece (e.g., "delicacy" → "del", "##ica", "##cy", each piece allotted a third of the ZuCo value), a technique used by Malmaud et al (2020). We preserve entity markers "<e>" and "</e>" in each sample by adding them as special tokens to the BERT tokenizer so their embeddings are learned with other tokens during fine-tuning.…”

“…For sentiment analysis, use MTL for a bidirectional Long Short-Term Memory (biLSTM) network, learning gaze behavior as the auxiliary task. Malmaud et al (2020) predict ET data with a variant of BERT as an auxiliary to question answering. Bautista and Naval (2020) predict gaze features with an LSTM to evaluate on sentiment classification and NER tasks.…”

Sentiment Analysis with Cognitive Attention Supervision

“…Obtaining human scores for sequence tokens requires preprocessing. As ZuCo uses a word-level lexicon but BERT uses subword tokenization, in order to find matching entries we first split the ZuCo words into BERT tokens, evenly dividing values between each subword piece (e.g., when tokenizing "delicacy" → ["del", "##ica", "##cy"] in sentence j, each piece is allotted a third of the ZuCo value), a technique previously used by [26]. We pass the human ET and EEG token values z ET and z EEG through a softmax layer to obtain two distributions over sentences, vectors α ′′ ET and α ′ EEG .…”

“…Cognitive NLP tasks that have been studied in recent years include sentiment analysis [23], part-of-speech (POS) tagging [24], and named entity recognition (NER) [25]. Typically for neural network-based approaches, a recurrent architecture such as a bidirectional Long Short-Term Memory (biLSTM) network has been used; more recently a variant of BERT was used for question answering with ET prediction as the auxiliary task [26]. MTL has been used for sentiment analysis and NER with learning gaze behavior as the auxiliary task [27,28], and a combination of gaze and brain data has been applied to a suite of NLP tasks [11], including sentiment analysis, using approaches such as predicting cognitive data or using those data to augment input embeddings.…”