Relation Classification with Cognitive Attention Supervision

Abstract

Many current language models such as BERT utilize attention mechanisms to transform sequence representations. We ask whether we can influence BERT's attention with human reading patterns by using eye-tracking and brain imaging data. We fine-tune BERT for relation extraction with auxiliary attention supervision in which BERT's attention weights are supervised by cognitive data. Through a variety of metrics we find that this attention supervision can be used to increase similarity between model attention distributions over sequences and the cognitive data without significantly affecting classification performance while making unique errors from the baseline. In particular, models with cognitive attention supervision more often correctly classified samples misclassified by the baseline.

On the Trade-off between Redundancy and Local Coherence in Summarization

“…On the other hand, tasks in NLP have benefited from cognitive science mainly in two aspects, the availability of datasets gathered during behavioral tests (Barrett et al, 2018;Mathias et al, 2021) and by leveraging cognitive theories for model design guidance. Firstly, eye tracking and brain activity data (captured by functional magnetic resonance imaging, fMRI, and electroencephalography, EEG) proved useful for a wide range of tasks such as sentiment analysis (Gu et al, 2014;Mishra et al, 2018), relation extraction (McGuire & Tomuro, 2021), name entity recognition , and text simplification (Klerke et al, 2016). Secondly, cognitive theories of text comprehension and production have guided model design for grammar induction and constituency parsing (Levy et al, 2008;Wintner, 2010), machine translation (Saini & Sahula, 2021), common-sense reasoning (Sap et al, 2020), and training strategies involving regularization (Wei et al, 2021) and curriculum learning (Xu et al, 2020).…”

The ZuCo benchmark on cross-subject reading task classification with EEG and eye-tracking data

“…For the present benchmark, this is possible by leveraging a naturalistic dataset of reading English sentences, the Zurich Cognitive Language Processing Corpus (Hollenstein et al, 2018 , 2020 ). The ZuCo dataset is publicly available and has recently been used in a variety of applications including leveraging EEG and eye-tracking data to improve NLP tasks (Barrett et al, 2018 ; Mathias et al, 2020 ; McGuire and Tomuro, 2021 ), evaluating the cognitive plausibility of computational language models (Hollenstein et al, 2019b ; Hollenstein and Beinborn, 2021 ), investigating the neural dynamics of reading (Pfeiffer et al, 2020 ), developing models of human reading (Bautista and Naval, 2020 ; Bestgen, 2021 ).…”

The ZuCo Benchmark on Cross-Subject Reading Task Classification with EEG and Eye-Tracking Data

“…For the present benchmark, this is possible by leveraging a naturalistic dataset of reading English sentences, the Zurich Cognitive Language Processing Corpus (Hollenstein et al, 2018, 2020). The ZuCo dataset is publicly available and has recently been used in a variety of applications including leveraging EEG and eye-tracking data to improve NLP tasks (Barrett et al, 2018; Mathias et al, 2020; McGuire and Tomuro, 2021), evaluating the cognitive plausibility of computational language models (Hollenstein et al, 2019b; Hollenstein and Beinborn, 2021), investigating the neural dynamics of reading (Pfeiffer et al, 2020), developing models of human reading (Bautista and Naval, 2020; Bestgen, 2021). Recently, ZuCo has also been leveraged for an ML competition on eye-tracking prediction (Hollenstein et al, 2021a).…”