How odd: Diverging effects of predictability and plausibility violations on sentence reading and word memory

Haeuser, Katja I.; Kray, Jutta

doi:10.1017/s0142716422000364

Cited by 10 publications

(3 citation statements)

References 82 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the one hand, sentence plausibility substantially facilitates language processing in humans (e.g., Bicknell et al., 2010; Federmeier & Kutas, 1999; Kutas & Hillyard, 1984; McRae & Matsuki, 2009). On the other hand, humans are also sensitive to lexical frequency effects when processing linguistic inputs (e.g., Broadbent, 1967; Goodkind & Bicknell, 2021; Haeuser & Kray, 2022; Rayner & Duffy, 1986) and can use both linguistic knowledge and event knowledge in real time depending on task demands (Willits, Amato, & MacDonald, 2015). As a result, LLM scores are a good predictor of human reading times (Oh & Schuler, 2023; Oh, Clark, & Schuler, 2022; Shain et al., 2022), neural predictability signatures like N400 (Michaelov, Bardolph, Van Petten, Bergen, & Coulson, 2023; Szewczyk & Federmeier, 2022), and brain response patterns to individual sentences (e.g., Caucheteux & King, 2022; Schrimpf et al., 2021; Tuckute et al., 2023).…”

Section: Discussionmentioning

confidence: 99%

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Kauf,

Ivanova,

Rambelli

et al. 2023

Cognitive Science

View full text Add to dashboard Cite

Word co‐occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs’ semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pretrained LLMs (from 2018's BERT to 2023's MPT) assign a higher likelihood to plausible descriptions of agent−patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n = 1215), we found that pretrained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign a higher likelihood to possible versus impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely versus unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow‐up analyses, we show that (i) LLM scores are driven by both plausibility and surface‐level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.

show abstract

Section: Discussionmentioning

confidence: 99%

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Kauf,

Ivanova,

Rambelli

et al. 2023

Cognitive Science

View full text Add to dashboard Cite

show abstract

“…In the face of a prediction violation, the hippocampus is proposed to switch to an encoding state to incorporate the sensory information (Mack et al, 2018;Sinclair et al, 2021). This suggests that prediction errors may drive the formation of new memories; a proposal that is supported by behavioural research demonstrating enhanced memory for incongruent stimuli such as reward prediction errors (Jang et al, 2019;Rouhani et al, 2018), surprising images (Greve et al, 2017;Michelon et al, 2003), bizarre sentences (Worthen & Roark, 2002), novel words (Habib et al, 2003), and unpredictable, implausible nouns (however, see Haeuser & Kray, 2022 for a discussion of this effect as being driven by the implausibility of the word, as opposed to the predictability itself). To further support the suggestion that the brain prioritises unpredictable information, Furutachi and colleagues (2023) observed amplified sensory representations in the visual cortex of mice for prediction erroreliciting stimuli.…”

Section: Predictive Processing Influences Memory Encodingmentioning

confidence: 99%

How predictability and individual alpha frequency shape memory: insights from an event-related potential investigation

Jano,

Chatburn,

Cross

et al. 2023

Preprint

View full text Add to dashboard Cite

Prediction and memory are strongly intertwined, with predictions relying on memory retrieval, whilst also influencing memory encoding. However, it is unclear how prediction errors during learning, as measured by the N400 event-related potential (ERP), influence explicit memory performance, and how individual neural factors may modulate this relationship. The current study sought to investigate the effect of prediction and individual neural variability on memory processing, whilst exploring the N400 as a prediction error signal in a context extending beyond language. Participants (N = 48, females = 33) completed a study-test paradigm where they first viewed predictable and unpredictable four-item ′ABCD′ sequences of outdoor scene images, whilst their brain activity was recorded using electroencephalography (EEG). Subsequently, their memory for the images was tested, and N400 patterns during learning were compared with memory outcomes. Strikingly, greater N400 amplitudes during learning were associated with enhanced memory at test for individuals with low versus high individual alpha frequencies (IAFs). This indicates that IAF may influence stimulus precision weightings and the extent to which prediction errors drive memory encoding. Memory was also strongest for predictable images in the ′B′ position, suggesting that when processing longer sequences, the brain may prioritise the data deemed most informative for predictive model updating. Ultimately, the results provide novel insight into the effect of prediction on memory, by highlighting the role of inter-subject variability, whilst shedding light on the accumulation of predictions across sequences.

show abstract

“…Accordingly, we manipulated whether stimuli violated expectations at the lexical level (Experiments 1 and 2) and at the semantic level (Experiment 3). Violation of expectations at the lexical and semantic levels have similar effects on markers of predictive processing like the N400, though effects of semantic violations are often observed later in processing (DeLong, Quante, & Kutas, 2014;Haeuser & Kray, 2022;Nieuwland et al, 2020;Quante, Bölte, & Zwitserlood, 2018).…”

Section: The Current Studymentioning

confidence: 99%

Discourse coherence modulates use of predictive processing during sentence comprehension

Carter¹,

Hoffman²

2022

Preprint

View full text Add to dashboard Cite

Context has been shown to be vitally important for comprehension. Lexical processing is facilitated when words are highly predictable given their local sentence context, suggesting that people pre-activate likely upcoming words to aid comprehension. However, this facilitation is affected by knowledge about the global context in which comprehension takes place: people predict less when in an environment where expectations are frequently violated. The current study investigated whether discourse coherence is an additional cue that comprehenders use to modulate lexical prediction. In a series of online, self-paced reading experiments, participants read target sentences preceded by short contextual preambles. Local facilitation effects were manipulated through the cloze predictability of a critical word within the target sentence and discourse coherence was manipulated by varying the degree to which the target sentence was consistent with the information presented in the preamble. In the first two experiments, target sentences were read more slowly when they occurred in less coherent discourses, but no local facilitation effects were observed. In the third experiment, we strengthened the predictability manipulation by using semantically anomalous critical words. In this experiment, facilitation did occur for predictable words but only when sentences appeared in a coherent discourse. When the target sentence deviated from the topic established by the preamble, highly predictable critical words were processed no more quickly than anomalous words. Our results suggest that comprehenders are sensitive to shifts in the topic of discourse and that they downregulate predictive processing when they encounter incoherence in the discourse. This is consistent with recent theoretical accounts suggesting that comprehenders flexibly engage in predictive processing, pre-activating words less when their expectations are less likely to be reliable.

show abstract

How odd: Diverging effects of predictability and plausibility violations on sentence reading and word memory

Cited by 10 publications

References 82 publications

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

How predictability and individual alpha frequency shape memory: insights from an event-related potential investigation

Discourse coherence modulates use of predictive processing during sentence comprehension

Contact Info

Product

Resources

About