A Psycholinguistic Analysis of BERT's Representations of Compounds

Abstract

This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e.g., sunlight) whose overall meaning depends-to a various extent-on the semantics of the constituent words (sun, light). We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD; quantifying the weight of each constituent toward the compound meaning) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents' semantics). We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for 'standard' words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide insights into how speakers represent compounds.

Meaning Modulations and Stability in Large Language Models: An Analysis of BERT Embeddings for Psycholinguistic Research

“…We thus do observe that different layers in BERT's architecture capitalise on and highlight different pieces of information extracted from co-occurrence. Even if BERT learns static representations, therefore, deriving prototypes at mid layers through averaging seems to capture semantic associations better, as evaluated using human ratings, in line with previous evidence that lexical semantics is best captured around layer 8 (Buijtelaar & Pezzelle, 2023;Rogers et al, 2020). On the contrary, non-contextualised prototypes' poor performance at deeper layers, i.e.…”

“…This core, moreover, can shifts following predictable and systematic context effects. We highlight that static and context-dependent representations co-exist in the same embedding space: in this, BERT offers a viable single implementation of both, and holds promise for analysing the interplay and interplay between prototypes and exemplars in a learning model (Buijtelaar & Pezzelle, 2023;Madabushi et al, 2022).…”

“…Following common practice in computer vision models (Günther et al, in press), where representations of specific images depicting a concept are combined to obtain a prototype representation of the concept, vectors of specific word tokens in context can be averaged to obtain an abstract representation at the type level of the same word (Bommasani et al, 2020;Lenci et al, 2022a). Unlike fully static embeddings, these representations combine context-dependent, token-level representations and have been often used as a proxy for type-level representations in NLP studies (Buijtelaar & Pezzelle, 2023;Lenci et al, 2022a). The intuition is that abstractions at different layers capture different linguistic aspects, with more superficial layers focusing on form-related patterns, mid-layers capturing structural and semantic aspects of tokens, and deeper layers focusing on discourseand sentence-level patterns (Rogers et al, 2020).…”

“…Next to averaging token-level representations and considering the initial, fully static representation on which contextualisation happens one can also derive type-level representations by presenting the model with the word in isolation, without any context, or in sentences which carry little to no contextual information, e.g. This is a X or They X, with X being replaced by any word form (Bommasani et al, 2020;Buijtelaar & Pezzelle, 2023). The assumption is that the absence of informative co-occurring words lets the model yield the representation of the word itself.…”