Extrapolating human judgments from skip-gram vector representations of word meaning

Abstract

There is a growing body of research in psychology that attempts to extrapolate human lexical judgments from computational models of semantics. This research can be used to help develop comprehensive norm sets for experimental research, it has applications to large-scale statistical modelling of lexical access and has broad value within natural language processing and sentiment analysis. However, the value of extrapolated human judgments has recently been questioned within psychological research. Of primary concern is the fact that extrapolated judgments may not share the same pattern of statistical relationship with lexical and semantic variables as do actual human judgments; often the error component in extrapolated judgments is not psychologically inert, making such judgments problematic to use for psychological research. We present a new methodology for extrapolating human judgments that partially addresses prior concerns of validity. We use this methodology to extrapolate human judgments of valence, arousal, dominance, and concreteness for 78,286 words. We also provide resources for users to extrapolate these human judgments for three million English words and short phrases. Applications for large sets of extrapolated human judgments are demonstrated and discussed.

The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics

“…Arousal judgments from the ANEW norms set (Bradley & Lang, 1999) correlated with those in the Warriner et al (2013) norms set at r = .76 (as compared to .95 for valence and .80 for dominance). Attempts at algorithmically extrapolating human judgments have likewise found that human judgments of arousal are much less predictable than other semantic measures (Hollis & Westbury, 2016;Mandera, Keuleers, & Brysbaert, 2015;Recchia & Louwerse, 2015;Westbury et al, 2015). These findings, along with the results of our analysis of the skip-gram model, suggest that the concept of arousal is not as clearly specified as a semantic construct.…”

“…For instance, the similarity of meaning between two words can be assessed by measuring the similarity between their co-occurrence vectors. Cooccurrence vectors contain enough information to pass tests for basic verbal ability (e.g., Landauer & Dumais, 1997), to accurately predict human judgments of valence and arousal (e.g., Hollis & Westbury, 2016;Mandera, Keuleers, & Brysbaert, 2015;Recchia & Louwerse, 2015;Westbury, Keith, Briesemeister, Hofmann, & Jacobs, 2015), and to account for behavioral effects of high-level lexical properties such as subjective familiarity (Westbury, 2014) and imageability (Westbury et al, 2013).…”

Exploring What Is Encoded in Distributional Word Vectors: A Neurobiologically Motivated Analysis

“…For example, Mandera et al (2015) reported that, among four DSMs (i.e., LSA, topic model [Griffiths, Steyvers, & Tenenbaum, 2007], HAL [Lund & Burgess, 1996], and skip‐gram), concreteness ratings and age of acquisition, respectively, were best predicted by skip‐gram and HAL vectors when they were combined with the k ‐nearest neighbors method. Hollis et al (2017) also showed that skip‐gram vectors predicted concreteness even better using a step‐wise regression algorithm. Hollis and Westbury (2016) revealed that concreteness (and other semantic properties) was correlated with a specific set of dimensions of skip‐gram vectors obtained by principal component analysis.…”

“…Lexical or psycholinguistic properties (e.g., concreteness, age of acquisition, and imagery) have also been found to be predicted by DSMs (Hollis & Westbury, 2016; Hollis, Westbury, & Lefsrud, 2017; Mandera, Keuleers, & Brysbaert, 2015; Paetzold & Specia, 2016; Thompson & Lupyan, 2018). For example, Mandera et al (2015) reported that, among four DSMs (i.e., LSA, topic model [Griffiths, Steyvers, & Tenenbaum, 2007], HAL [Lund & Burgess, 1996], and skip‐gram), concreteness ratings and age of acquisition, respectively, were best predicted by skip‐gram and HAL vectors when they were combined with the k ‐nearest neighbors method.…”

IMS at EmoInt-2017: Emotion Intensity Prediction with Affective Norms, Automatically Extended Resources and Deep Learning

“…An alternative to discrete models are continuous models that map emotions to an n-dimensional space with valence, arousal and dominance (VAD) being usual dimensions. Previous works that rely on the VAD-scheme focus mainly on extending and adapting the affective lexicons (Bestgen and Vincze, 2012;Turney and Littman, 2003), including to historical texts (Buechel et al, 2016), and on the prediction and extrapolation of affective ratings (Recchia and Louwerse, 2015a;Hollis et al, 2017).…”