Estimating the average need of semantic knowledge from distributional semantic models

Abstract

Continuous bag of words (CBOW) and skip-gram are two recently developed models of lexical semantics (Mikolov, Chen, Corrado, & Dean, Advances in Neural Information Processing Systems, 26, 3111-3119, 2013). Each has been demonstrated to perform markedly better at capturing human judgments about semantic relatedness than competing models (e.g., latent semantic analysis; Landauer & Dumais, Psychological Review, 104(2), 1997 211; hyperspace analogue to language; Lund & Burgess, Behavior Research Methods, Instruments, & Computers, 28(2), 203-208, 1996). The new models were largely developed to address practical problems of meaning representation in natural language processing. Consequently, very little attention has been paid to the psychological implications of the performance of these models. We describe the relationship between the learning algorithms employed by these models and Anderson's rational theory of memory (J. R. Anderson & Milson, Psychological Review, 96(4), 703, 1989) and argue that CBOW is learning word meanings according to Anderson's concept of needs probability. We also demonstrate that CBOW can account for nearly all of the variation in lexical access measures typically attributable to word frequency and contextual diversity-two measures that are conceptually related to needs probability. These results suggest two conclusions: One, CBOW is a psychologically plausible model of lexical semantics. Two, word frequency and contextual diversity do not capture learning effects but rather memory retrieval effects.

Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions

“…This network is trained on a corpus, word by word, and so the incremental development of distributional vectors over time is an inherent property of this algorithm. In the psychological literature, it has been demonstrated that these predicition-based models are mathematically equivalent to psychologically plausible learning models (Hollis, 2017;Mandera et al, 2017), such as the Rescorla-Wagner model of reinforcement learning (Rescorla & Wagner, 1972; see also Gluck & Bower, 1988;Sutton & Barto, 1981). It has also been shown that the word2vec model outperforms traditional count-based models in a variety of tasks, including the prediction of human behaviour (Baroni, Dinu, & Kruszewski, 2014;Mandera et al, 2017;Pereira, Gershman, Ritter, & Botvinick, 2016).…”

“…Interestingly, more recent work in the DSM literature has more thoroughly considered the question of acquisition. DSMs such as BEAGLE and word2vec have been shown to incorporate psychologically plausible learning mechanisms (Murdock, 1982;Rescorla & Wagner, 1972) to create their word representations, raising their algorithmic plausibility in comparison to earlier models (Hollis, 2017;Jones et al, 2015;Mandera et al, 2017, see the previous section). 3 Following from these arguments, DSMs should not be refuted from the outset as being mere engineering tools, as they are formulated as cognitive theories, rely on psychologically plausible assumptions, and are able to account for behavioural data.…”

Semantic transparency is not invisibility: A computational model of perceptually-grounded conceptual combination in word processing

“…Thus, we can in principle derive language-based representation for all words occurring in a language via the cbow model. In several works, the cbow model has been identified as a psychologically plausible model for the acquisition of semantic representations (Hollis, 2017;Mandera et al, 2017). In order to obtain reliable word embeddings, we only considered words with a frequency larger than 50 in the source corpus.…”

Semantic transparency effects in German compounds: A large dataset and multiple-task investigation

“…This algorithm trains distributional vectors as the hidden layer of a neural network predicting the n words surrounding a target word to each side (see Hollis, 2017;Mandera, Keuleers, & Brysbaert, 2017;Mikolov et al, 2013, for technical details). This model has SEMANTIC TRANSPARENCY OF GERMAN COMPOUNDS 11 been shown to implement psychologically plausible learning mechanisms for semantic representations (Hollis, 2017;Mandera et al, 2017).…”