Extracting semantic representations from word co-occurrence statistics: A computational study

Disentangling narrow and coarse semantic networks in the brain: The role of computational models of word meaning

“…Fyshe, Talukdar, Murphy, and Mitchell (2013) report peak performance with around 600 dimensions. Bullinaria and Levy (2007) report maximum performance around 1,000 dimensions. The authors of these two studies used PPMI instead of a raw co-occurrence frequency prior to dimension reduction and argued that PPMI may capture more semantic information in these higher dimensions than raw cooccurrence frequencies.…”

“…Additionally, in the HAL model multidimensional scaling was performed on a different set of semantic vectors from three different, superordinate, semantic categories, and the results indicated that the same vectors contain Bcategorical^semantic information. These and a number of other word similarity judgment tasks have remained common benchmarks for assessing the quality of semantic space models (Bullinaria & Levy, 2007;Fyshe et al, 2013;Rohde, Gonnerman, & Plaut, 2006). A number of similarity metrics have also been used for comparing vectors in semantic spaces; the Euclidean distance and the cosine similarity are among the most popular (see Rohde et al, 2006, for a detailed discussion).…”

“…Many factors are known to affect the quality of semantic space models such as the dimensionality of the semantic space (Bullinaria & Levy, 2007;Landauer & Dumais, 1997;Lund & Burgess, 1996), the method of dimensionality reduction (Bullinaria & Levy, 2012;Rohde, Gonnerman, & Plaut, 2006), the size of the window used to count co-occurrences (Fyshe et al, 2013), and factors such as whether or not to exclude function words (stop lists) and removing grammatical endings (stemming; Bullinaria & Levy, 2012). The proper number of dimensions is, for example, determined empirically in most cases, with many researchers reporting a number around 300 despite using different dimensionality reduction methods (Landauer & Dumais, 1997;Lund & Burgess, 1996).…”

“…For example, probabilistic LSA (Hoffman, 2001) and its fully Bayesian extension the Topic model (Griffiths, Steyvers, & Tenenbaum, 2007) can identify lexemes with multiple senses (Tomar et al, 2013) and generate semantic representations as probability distributions rather than points in a high-dimension space. Positive pointwise mutual information (PPMI) has been used in place of raw cooccurrence frequencies (Bullinaria & Levy, 2007). Zhao, Li, and Kohonen (2011) integrated these models into a selforganizing map framework, and Fyshe, Talukdar, Murphy, and Mitchell (2013) discussed how different types of constraints on what counts as a co-occurrence qualitatively affect semantic information.…”

Exploring Diachronic Lexical Semantics with JeSemE

“…Concordances are probably the most simple scheme to examine contextual semantic effects, but leave semantic inferences entirely to the human observer. A more complex layer is reached with collocations which can be identified automatically via statistical word co-occurrence metrics (Manning and Schütze, 1999;Wermter and Hahn, 2006), two of which are incorporated in JESEME as well: Positive pointwise mutual information (PPMI), developed by Bullinaria and Levy (2007) as an improvement over the probability ratio of normal pointwise mutual information (PMI; Church and Hanks (1990)) and Pearson's χ 2 , commonly used for testing the association between categorical variables (e.g., POS tags) and considered to be more robust than PMI when facing sparse information (Manning and Schütze, 1999).…”