Exploring lexical co-occurrence space using HiDEx

Abstract

This work investigates a class of models of lexical semantics derived from the hyperspace analog to language (HAL; Burgess, 1998;Burgess & Lund, 2000), a computational model of word meaning that derives semantic relationships from lexical co-occurrence. Although the original HAL model was well specified, it contains several parameters whose values were set without formal or empirical justification. We have created a novel and freely available implementation of the HAL model-called High Dimensional Explorer (HiDEx)-that allows users to systematically vary those parameters, creating a class of models that are algorithmically identical but parameterized differently. Given the absence of any a priori formal justifications for parameter values in HAL, we have elected to assess different parameter settings by how well they perform at predicting human behavioral data on two lexical access tasks: lexical decision and semantic decision. In this article we explain how HiDEx works and how we were able to use it to explore HAL's parameter space.We begin with a brief overview of the HAL class of models and some related models. HAL uses word cooccurrence to build a vector space that contains contextual information for every word in a specified dictionary. A vector space is a geometric representation of data that has an ordered set of N numbers associated with each point in an N-dimensional space. Each such set of numbers defines the point's location in the space and is called its vector. HAL space is made up of vectors with one dimension for each word in the language. In the original HAL work, these word vectors had more than 100,000 dimensions.Each point in a word's vector is a weighted count of the number of times another word co-occurs with that word in a corpus of text. Words can co-occur when they are adjacent or when they are separated by a small number of intervening words. The maximum distance between words considered to co-occur is called the window size. Window size is one of the free parameters in the HAL model. In the original model, words were considered to have cooccurred if they occurred within 10 words of each other in either direction.Words in another word's co-occurrence window are weighted according to their proximity to that word, using a weighting function. The original HAL model used a linear weighting function called a linear ramp as a multiplier to give more weight to the words that co-occurred closer to the center of the window. Words that occurred on either side of the center word of the window were assigned 10 co-occurrence points. Each word's outside neighbors were assigned 9 co-occurrence points, and so on, down to a single point for a word that occurred 10 words away from the center word. This weighting function is another free parameter in HAL that has no a priori justification and can be changed in HiDEx.Lexical memories in the HAL model are built by making the model read words in text one window at a time and then sliding the window forward one word. This process of counting local co-occurrenc...

The principals of meaning: Extracting semantic dimensions from co-occurrence models of semantics

“…However, the idea that semantics can be expressed as a dimensioned space has not. Recent work with co-occurrence models of lexical semantics can be viewed as a technologically updated extension of much of Osgood's earlier work (e.g., Durda & Buchanan, 2008;Jones & Mewhort, 2007;Landauer & Dumais, 1997;Lund & Burgess, 1996;Rohde, Gonnerman, & Plaut, 2006;Shaoul & Westbury, 2010). Although their technical details vary, all of these models work from the basic assumption that the surrounding context of a word is informative of its meaning.…”

Performance impact of stop lists and morphological decomposition on word–word corpus-based semantic space models

“…Given that one of the aims of this study was to compare the performance impact of retaining stripped affixes as context dimensions, a larger window size was deemed appropriate (i.e., to handle inflected word forms with more than one prefix, like unpremeditated [un+ pre+ meditate +ed], or more than one suffix, like computerization [compute +er +ize +ation], while still capturing cooccurrences with adjacent words, which could also be surrounded by their own stripped affixes). Levy (2007, 2012) found that window sizes of up to three words in either direction still performed very well (i.e., major drops in performance were not noted in most of the tasks until reaching window sizes of four or five), and Shaoul and Westbury (2010) found that a window size of five words in either direction (using a linear ramp weighting scheme) led to optimal performance in a lexical decision task (also used in this study). We chose a window size of four words in either direction to accommodate affix retention, while minimizing the trade off in performance.…”

“…The High Dimensional Explorer implementation (HiDEx; Shaoul & Westbury, 2010;, an extension of HAL, was developed to address three major shortcomings of HAL. First, despite its vector normalization, HAL is prone to the influence of strong orthographic word frequency effects on vectors and vector distances (Shaoul & Westbury, 2006), and HiDEx uses improved vector normalization schemes to counter these effects.…”

“…In the corpus-based semantic space literature, it has been suggested that closed-class words, which are typically found in close proximity to target words (Shaoul & Westbury, 2010), are more syntactically related, rather than semantically related, to target words (Rohde et al, 2006). Unstructured cooccurrence models of lexical semantics generally ignore syntactical information in meaning creation, yet when human subjects perform semantic judgment tasks, they rely, in part, on syntactic properties such as word class (Rohde et al, 2006).…”

“…Relative to the DC and SC tasks, LD is a more difficult task for co-occurrence models-they typically account for slightly less than a third of the variance in LD reaction times (Buchanan, Westbury, & Burgess, 2001;Shaoul & Westbury, 2010). As such, the LD task was expected to provide a better scale-that is, a greater range of performanceto reveal differences between the 12 factor combinations.…”