What company do words keep? Revisiting the distributional semantics of J.R. Firth &amp; Zellig Harris

Brunila, Mikael; LaViolette, Jack

doi:10.18653/v1/2022.naacl-main.327

Cited by 7 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Distributional semantics methods allow us to quantify word meanings without specifying semantic features (e.g., that a door and gate are both "movable barriers"; Rogers & McClelland, 2004;Miller, 1995). Distributional techniques presuppose that two words that typically occur in similar (distributional) contexts are probably similar in meaning and thus have similar vector representations (Brunila & LaViolette, 2022). The similarity in meaning between two words is approximated by some measure of distance (e.g., Euclidean distance or cosine similarity) between the vectors, with more distant vectors being less similar in meaning.…”

Section: The Cloze Task and Large Language Models (Llms)mentioning

confidence: 99%

Uncovering patterns of semantic predictability in sentence processing

Jacobs,

Hubbard,

Federmeier

et al. 2024

Preprint

View full text Add to dashboard Cite

Psycholinguistic researchers have used the cloze task to measure the predictability of upcoming words, but have largely discarded the variability in the structure of responses people provide. This variability in the semantic structure of responses may be important for understanding selection during language production; however, it has proven difficult to model the semantic variability of participants' responses, and thus upcoming semantic uncertainty. Recent advances in large language models (LLMs) permit us to approximate the degree of semantic variability in cloze responses, but most methods are restricted to symbolic or hand-crafted meaning representations. We show in two studies that Bayesian Gaussian mixture models can cluster LLM representations of participants' responses and produce coherent, taxonomically similar clusters. We apply these clustering algorithms to response time data in a serial cloze task and show that the semantic structure of cloze responses influences how quickly people are able to provide a response. We show clear effects of semantic competition on naming speed. In addition to providing novel operationalizations of what semantic competition might look like in the cloze task, we explain how this clustering method is extensible to other datasets and applications of interest to researchers of semantic processing in psycholinguistics.

show abstract

Section: The Cloze Task and Large Language Models (Llms)mentioning

confidence: 99%

Uncovering patterns of semantic predictability in sentence processing

Jacobs,

Hubbard,

Federmeier

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Similar advances occurred in NLP, which refers to the branch of AI Zygon focused on developing software to understand and generate natural language (such as text and spoken language). NLP built upon a shift in understanding "meaning" in computational linguistics from depending upon symbol references to deriving from word cooccurrence and shared contexts (Brunila and LaViolette 2022). This shift to associative and distributional theories of semantics (Firth 1957;Harris 1968;Sahlgren 2008) facilitated statistical methods for processing and understanding natural language and enabled significant improvements in modeling language and performing language-related tasks (Sejnowski 2020;Goldberg 2016;Bommasani et al 2022).…”

Section: Paradigms For Ai Researchmentioning

confidence: 99%

Embodied Experience in Socially Participatory Artificial Intelligence

Graves¹

2023

Zygon

View full text Add to dashboard Cite

As artificial intelligence (AI) becomes progressively more engaged with society, its shift from technical tool to participating in society raises questions about AI personhood. Drawing upon developmental psychology and systems theory, a mediating structure for AI proto‐personhood is defined analogous to an early stage of human development. The proposed AI bridges technical, psychological, and theological perspectives on near‐future AI and is structured by its hardware, software, computational, and sociotechnical systems through which it experiences its world as embodied (even for putatively disembodied AI). Further social and moral construction can occur building upon a simple “self” for AI synthesized from symbolic and statistical approaches to AI.

show abstract

“…The distributional semantics (Harris, 1954;Firth, 1957) suggests that "similar words will appear in similar contexts" (Brunila and LaViolette, 2022). This implies that the conditional probability distribution p(•|w) represents the meaning of a word w.…”

Section: Kl Divergence Measures Information Gainmentioning

confidence: 99%

Norm of Word Embedding Encodes Information Gain

Oyama,

Yokoi,

Shimodaira

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Distributed representations of words encode lexical semantic information, but what type of information is encoded and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. This theory also extends to contextualized word embeddings in language models or any neural networks with the softmax output layer. We also demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of the informativeness of a word in tasks such as keyword extraction, proper-noun discrimination, and hypernym discrimination.

show abstract

What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris

Cited by 7 publications

References 15 publications

Uncovering patterns of semantic predictability in sentence processing

Uncovering patterns of semantic predictability in sentence processing

Embodied Experience in Socially Participatory Artificial Intelligence

Norm of Word Embedding Encodes Information Gain

Contact Info

Product

Resources

About