Learning word embeddings from dependency relations

Zhao, Yinggong; Huang, Shujian; Dai, Xinyu; Zhang, Jianbing; Chen, Jiajun

doi:10.1109/ialp.2014.6973490

Cited by 10 publications

(7 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, PPMI is conceptually simple and fast to calculate, so we chose it over the newer alternatives. 4 There are some existing word embeddings that take dependency structure into account, for example, Gamallo (2017), Levy and Goldberg (2014a), Padó and Lapata (2007), Rei and Briscoe (2014), and Zhao, Huang, Dai, Zhang, and Chen (2014). However, these embeddings do not encode the information that we need for use in cognitive models of sentence processing.…”

Section: Methodsmentioning

confidence: 99%

A Principled Approach to Feature Selection in Models of Sentence Processing

Smith

Vasishth

2020

Cognitive Science

View full text Add to dashboard Cite

Among theories of human language comprehension, cue‐based memory retrieval has proven to be a useful framework for understanding when and how processing difficulty arises in the resolution of long‐distance dependencies. Most previous work in this area has assumed that very general retrieval cues like [+subject] or [+singular] do the work of identifying (and sometimes misidentifying) a retrieval target in order to establish a dependency between words. However, recent work suggests that general, handpicked retrieval cues like these may not be enough to explain illusions of plausibility (Cunnings & Sturt, 2018), which can arise in sentences like The letter next to the porcelain plate shattered. Capturing such retrieval interference effects requires lexically specific features and retrieval cues, but handpicking the features is hard to do in a principled way and greatly increases modeler degrees of freedom. To remedy this, we use well‐established word embedding methods for creating distributed lexical feature representations that encode information relevant for retrieval using distributed retrieval cue vectors. We show that the similarity between the feature and cue vectors (a measure of plausibility) predicts total reading times in Cunnings and Sturt’s eye‐tracking data. The features can easily be plugged into existing parsing models (including cue‐based retrieval and self‐organized parsing), putting very different models on more equal footing and facilitating future quantitative comparisons.

show abstract

Section: Methodsmentioning

confidence: 99%

A Principled Approach to Feature Selection in Models of Sentence Processing

Smith

Vasishth

2020

Cognitive Science

View full text Add to dashboard Cite

show abstract

“…Other work on incorporating syntax into language modeling include Chelba et al (1997) and Pauls and Klein (2012), however none of these approaches considered neural language models, only count-based ones. Levy and Goldberg (2014) and Zhao et al (2014) proposed to train neural word embeddings using skip-grams and CBOWs on dependency parse trees, but did not extend their approach to actual language models such as LBL and RNN and did not evaluate the word embeddings on word completion tasks.…”

Section: Discussionmentioning

confidence: 99%

Dependency Recurrent Neural Language Models for Sentence Completion

Mirowski

Vlachos

2015

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confere

View full text Add to dashboard Cite

Recent work on language modelling has shifted focus from count-based models to neural models. In these works, the words in each sentence are always considered in a left-to-right order. In this paper we show how we can improve the performance of the recurrent neural network (RNN) language model by incorporating the syntactic dependencies of a sentence, which have the effect of bringing relevant contexts closer to the word being predicted. We evaluate our approach on the Microsoft Research Sentence Completion Challenge and show that the dependency RNN proposed improves over the RNN by about 10 points in accuracy. Furthermore, we achieve results comparable with the stateof-the-art models on this task.

show abstract

“…There are some existing word embeddings that take dependency structure into account, e.g., Gamallo (2017), O. Levy and Goldberg (2014a), Padó and Lapata (2007), Rei and Briscoe (2014), Zhao et al (2014). However, these embeddings do not encode the information that we need for use in cognitive models of sentence processing.…”

Section: Methodsmentioning

confidence: 99%

A principled approach to feature selection in models of sentence processing

Smith¹,

Vasishth²

2019

Preprint

View full text Add to dashboard Cite

Among theories of human language comprehension, cue-based memory retrieval has proven to be a useful framework for understanding when and how processing difficulty arises in the resolution of long-distance dependencies. Most previous work in this area has assumed that very general retrieval cues like [+subject] or [+singular] do the work of identifying (and sometimes misidentifying) a retrieval target in order to establish a dependency between words. However, recent work suggests that general, hand-picked retrieval cues like these may not be enough to explain illusions of plausibility (Cunnings & Sturt, 2018), which can arise in sentences like The letter next to the porcelain plate shattered. Capturing such retrieval interference effects requires lexically specific features and retrieval cues, but hand-picking the features is hard to do in a principled way and greatly increases modeler degrees of freedom. To remedy this, we use word embeddings, a well-established method for creating distributed feature representations, for lexical features and retrieval cues. We show that the similarity between the features and the cues (a measure of plausibility) predicts total reading times in Cunnings and Sturt’s eye-tracking data. The features can easily be plugged into existing parsing models (including cue-based retrieval and self-organized parsing), putting very different models on more equal footing and facilitating future quantitative comparisons. In addition to this methodological contribution, our results suggest that, contrary to Cunnings and Sturts’ original conclusions, focused words might be more prominent in memory, making them less susceptible to interference, as predicted by a recent extension to ACT-R (Engelmann, Jäger, & Vasishth, 2019).

show abstract

Learning word embeddings from dependency relations

Cited by 10 publications

References 4 publications

A Principled Approach to Feature Selection in Models of Sentence Processing

A Principled Approach to Feature Selection in Models of Sentence Processing

Dependency Recurrent Neural Language Models for Sentence Completion

A principled approach to feature selection in models of sentence processing

Contact Info

Product

Resources

About