Investigating Different Syntactic Context Types and Context
            Representations for Learning Word Embeddings

Li, Bofang; Liu, Tao; Zhao, Zhe; Tang, Buzhou; Drozd, Aleksandr; Rogers, Anna; Du, Xiaoyong

doi:10.18653/v1/d17-1257

Cited by 27 publications

(26 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Subjectivity/objectivity classification: Rotten Tomato snippets (Pang and Lee, 2004), using a logistic regression over summed word embeddings (Li et al, 2017a).…”

Section: Discussionmentioning

confidence: 99%

Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for

2019

View full text Add to dashboard Cite

Distributed word vector spaces are considered hard to interpret which hinders the understanding of natural language processing (NLP) models. In this work, we introduce a new method to interpret arbitrary samples from a word vector space. To this end, we train a neural model to conceptualize word vectors, which means that it activates higher order concepts it recognizes in a given vector. Contrary to prior approaches, our model operates in the original vector space and is capable of learning non-linear relations between word vectors and concepts. Furthermore, we show that it produces considerably less entropic concept activation profiles than the popular cosine similarity.

show abstract

“…• Subjectivity/objectivity classification: Rotten Tomato snippets (Pang and Lee, 2004), using a logistic regression over summed word embeddings (Li et al, 2017a).…”

Section: Discussionmentioning

confidence: 99%

Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for

2019

View full text Add to dashboard Cite

show abstract

“…We follow the evaluation protocol for sequential labeling used by Kiros et al (2015) and Li et al (2017), and use logistic regression classifier 13 as the model for POS tagging. When predicting the tag for the i-th word w i in a sentence, the input to the classifier is the concatenation of the vectors w i−2 , w i−1 , w i , w i+1 , w i+2 for the word itself and the words in its context.…”

Section: Pos Tagging Modelmentioning

confidence: 99%

PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding

Zhao

Zhong

Zhang

et al. 2020

Findings of the Association for Computational Linguistics: EMNLP 2020

View full text Add to dashboard Cite

We look into the task of generalizing word embeddings: given a set of pre-trained word vectors over a finite vocabulary, the goal is to predict embedding vectors for out-of-vocabulary words, without extra contextual information. We rely solely on the spellings of words and propose a model, along with an efficient algorithm, that simultaneously models subword segmentation and computes subword-based compositional word embedding. We call the model probabilistic bag-of-subwords (PBoS), as it applies bag-of-subwords for all possible segmentations based on their likelihood. Inspections and affix prediction experiment show that PBoS is able to produce meaningful subword segmentations and subword rankings without any source of explicit morphological knowledge. Word similarity and POS tagging experiments show clear advantages of PBoS over previous subword-level models in the quality of generated word embeddings across languages.

show abstract

“…For example, using California as a negative sample for Oregon helps the model to learn that the pattern "X is located in Y" fits the pair (Portland, Oregon), but not the pair (Portland, California). Similar adversarial constraints were used in knowledge base completion (Toutanova et al, 2015) and word embeddings (Li et al, 2017). 4…”

Section: Objectivementioning

confidence: 99%

pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference

Joshi¹,

Choi²,

Levy³

et al. 2019

Proceedings of the 2019 Conference of the North

View full text Add to dashboard Cite

Reasoning about implied relationships (e.g. paraphrastic, common sense, encyclopedic) between pairs of words is crucial for many cross-sentence inference problems. This paper proposes new methods for learning and using embeddings of word pairs that implicitly represent background knowledge about such relationships. Our pairwise embeddings are computed as a compositional function on word representations, which is learned by maximizing the pointwise mutual information (PMI) with the contexts in which the two words cooccur. We add these representations to the cross-sentence attention layer of existing inference models (e.g. BiDAF for QA, ESIM for NLI), instead of extending or replacing existing word embeddings. Experiments show a gain of 2.7% on the recently released SQuAD 2.0 and 1.3% on MultiNLI. Our representations also aid in better generalization with gains of around 6-7% on adversarial SQuAD datasets, and 8.8% on the adversarial entailment test set by Glockner et al. (2018).

show abstract

Investigating Different Syntactic Context Types and Context Representations for Learning Word Embeddings

Cited by 27 publications

References 36 publications

Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for

Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for

PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding

pair2vec: Compositional Word-Pair Embeddings for Cross-Sentence Inference

Contact Info

Product

Resources

About