Revisiting Correlations between Intrinsic and Extrinsic Evaluations of Word Embeddings

Qiu, Yuanyuan; Li, Wanqing; Shen, Li; Jiang, Yingdi; Hu, Renfen; Yang, Leiku

doi:10.1007/978-3-030-01716-3_18

Cited by 67 publications

(32 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although correlation between intrinsic and extrinsic evaluators was studied before [16], [17], this topic is never thoroughly and seriously treated. For example, producing models by changing the window size only does not happen often in real world applications, and the conclusion drawn in [16] might be biased.…”

Section: Introductionmentioning

confidence: 99%

Evaluating word embedding models: methods and experimental results

Wang

Wang²,

Chen

et al. 2019

SIP

250

107

View full text Add to dashboard Cite

Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evalutors.

show abstract

Section: Introductionmentioning

confidence: 99%

Evaluating word embedding models: methods and experimental results

Wang

Wang²,

Chen

et al. 2019

SIP

250

107

View full text Add to dashboard Cite

show abstract

“…Use of Pre-trained Model Weights. We initialize the word embeddings using pre-trained word2vec model weights for common words on a large mixed corpus [22]. The word embeddings have 300 dimensions.…”

Section: Test Steps Preprocessing Word Embedding Trainingmentioning

confidence: 99%

Clustering test steps in natural language toward automating test automation

Zhang

et al. 2020

Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softw

View full text Add to dashboard Cite

“…Extrinsic evaluations for word embeddings might consider how well different word vectors help models for tasks like sentiment analysis (Petrolito, 2018;Mishev et al, 2019), machine translation (Wang et al, 2019b), or named entity recognition (Wu et al, 2015;Nayak et al, 2016). 4 Although recent work suggest that some intrinsic evaluations for word vectors do indeed correlate with extrinsic evaluations (Qiu et al, 2018;Thawani et al, 2019).…”

Section: Intrinsic Vs Extrinsic Evaluationsmentioning

confidence: 99%

A survey on Recognizing Textual Entailment as an NLP Evaluation

Poliak¹

2020

Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems

View full text Add to dashboard Cite

Recognizing Textual Entailment (RTE) was proposed as a unified evaluation framework to compare semantic understanding of different NLP systems. In this survey paper, we provide an overview of different approaches for evaluating and understanding the reasoning capabilities of NLP systems. We then focus our discussion on RTE by highlighting prominent RTE datasets as well as advances in RTE dataset that focus on specific linguistic phenomena that can be used to evaluate NLP systems on a fine-grained level. We conclude by arguing that when evaluating NLP systems, the community should utilize newly introduced RTE datasets that focus on specific linguistic phenomena.

show abstract

Revisiting Correlations between Intrinsic and Extrinsic Evaluations of Word Embeddings

Cited by 67 publications

References 20 publications

Evaluating word embedding models: methods and experimental results

Evaluating word embedding models: methods and experimental results

Clustering test steps in natural language toward automating test automation

A survey on Recognizing Textual Entailment as an NLP Evaluation

Contact Info

Product

Resources

About