Word frequencies: A comparison of Pareto type distributions

Wiegand, Martin; Nadarajah, Saralees; Si, Yuancheng

doi:10.1016/j.physleta.2017.12.056

Cited by 2 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…0.64 million unique tokens). The word occurrences in this training set follow a Pareto-like distribution as described by Wiegand et al [32]. This distribution indicates that our original data has similar properties as standard English texts.…”

Section: Datasetsupporting

confidence: 58%

Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF

Meijer,

Truong,

Karimi

2021

Preprint

View full text Add to dashboard Cite

Over the last few years, neural network derived word embeddings became popular in the natural language processing literature. Studies conducted have mostly focused on the quality and application of word embeddings trained on public available corpuses such as Wikipedia or other news and social media sources. However, these studies are limited to generic text and thus lack technical and scientific nuances such as domain specific vocabulary, abbreviations, or scientific formulas which are commonly used in academic context. This research focuses on the performance of word embeddings applied to a large scale academic corpus. More specifically, we compare quality and efficiency of trained word embeddings to TFIDF representations in modeling content of scientific articles. We use a word2vec skip-gram model trained on titles and abstracts of about 70 million scientific articles. Furthermore, we have developed a benchmark to evaluate content models in a scientific context. The benchmark is based on a categorization task that matches articles to journals for about 1.3 million articles published in 2017. Our results show that content models based on word embeddings are better for titles (short text) while TFIDF works better for abstracts (longer text). However, the slight improvement of TFIDF for larger text comes at the expense of 3.7 times more memory requirement as well as up to 184 times higher computation times which may make it inefficient for online applications. In addition, we have created a 2-dimensional visualization of the journals modeled via embeddings to qualitatively inspect embedding model. This graph shows useful insights and can be used to find competitive journals or gaps to propose new journals.

show abstract

Section: Datasetsupporting

confidence: 58%