From static to dynamic word representations: a survey

Wang, Yuxuan; Hou, Yutai; Che, Wanxiang; Liu, Ting

doi:10.1007/s13042-020-01069-8

Cited by 61 publications

(28 citation statements)

References 68 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Neural network language models introduced the idea of deep learning into language modeling by learning a distributed representation of words. These distributed word representations, trained on massive amounts of unannotated textual data, have been proved to provide good lower dimension feature representations in a wide range of NLP tasks (Wang et al, 2020). The Continuous Bag-of-Words and Skip-gram models proposed to reduce the computational complexity were considered as a milestone in the development of the so-called word embeddings (Mikolov et al, 2013), followed by the Global Vector (GloVe) (Pennington et al, 2014) and the fastText (Bojanowski et al, 2016) models.…”

Section: Introductionmentioning

confidence: 99%

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Schneider¹,

Souza²,

Knafou³

et al. 2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

With the growing number of electronic health record data, clinical NLP tasks have become increasingly relevant to unlock valuable information from unstructured clinical text. Although the performance of downstream NLP tasks, such as named-entity recognition (NER), in English corpus has recently improved by contextualised language models, less research is available for clinical texts in low resource languages. Our goal is to assess a deep contextual embedding model for Portuguese, so called BioBERTpt, to support clinical and biomedical NER. We transfer learned information encoded in a multilingual-BERT model to a corpora of clinical narratives and biomedical-scientific papers in Brazilian Portuguese. To evaluate the performance of BioBERTpt, we ran NER experiments on two annotated corpora containing clinical narratives and compared the results with existing BERT models. Our in-domain model outperformed the baseline model in F1-score by 2.72%, achieving higher performance in 11 out of 13 assessed entities. We demonstrate that enriching contextual embedding models with domain literature can play an important role in improving performance for specific NLP tasks. The transfer learning process enhanced the Portuguese biomedical NER model by reducing the necessity of labeled data and the demand for retraining a whole new model.

show abstract

Section: Introductionmentioning

confidence: 99%

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Schneider¹,

Souza²,

Knafou³

et al. 2020

Proceedings of the 3rd Clinical Natural Language Processing Workshop

View full text Add to dashboard Cite

show abstract

“…Our proposed research methods based on textual alignment and word embeddings can naturally be extended to other languages, since alignment methods have a strong foundation in a multilingual context (Tiedemann 2011). Furthermore word embeddings for other languages (Wang et al 2020) can be utilized for the detection of paraphrase types for multiple languages.…”

Section: Discussionmentioning

confidence: 99%

Paraphrase type identification for plagiarism detection using contexts and word embeddings

Alvi

Stevenson

Clough

2021

Int J Educ Technol High Educ

View full text Add to dashboard Cite

Paraphrase types have been proposed by researchers as the paraphrasing mechanisms underlying acts of plagiarism. Synonymous substitution, word reordering and insertion/deletion have been identified as some of the common paraphrasing strategies used by plagiarists. However, similarity reports generated by most plagiarism detection systems provide a similarity score and produce matching sections of text with their possible sources. In this research we propose methods to identify two important paraphrase types – synonymous substitution and word reordering in paraphrased, plagiarised sentence pairs. We propose a three staged approach that uses context matching and pretrained word embeddings for identifying synonymous substitution and word reordering. Our proposed approach indicates that the use of Smith Waterman Algorithm for Plagiarism Detection and ConceptNet Numberbatch pretrained word embeddings produces the best performance in terms of $$\hbox {F}_1$$ F 1 scores. This research can be used to complement similarity reports generated by currently available plagiarism detection systems by incorporating methods to identify paraphrase types for plagiarism detection.

show abstract

“…However, for complex tasks, which require finer discrimination between classes, their main drawback is that Figure 2 The bag of polarities representation; in it each document from the collection is represented by a dual vector that captures the occurrence of words in both positive and negative contexts they do not capture information of the contexts of the words. To address this limitation, different types of contextualized word embeddings (e.g., ELMo, Flair, and BERT) have been recently proposed [45,46]. Their idea is to add syntactic and semantic information to the words' representations aiming to dynamically capture their meaning.…”

Section: Bag Of Polarities: a Dual Word-sentiment Representation For Depression Detectionmentioning

confidence: 99%

A profile-based sentiment-aware approach for depression detection in social media

et al. 2021

View full text Add to dashboard Cite

Depression is a severe mental health problem. Due to its relevance, the development of computational tools for its detection has attracted increasing attention in recent years. In this context, several research works have addressed the problem using word-based approaches (e.g., a bag of words). This type of representation has shown to be useful, indicating that words act as linguistic markers of depression. However, we believe that in addition to words, their contexts contain implicitly valuable information that could be inferred and exploited to enhance the detection of signs of depression. Specifically, we explore the use of user’s characteristics and the expressed sentiments in the messages as context insights. The main idea is that the words’ discriminative value depends on the characteristics of the person who is writing and on the polarity of the messages where they occur. Hence, this paper introduces a new approach based on specializing the framework of classification to profiles of users (e.g., males or women) and considering the sentiments expressed in the messages through a new text representation that captures their polarity (e.g., positive or negative). The proposed approach was evaluated on benchmark datasets from social media; the results achieved are encouraging, since they outperform those of state-of-the-art corresponding to computationally more expensive methods.

show abstract

From static to dynamic word representations: a survey

Cited by 61 publications

References 68 publications

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

BioBERTpt - A Portuguese Neural Language Model for Clinical Named Entity Recognition

Paraphrase type identification for plagiarism detection using contexts and word embeddings

A profile-based sentiment-aware approach for depression detection in social media

Contact Info

Product

Resources

About