Deep contextualized word representations

Peters, Matthew E.; Neumann, Mark E; Iyyer, Mohit; Gardner, Matt; Clark, Christopher M.; Lee, Kenton; Zettlemoyer, Luke

doi:10.48550/arxiv.1802.05365

Cited by 460 publications

(566 citation statements)

References 0 publications

Supporting

Mentioning

559

Contrasting

Unclassified

Order By: Relevance

“…It origins from pre-training contextual representations, e.g. ELMo [27], ULM-FiT [14], OpenAI [28], etc. The BERT coverts an input sequence (x 1 , ..., x n ) to a sequence of vector representations z = (z 1 , ..., z n ) [35].…”

Section: Bertmentioning

confidence: 99%

SQRQuerier: A Visual Querying Framework for Cross-national Survey Data Recycling

Tu¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Section: Bertmentioning

confidence: 99%

SQRQuerier: A Visual Querying Framework for Cross-national Survey Data Recycling

Tu¹,

Li²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

“…GloVe presents a regression-based model to predict the conditional probability of a word appearing given another word. Context-aware word embeddings, such as such as Embeddings from Language Model (ELMo) [8] and Bidirectional Encoder Representations from Transformers (BERT) [43], were more recently proposed to generate word representations that better consider the context of the sentence. However, all these embeddings are usually trained on common text corpora [7].…”

Section: Textual Word Embeddingsmentioning

confidence: 99%

“…These classical approaches are linear language modeling approaches and often fail to model the true contextual meaning of text corpora. In contrast, Word2Vec [6], GloVe [7], and ELMO [8] are some of the more modern techniques of contextualizing meanings of text corpora, which incorporate neural networks for non-linear language modelling. However, these models are often trained on datasets derived from Twitter, Wikipedia, or general pieces of text and are therefore not entirely suitable for the analysis of scientific publications due to the existence of domain-specific words in these corpora.…”

Section: Introductionmentioning

confidence: 99%

Analyzing Scientific Publications using Domain-Specific Word Embedding and Topic Modelling

Singhal¹,

Liu²,

Blessing³

et al. 2021

Preprint

View full text Add to dashboard Cite

The scientific world is changing at a rapid pace, with new technology being developed and new trends being set at an increasing frequency. This paper presents a framework for conducting scientific analyses of academic publications, which is crucial to monitor research trends and identify potential innovations. This framework adopts and combines various techniques of Natural Language Processing, such as word embedding and topic modelling. Word embedding is used to capture semantic meanings of domain-specific words. We propose two novel scientific publication embedding, i.e., PUB-G and PUB-W, which are capable of learning semantic meanings of general as well as domain-specific words in various research fields. Thereafter, topic modelling is used to identify clusters of research topics within these larger research fields. We curated a publication dataset consisting of two conferences and two journals from 1995 to 2020 from two research domains. Experimental results show that our PUB-G and PUB-W embeddings are superior in comparison to other baseline embeddings by a margin of ∼0.18-1.03 based on topic coherence.

show abstract

“…Therefore the same word may have different embedding representations depending on the context the word appear in the request to the chatbot. Recent developments of context-dependent embeddings [10,25] show that systems based on such representations achieve good results in different text classification tasks [26], including fake news detection [27], detection of user satisfaction in chatbot systems and call centers [28,29], document classification [30], and health-care applications [29].…”

Section: Context-dependent: Bertmentioning

confidence: 99%

One System to Rule them All: a Universal Intent Recognition System for Customer Service Chatbots

Vásquez-Correa¹,

Guerrero-Sierra²,

Luis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Customer service chatbots are conversational systems designed to provide information to customers about products/services offered by different companies.Particularly, intent recognition is one of the core components in the natural language understating capabilities of a chatbot system. Among the different intents that a chatbot is trained to recognize, there is a set of them that is universal to any customer service chatbot. Universal intents may include salutation, switch the conversation to a human agent, farewells, among others. A system to recognize those universal intents will be very helpful to optimize the training process of specific customer service chatbots. We propose the development of a universal intent recognition system, which is trained to recognize a selected group of 11 intents that are common in 28 different chatbots. The proposed system is trained considering state-of-the-art word-embedding models such as word2vec and BERT, and deep classifiers based on convolutional and recurrent neural networks. The proposed model is able to discriminate between those universal intents with a balanced accuracy up to 80.4%. In addition, the proposed system is equally accurate to recognize intents expressed both in short and long text requests. At the same time, misclassification errors often occurs between intents with very similar semantic fields such as farewells and positive

show abstract

Deep contextualized word representations

Cited by 460 publications

References 0 publications

SQRQuerier: A Visual Querying Framework for Cross-national Survey Data Recycling

SQRQuerier: A Visual Querying Framework for Cross-national Survey Data Recycling

Analyzing Scientific Publications using Domain-Specific Word Embedding and Topic Modelling

One System to Rule them All: a Universal Intent Recognition System for Customer Service Chatbots

Contact Info

Product

Resources

About