Annotating the Tweebank Corpus on Named Entity Recognition and Building NLP Models for Social Media Analysis

Abstract

Social media data such as Twitter messages ("tweets") pose a particular challenge to NLP systems because of their short, noisy, and colloquial nature. Tasks such as Named Entity Recognition (NER) and syntactic parsing require highly domain-matched training data for good performance. While there are some publicly available annotated datasets of tweets, they are all purpose-built for solving one task at a time. As yet there is no complete training corpus for both syntactic analysis (e.g., part of speech tagging, dependency parsing) and NER of tweets. In this study, we aim to create Tweebank-NER, an NER corpus based on Tweebank V2 (TB2), and we use these datasets to train state-of-the-art NLP models. We first annotate named entities in TB2 using Amazon Mechanical Turk and measure the quality of our annotations. We train a Stanza NER model on the new benchmark, achieving competitive performance against other non-transformer NER systems. Finally, we train other Twitter NLP models (a tokenizer, lemmatizer, part of speech tagger, and dependency parser) on TB2 based on Stanza, and achieve state-of-the-art or competitive performance on these tasks. We release the dataset and make the models available to use in an "off-the-shelf" manner for future Tweet NLP research. Our source code, data, and pre-trained models are available at: https://github.com/social-machines/TweebankNLP.

Using Twitter data to understand public perceptions of approved versus off-label use for COVID-19-related medications

“…For NER word clouds, Stanza's 4-class (person, organization, location, and miscellaneous) tweet NER model was used to extract named entities. 34,35 The extracted NERs were plotted per time period per drug to show the change in people's foci over time.…”

An exploratory design science research on troll factories

“…Perhaps the best solutions currently available for this task are the ArkTweet tagger [72] and the finetuned BERTweet [113]. The accuracy of these models evaluated in Tweebank2 is 94.6% [114] and 95.3% [113] respectively.…”

“…Perhaps the best solutions currently available for this task are the ArkTweet tagger [72] and the finetuned BERTweet [113]. The accuracy of these models evaluated in Tweebank2 is 94.6% [114] and 95.3% [113] respectively. However, since ArkTweet introduces Twitter-specific tags (see table 2), the output is more informative from a human perspective.…”

Structural Vetting of Academic Proposals

“…Researchers have used various forms of text processing technique to automatically extract and analyse documents such as business documents [20], clinical notes [21], legal documents [2], [22] and so on. NLP techniques have been used to perform text information extraction [23], named entity recognition [24], language to SQL translator [25], [26], summarisation [27], classification and examination of other textual contents such as CVs [28], invoices [20] and social media texts [29]. These NLP techniques and others have been largely used around the text content of a document, and sometimes shorttext based documents.…”