TwitterNEED: A hybrid approach for named entity extraction and disambiguation for tweet

Abstract

Twitter is a rich source of continuously and instantly updated information. Shortness and informality of tweets are challenges for Natural Language Processing tasks. In this paper, we present TwitterNEED, a hybrid approach for Named Entity Extraction and Named Entity Disambiguation for tweets. We believe that disambiguation can help to improve the extraction process. This mimics the way humans understand language and reduces error propagation in the whole system. Our extraction approach aims for high extraction recall first, after which a Support Vector Machine attempts to filter out false positives among the extracted candidates using features derived from the disambiguation phase in addition to other word shape and Knowledge Base features. For Named Entity Disambiguation, we obtain a list of entity candidates from the YAGO Knowledge Base in addition to top-ranked pages from the Google search engine for each extracted mention. We use a Support Vector Machine to rank the candidate pages according to a set of URL and context similarity features. For evaluation, five data sets are used to evaluate the extraction approach, and three of them to evaluate both the disambiguation approach and the combined extraction and disambiguation approach. Experiments show better results compared to our competitors DBpedia Spotlight, Stanford Named Entity Recognition, and the AIDA disambiguation system.

Named Entity Extraction for Knowledge Graphs: A Literature Overview

“…TwitterNEED [24] supports NERD of short and informal tweets. The extraction method focuses on high recall.…”

“…Also, weak NER precision may decrease subsequent NED accuracy. Named Entity Recognition and Disambiguation (NERD) therefore deals with NER and NED jointly [24], [71], [72].…”

A hybrid approach for paraphrase identification based on knowledge-enriched semantic heuristics

“…For instance, if you search for the world's largest corporations such as Microsoft and Apple, you are unlikely to find them in the well-established linguistic knowledge resources such as WordNet. Constantly updated online repositories such as Wikipedia, possess a much higher coverage than WordNet in terms of namedentities (Ponzetto 2010;Habib and van Keulen 2016). Therefore, we use Wikipedia utility for named-entity similarity approximation underpinned with the NGD algorithm.…”

Rule-Based Conditioning of Probabilistic Data

“…In this example, we assume that the initial information extraction produces a probabilistic database with uncertain annotations [9,15]: the type of the first "Paris Hilton" can be either a hotel, person, or fragrance with probabilities 0.5, 0.4, 0.1, respectively. The second "Paris Hilton" analogously.…”

“…For details on the first phase, we refer to [2,3], as well as [7][8][9] for techniques on specific extraction and integration problems (merging semantic duplicates, merging grouping data, and information extraction from natural language text, respectively). This paper focuses on the second phase of this process, namely on the problem of how to incorporate evidence of users in the probabilistically integrated data with the purpose to continuously improve its quality as more evidence is gathered.…”