Building a large-scale corpus for evaluating event detection on twitter

Abstract

Despite the popularity of Twitter for research, there are very few publicly available corpora, and those which are available are either too small or unsuitable for tasks such as event detection. This is partially due to a number of issues associated with the creation of Twitter corpora, including restrictions on the distribution of the tweets and the difficultly of creating relevance judgements at such a large scale. The difficulty of creating relevance judgements for the task of event detection is further hampered by ambiguity in the definition of event. In this paper, we propose a methodology for the creation of an event detection corpus. Specifically, we first create a new corpus that covers a period of 4 weeks and contains over 120 million tweets, which we make available for research. We then propose a definition of event which fits the characteristics of Twitter, and using this definition, we generate a set of relevance judgements aimed specifically at the task of event detection. To do so, we make use of existing state-of-the-art event detection approaches and Wikipedia to generate a set of candidate events with associated tweets. We then use crowdsourcing to gather relevance judgements, and discuss the quality of results, including how we ensured integrity and prevented spam. As a result of this process, along with our Twitter corpus, we release relevance judgements containing over 150,000 tweets, covering more than 500 events, which can be used for the evaluation of event detection approaches.

Editorial: Survey and Experimental Analysis of Event Detection Techniques for Twitter

“…McMinn et al [55] propose a methodology for creating a corpus to evaluate event detection methods. They used two existing state-of-the art event detection approaches [28,54] together with Wikipedia to create a set of candidate events together with a list of associated tweets.…”

“…For example, the organizers of the 2014 SNOW challenge [12] could only crawl 1 106 712 of the original 3 630 816 tweets of the above-mentioned 2012 US Presidential Election data set [37]. In order to assess how useable these collections of tweet identifiers are, we attempted to download the corpus of McMinn et al [55]. The standard restriction of crawling tweets with the Twitter API 4 is set to 180 queries per 15 minute window.…”

Challenges of event detection from social media streams

“…On the other hand, Aggarwal, et al [3] states that a news event is "something that happens at a specific time and place, but it is also an object of interest to the news media". Similarly, McMinn et al [9] define an event as something significant happening in a specific time and place beside it lead to discussions by the news media. This event might be a political event, natural disaster, terror attack or a protest, etc.…”

“…Thus, the methods and techniques used for these kinds of events should be evaluated in terms of how fast they can be identified rather than just evaluating based on precision and recall measurements [1]. Unfortunately, there are very few ED evaluation datasets [9]. The TDT5 dataset has been utilized by many studies [43], to evaluate precision.…”

Graph-based Event Extraction from Twitter

“…We test our approach on two gold standard corpora: the First Story Detection (FSD) corpus (Petrović et al, 2012) and the EVENT2012 corpus (McMinn et al, 2013).…”