A general evaluation measure for document organization tasks

Amigó, Enrique; Gonzalo, Julio; Verdejo, Felisa

doi:10.1145/2484028.2484081

Cited by 76 publications

(83 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Table 5.21 presents a summary of the evaluation measures presented by the task organizers i.e. Reliability and Sensitivity [11] in comparison with the baseline system and best performing systems of the CLEF RepLab 2013 task. From this table it can seen that our system performs comparably to the best reported system.…”

Section: Experimental Results For Filtering Taskmentioning

confidence: 99%

“…These are reliability and sensitivity described in detail by Amigo et al [11]. The property that makes them particularly suitable for the filtering problem is that they are strict with respect to standard measures, i.e., a high value according to reliability and sensitivity implies a high value in all standard measures.…”

Section: Evaluation Measuresmentioning

confidence: 99%

“…For example, consider having only two n-grams in the index; 'a b' with score 'n' and 'b c' with score 'k', so now the extracted single terms would have scores 'a': 'n', 'b':'n+k', and 'c':'k'. Furthermore, we lemmatize all the obtained terms in order to use a conceptual representation of each term (e.g., sciences becomes science) while scoring them 11 .…”

Section: Experiments Typementioning

confidence: 99%

“…Our analysis led to the observation that the datasets with Precision at 20 of less than 0.6 contain noisy terms not related to academic research. This is on account of the dataset of corresponding universities from within our academic web sites' collection being from countries that are not 11 We count the variations just once as a key term in order to avoid redundant terms. Table 7.14: Extracted data from Milano Web site well-known for academic research and, in cases when they are engaged in academic research it is not promoted on the university's web site.…”

Section: Failure Analysismentioning

confidence: 99%

See 3 more Smart Citations

Utilising Wikipedia for Text Mining Applications

Qureshi

2016

SIGIR Forum

View full text Add to dashboard Cite

The process whereby inferences are made from textual data is broadly referred to as text mining. In order to ensure the quality and effectiveness of the derived inferences, several approaches have been proposed for different text mining applications. Among these applications, classifying a piece of text into pre-defined classes through the utilisation of training data falls into supervised approaches while arranging related documents or terms into clusters falls into unsupervised approaches. In both these approaches, processing is undertaken at the level of documents to make sense of text within those documents. Recent research efforts have begun exploring the role of knowledge bases in solving the various problems that arise in the domain of text mining. Of all the knowledge bases, Wikipedia on account of being one of the largest human-curated, online encyclopaedia has proven to be one of the most valuable resources in dealing with various problems in the domain of text mining. However, previous Wikipedia-based research efforts have not taken both Wikipedia categories and Wikipedia articles together as a source of information. This thesis serves as a first step in eliminating this gap and throughout the contributions made in this thesis, we have shown the effectiveness of Wikipedia category-article structure for various text mining tasks. Wikipedia categories are organized in a taxonomical manner serving as semantic tags for Wikipedia articles and this provides a strong abstraction and expressive mode of knowledge representation. In this thesis, we explore the effectiveness of this mode of Wikipedia's expression (i.e., the category-article structure) via its application in the domains of text classification, subjectivity analysis (via a notion of "perspective" in news search), and keyword extraction. First, we show the effectiveness of exploiting Wikipedia for two classification tasks i.e., 1- classifying the tweets1 being relevant/irrelevant to an entity or brand, 2- classifying the tweets into different topical dimensions such as tweets related with workplace, innovation, etc. To do so, we define the notion of relatedness between the text in tweet and the information embedded within the Wikipedia category-article structure. Then, we present an application in the area of news search by using the same notion of relatedness to show more information related to each search result highlighting the amount perspective or subjective bias in each returned result towards a certain opinion, topical drift, etc. Finally, we present a keyword extraction strategy using community detection over the Wikipedia categories to discover related keywords arranged in different communities. The relationship between Wikipedia categories and articles is explored via a textual phrase matching framework whereby the starting point is textual phrases that match Wikipedia articles' titles/redirects. The Wikipedia articles for which a match occurs are then utilised by extraction of their associated categories, and these Wikipedia categories are used to derive various structural measures such as those relating to taxonomical depth and Wikipedia articles they contain. These measures are utilised in our proposed text classification, subjectivity analysis, and keyword extraction framework and the performance is analysed via extensive experimental evaluations. These experimental evaluations undertake comparisons with standard text mining approaches in the literature and our Wikipedia framework based on its category-article structure outperforms the standard text mining techniques.

show abstract

Section: Experimental Results For Filtering Taskmentioning

confidence: 99%

Section: Evaluation Measuresmentioning

confidence: 99%

Section: Experiments Typementioning

confidence: 99%

Section: Failure Analysismentioning

confidence: 99%

See 2 more Smart Citations

Utilising Wikipedia for Text Mining Applications

Qureshi

2016

SIGIR Forum

View full text Add to dashboard Cite

show abstract

“…For evaluation we use two complementary measures, Reliability and Sensitivity as defined by Amigó et al [3]. Let us consider a system output X and a gold standard G, which are both a set of document relationships r(d, d ).…”

Section: Methodsmentioning

confidence: 99%

Entity Tracking in Real-Time Using Sub-topic Detection on Twitter

Panem

Bansal

Gupta

et al. 2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. The velocity, volume and variety with which Twitter generates text is increasing exponentially. It is critical to determine latent sub-topics from such tweet data at any given point of time for providing better topic-wise search results relevant to users' informational needs. The two main challenges in mining subtopics from tweets in real-time are (1) understanding the semantic and the conceptual representation of the tweets, and (2) the ability to determine when a new sub-topic (or cluster) appears in the tweet stream. We address these challenges by proposing two unsupervised clustering approaches. In the first approach, we generate a semantic space representation for each tweet by keyword expansion and keyphrase identification. In the second approach, we transform each tweet into a conceptual space that represents the latent concepts of the tweet. We empirically show that the proposed methods outperform the state-of-the-art methods.

show abstract