Interesting-phrase mining for ad-hoc text analytics

Abstract

Large text corpora with news, customer mail and reports, or Web 2.0 contributions offer a great potential for enhancing business-intelligence applications. We propose a framework for performing text analytics on such data in a versatile, efficient, and scalable manner. While much of the prior literature has emphasized mining keywords or tags in blogs or social-tagging communities, we emphasize the analysis of interesting phrases. These include named entities, important quotations, market slogans, and other multi-word phrases that are prominent in a dynamically derived ad-hoc subset of the corpus, e.g., being frequent in the subset but relatively infrequent in the overall corpus. We develop preprocessing and indexing methods for phrases, paired with new search techniques for the top-k most interesting phrases in ad-hoc subsets of the corpus. Our framework is evaluated using a large-scale real-world corpus of New York Times news articles.

A Parallel Comparator of Documents

“…It allows efficient parallel processing of data in a functional programming. In [1]a preprocessing and indexing methods for phrases, paired with new search techniques for the top-k most interesting phrases in ad-hoc subsets of the corpus are developed.…”

“…To achieve a more accurate document clustering, a more informative feature word-sentence has been considered in recent research work [1]. While considering three levels (documentssentences-words) based on k-partite graph [2], to represent the data set, we are able to deal with a dependency between all of them.…”

Entity Summarisation with Limited Edge Budget on Undirected and Directed Knowledge Graphs

“…More recent results include text summarization with latent semantic indexing [1], extraction of key phrases [3,23], and summarization of information from multiple documents [22]. Summarisation of scientific papers was studied by Hassan et al [9].…”

Top-k interesting phrase mining in ad-hoc collections using sequence pattern indexing

“…Bedathur et al [6] devise a new algorithm called Forward Indexing to solve this problem of reading the entire set of inverted lists. Instead of storing each phrase's inverted list of document ID, Forward Indexing stores containing phrase ID lists (called forward list) for each document.…”

“…Besides Forward Indexing, [6] proposes another algorithm Prefix-Maximal Indexing to solve the problem, with a very compact index structure in memory of only up to half of the index size of Forward Indexing. This is achieved by storing only prefix-maximal phrases for each document, instead of all the containing phrase IDs.…”