Scalable clustering of news search results

Abstract

In this paper, we present a system for clustering the search results of a news search engine. The news search interface includes the relevant news articles to a given query organized in terms of related news stories. Here each cluster corresponds to a news story and the news articles are clustered into stories. We present a system that clusters the search results of a news search system in a fast and scalable manner. The clustering system is organized into three components including offline clustering, incremental clustering and realtime clustering. We propose novel techniques for clustering the search results in realtime. The experimental results with large collections of news documents reveal that our system is both scalable and also achieves good accuracy in clustering the news search results.

Multiple developing news stories identified and tracked by social insects and visualized using the new galactic streams and concurrent streams metaphors

“…A broad array of methods rely on clustering algorithms to detect topics in documents. Vadrevu et al [2] propose an incremental version of k-means clustering algorithm, suited for dynamic sets of documents by incrementally reviewing already detected topics after a given number of new documents is introduced into the documents set. Forsati et al [3] propose a clustering based approach to content analysis utilized in a recommender system.…”

Thirty years of software problems in the news

“…While there are many news clustering techniques, all were inappropriate for our analysis: kmeans clustering on a word vector space and Latent Dirichlet Allocation, used both in automated news clustering approaches [9,6,30] and studies of blogs [18], require a specific number of topics to be selected a priori, but we had no basis for choosing such a number. These techniques also ignore time: stories in the 1980s should not be clustered with problems from the 2000s just because they share a topic.…”

Social insect inspired approach for identification and dynamic tracking of news stories on the Web

“…A considerable research in topic detection and tracking has been focusing on news stories and there are multiple methods of detecting a news story, that yield different representations of topics. Vadrevu et al [1] propose an incremental version of k-means clustering algorithm, suited for dynamic sets of documents by incrementally reviewing already detected topics after a given number of new documents is introduced into the documents set.…”