Diversity-Aware Top-k Publish/Subscribe for Text Stream

Abstract

Massive amount of text data are being generated by a huge number of web users at an unprecedented scale. These data cover a wide range of topics. Users are interested in receiving a few up-todate representative documents (e.g., tweets) that can provide them with a wide coverage of different aspects of their query topics. To address the problem, we consider the Diversity-Aware Topk Subscription (DAS) query. Given a DAS query, we continuously maintain an up-to-date result set that contains k most recently returned documents over a text stream for the query. The DAS query takes into account text relevance, document recency, and result diversity. We propose a novel solution to efficiently processing a large number of DAS queries over a stream of documents. We demonstrate the efficiency of our approach on realworld dataset and the experimental results show that our solution is able to achieve a reduction of the processing time by 60-75% compared with two baselines. We also study the effectiveness of the DAS query.

Learning to Un-Rank

“…For example, for the models with the Euclidean distance or cosine similarity, many ltering methods have been suggested based on di erent geometric properties [20,26,30]. However, for the models with textual queries only a few works exist, which cannot be fully applied to our problem as either the se ing is monochromatic [21], the indexing tree of the queries should be built which is not scalable to very high dimensions [39], the storage of k nearest neighbors for all the queries is required with a xed k [9], or only the conjunctive queries are considered [1]. In the remainder of this section, we present our algorithm for the dynamic generation of the RkNNs for the textual data.…”

“…Finding the set of reverse k nearest neighbors (a.k.a. the in uence set) of a point has been studied in various contexts such as matching the user preferences to a given product [33] or the assignment of a publication to a set of subscriptions [1,9]. e se ing of such problems falls into either of these categories: monochromatic or bichromatic.…”

Search Result Diversification in Short Text Streams

“…Diversifying search results given a query in short text streams is of importance and has many applications. For instance, top-k publish/subscribe systems for tweets [7,39] are required to return to a subscriber the top-k recent tweets that are relevant and diversified given a subscribed keyword. The problem of diversifying search results in long text streams has previously been investigated by Refs.…”

“…The problem of diversifying search results in long text streams has previously been investigated by Refs. [7,33]. Both models penalize redundancy in a ranked list of documents in a stream, where redundancy is directly measured as a sum of pairwise similarities between any two documents.…”

“…[7,33]; and (3) algorithms that combine existing dynamic topic models with effective diversification retrieval models that have not been designed to work with text streams. Our approach outperforms state-of-the-art diversification methods in terms of a range of diversification metrics.…”