Scalable ranked publish/subscribe

Abstract

Publish/subscribe (pub/sub) systems are designed to efficiently match incoming events (e.g., stock quotes) against a set of subscriptions (e.g., trader profiles specifying quotes of interest). However, current pub/sub systems only support a simple binary notion of matching: an event either matches a subscription or it does not; for instance, a stock quote will either match or not match a trader profile. In this paper, we argue that this simple notion of matching is inadequate for many applications where only the "best" matching subscriptions are of interest. For instance, in targeted Web advertising, an incoming user ("event") may match several different advertiser-specified user profiles ("subscriptions"), but given the limited advertising real-estate, we want to quickly discover the best (e.g., most relevant) ads to display.To address this need, we initiate a study of ranked pub/sub systems. We focus on the case where subscriptions correspond to interval ranges (e.g, age in [25,35] and salary > $50, 000), and events are points that match all the intervals that they stab (e.g., age=28, salary = $65,000). In addition, each interval has a score and our goal is to quickly recover the top-scoring matching subscriptions. Unfortunately, adapting existing index structures to solve this problem results in either an unacceptable space overhead or a significant performance degradation. We thus propose two novel index structures that are both compact and efficient. Our experimental evaluation shows that the proposed structures provide a scalable basis for designing ranked pub/sub systems.

Top-k/w publish/subscribe: A publish/subscribe model for continuous top-k processing over data streams

“…Our initial idea to design a stateful matching model for publish/subscribe systems which ranks publications with respect to a subscription is presented in [10], where we also introduce an algorithm for probabilistic computation of top-k/w queries. This idea was developed in parallel with the following two papers [11,12] that also introduce ranking functions into publish/subscribe systems, but do not use sliding windows to limit the temporal scope of subscription processing. Compared to [10], this paper introduces an extended top-k/w model for distributed publish/subscribe systems, discusses the scoring functions supported by the model, and identifies the routing algorithms that are adequate for top-k/w publish/subscribe.…”

“…The idea to rank publications in publish/subscribe systems according to a subscription has been developed and published in parallel in our paper [10] and the following two papers [11,12]. However, the other authors did not recognize the importance of sliding-window in publish/subscribe setting.…”

Towards an extensible efficient event processing kernel

“…In user-centric processing applications, there are computational advertising [28,10], online job sites [17,28], and location-based services for emerging applications in the co-spaces [1,19]; common to all are patterns and specifications (e.g., advertising campaigns, job profiles, service descriptions) modeled as Boolean expressions, XPath expressions, or SQL queries and incoming user information (e.g., user profiles and preferences) modeled as events using attribute-value pairs, XML document, or relational tuples. In the real-time analysis domain, there are (complex) event processing [11,2,6,7,5], XML filtering [3,18,15], intrusion detection [27], and computational finance [23]; again, common among these applications are predefined set of patterns (e.g., investment strategies and attack specifications) modeled as subscriptions and streams of incoming data (e.g., XML documents, data packets, stock feeds) modeled as events.…”

“…Unique to user-centric processing and personalization are strict requirements to determine only the most relevant content (e.g., ads) that is both user-consumable and suitable for the often limited screen real estate of client devices [17,28,10]. In addition, the usercentric processing demands scaling to millions of patterns and specifications (e.g., advertising campaigns) for supporting large-scale enterprise-level user-services, processing latency constraints in the subsecond range for meeting an acceptable service-level agreement, and improve expression expressiveness for capturing interesting patterns and desired preferences.…”

“…First and foremost, the index structure designed for matching Boolean expressions must support top-k matching to quickly retrieve only the most relevant expressions, which is not addressed by prior Boolean expression matching approaches (e.g., [2,7].) Furthermore, the relevance computation must be based on a generic and preference-aware scoring function, which is not addressed by prior approaches (e.g., [17].) Moreover, the top-k model must cope with much higher dimensionality (e.g., a dimension can represent an attribute in the user profile such as age, gender, and location) that is beyond the scope of dominant database top-k techniques (e.g., [14]) and skyline query processing (e.g., [4].)…”