Benchmarking Publish/Subscribe-Based Messaging Systems

Abstract

Abstract. Publish/subscribe-based messaging systems are used increasingly often as a communication mechanism in data-oriented web applications. Such applications often pose serious performance and scalability challenges. To address these challenges, it is important that systems are tested using benchmarks to evaluate their performance and scalability before they are put into production. In this paper, we present jms2009-PS, a new benchmark for publish/subscribe-based messaging systems built on top of the SPECjms2007 standard workload. We introduce the benchmark and discuss its configuration parameters showing how the workload can be customized to evaluate various aspects of publish/subscribe communication. Finally, we present a case study illustrating how the benchmark can be used for performance analysis of messaging servers.

Impact analysis for distributed event-based systems

“…Column Application Type gives a short description of the application's domain; SLOC shows the source lines of code of each application; Components shows how many component objects each application has; Message Types totals the different message types in each application; and MOM Platform names the middleware that provides the connector services to the application. All applications have been developed independently of Helios, in Java, and their architectures have been described in prior publications [21,23,24,30,29,33]. Initially, each application utilized subject-based filtering as its main filtering mechanism.…”

“…The evaluated systems include an architecture-based modeling and analysis environment [21], an arcade game [21], an emergency response system [33], a stock ticker notification system [23] and the jms2009-PS [30] benchmark that is based on the official SPECJms2007 [29] benchmark for evaluating the performance of enterprise MOM servers. For each system we are able to demonstrate that Helios enables impact analysis…”

A Logistics Workload for Event Notification Middleware

“…The setup of consumer, producer and subscriptions is static and does not change at runtime. A benchmark for publish/subscribe systems built on top of the SPECjms2007 workload is jms2009-PS [12]. However, the implemented topology is static and subscriptions do not change at runtime.…”

Modeling and optimizing content-based publish/subscribe systems

“…Thus, for the further development there is strong need to get (possibly anonymized) real data which also include the spatial distribution of events and subscriptions so that the publish/subscribe research community can increase the significance of the achieved results. Accepted standard benchmarks such as SPECjms [88] and jms2009-PS [87] are important first steps in this direction.…”