Auto-Tuned Publisher in a Pub/Sub System: Design and Performance Evaluation

Abstract

Pub/sub systems form the underlying framework for many distributed applications. It consists of one or more publishers that publish to a broker from which subscriber can retrieve the published content. Many large and small distributed applications including social networking applications use the pub/sub model. In this paper we consider a pub/sub system in which the publisher, the broker, and the subscriber are in different administrative domains. While general pub/sub systems provide reliability of message delivery, good end-to-end latency performance in a multi-domain environment require that the pub/sub system adapt to workload changes and bottlenecks in the different sub-systems. This study is motivated by two applications. First, a pub/sub based Simple Lookup Service (sLS) that is used in perfSONAR to provide information about network performance in R&E networks. Second, the pub/sub system that is used to distribute alerts generated in the data pipeline in the Zwicky Transient Factory (ZTF). We consider a publisher with a multi-threaded architecture that uses batching to coalesce messages over some variable polling period. We propose a control algorithm that auto-tunes the batch-processing parameters namely, the batch size and the polling interval, to the input message load and to any upstream broker-side congestion that minimizes end-to-end latency and maximizes throughput. Using a detailed simulation model, we demonstrate the performance of the control algorithm for different scenarios. We then study the performance using a real-trace obtained from the Simple Lookup Service (sLS). We show that the proposed algorithm quickly adapts to rapid changes in the workload and yields lower mean end-to-end delay performance when compared with the delays in the current deployment.

Topic-Oriented Bucket-Based Fast Multicast Routing in SDN-Like Publish/Subscribe Middleware

“…Two groups of experiments are performed for SDN-like pub/sub topology construction, Namely, Steiner nodes under different node degrees and network sizes. In the first group of experiments, the number of network nodes is 3000; the degree of nodes changes over [1,5], [3,8] and [6,12], respectively; the pub/sub node percent varies among 20%, 30%, 50%, 70%, and 80%. In the second group of experiments, the network size changes over 500, 1000, 1500, 2000 and 3000; the degree of nodes is [1,5] or [3,8]; the pub/sub node percent is 20% or 50%.…”

“…Software Defined Networking (SDN) [6]- [11] is used to solve the difficult problem of Quality of Service (QoS) guarantees of delivering events from publishers to subscribers in IoT. For traditional IP-based pub/sub middlewares [12], [13], event matching with defined filters takes more latency owing to a detour to broker network. However, event matching and forwarding can become more efficient in SDN-based pub/sub middlewares [14], [15] because these processes can be executed directly…”