Classification and evaluation of IoT brokers: A methodology

Abstract

Since the term Internet of Things (IoT) was coined by Kevin Ashton in 1999, a number of middleware platforms have been developed to cope with important challenges such as the integration of different technologies. In this context of heterogeneous technologies, IoT message brokers become critical elements for the proper function of smart systems and wireless sensor networks (WSN) infrastructures. There are several evaluations made on IoT messaging middleware performance. Nevertheless, most of them ignore crucial aspects of the IoT context that also need to be included, such as reliability and other qualitative aspects. Thus, in this article, we propose a methodology for classification and evaluation of IoT brokers to help the scientific community and technology industry on evaluating them according to their interests, without leaving out important aspects for the context of smart environments. Our methodology bases its qualitative evaluations on the ISO/IEC 25000 (SQuaRE) set of standards and its quantitative evaluations on Jain's process for performance evaluation. We developed a case study to illustrate our proposal with 12 different open-source brokers, validating the feasibility of our methodological approach. | INTRODUCTIONThe Internet of Things (IoT) has gained increasing attention during the last decade, with continuous innovations in hardware, software, and connection solutions. Initiatives in different countries such as China, 1 Hong Kong, 2 the United States, 3 and various countries in Europe among others 4 reflect the worldwide interest in the topic. In Brazil, initiatives

BORDER: A Benchmarking Framework for Distributed MQTT Brokers

“…In addittion to pure performance, works focuses also on qualitative metrics, such as portability, usability and maintainability, classifing nine different open-source brokers by the ISO/IEC 25010 normative using a single machine testbed [23]. In a white paper, Scaleagent.com [24] benchmarks single node MQTT brokers including Mosquitto, RabbitMQ, ActiveMQ, JoramMQ to evaluate the client scalability in a multi-publisher scenario, including up to 1000 publishers and one subscriber.…”

Assessing the Quality of Low-Code and Model-Driven Engineering Platforms for Engineering IoT Systems

“…ISO/IEC 25010:2011 standard has been heavily used to assess the quality of complex software and systems. To name a few, [21] relied on it to assess the quality of online health awareness systems, [22], [49] for IoT brokers, and [23] for IoT implementations. Although this quality model's scope is intended general for software and computer systems, it can also be applied to assess larger systems and services [10].…”

An automated evaluation of broker compatibility for the Message Queuing Telemetry Transport protocol

“…A similar objective was followed by Aichernig and Schumi; 33 A general approach has been proposed by Koziolek et al 35 to evaluate MQTT brokers from the point of view of performance, scalability, resilience, security, extensibility, and usability. Furthermore, Bertrand-Martinez et al 30 base their qualitative evaluations on the ISO/IEC 25000 (SQuaRE) set of standards, and its quantitative evaluations on Jain's process for performance evaluation. This includes an analysis regarding interoperability, but in the context of their work, interoperability refers to programming language bindings and compatibility with others IoT protocols.…”

“…The selection of MQTT brokers that we evaluate in this paper extends the list we investigated in our previous work 11 and it corresponds to the list used in a recent study of IoT brokers on classification and selection criteria conducted by Bertrand‐Martinez et al 30 The main rationale for our selection was to include popular and mature brokers that are commonly used in practice. All selected brokers are among the top ranked suggestions in online discussion fora and blog posts.…”

“…Indeed, a general classification and evaluation of IoT brokers in general, and MQTT brokers in particular, can be found in the work by Bertrand‐Martinez et al 30 They propose and apply a quantitative method for performance analysis. Ismail et al 31 as well as Mishra 32 also conduct a performance evaluation and a related comparison of different MQTT brokers in their respective works.…”