ECSNeT++ : A simulator for distributed stream processing on edge and cloud environments

Abstract

The objective of Internet of Things (IoT) is ubiquitous computing. As a result many computing enabled, connected devices are deployed in various environments, where these devices keep generating unbounded event streams related to the deployed environment. The common paradigm is to process these event streams at the cloud using the available Distributed Stream Processing (DSP) frameworks. However, with the emergence of Edge Computing, another convenient computing paradigm has been presented for executing such applications. Edge computing introduces the concept of using the underutilised potential of a large number of computing enabled connected devices such as IoT, located outside the cloud. In order to develop optimal strategies to utilise this vast number of potential resources, a realistic test bed is required. However, due to the overwhelming scale and heterogeneity of the edge computing device deployments, the amount of effort and investment required to set up such an environment is high. Therefore, a realistic simulation environment that can accurately predict the behaviour and performance of a large-scale, real deployment is extremely valuable. While the state-of-the-art simulation tools consider different aspects of executing applications on edge or cloud computing environments, we found that no simulator considers all the key characteristics to perform a realistic simulation of the execution of DSP applications on edge and cloud computing environments. To the best of our knowledge, the publicly available simulators lack being verified against real world experimental measurements, i.e. for calibration and to obtain accurate estimates of e.g. latency and power consumption. In this paper, we present our ECSNeT++ simulation toolkit which has been verified using real world experimental measurements for executing DSP applications on edge and cloud computing environments. ECSNeT++ models deployment and processing of DSP applications on edge-cloud environments and is built on top of OMNeT++/INET. By using multiple configurations of two real DSP applications, we show that ECSNeT++ is able to model a real deployment, with proper calibration. We believe that with the public availability of ECSNeT++ as an open source framework, and the verified accuracy of our results, ECSNeT++ can be used effectively for predicting the behaviour and performance of DSP applications running on large scale, heterogeneous edge and cloud computing device deployments.

IFogSim2: An Extended iFogSim Simulator for Mobility, Clustering, and Microservice Management in Edge and Fog Computing Environments

“…However, the simulation components of IoTSim-Osmosis are tightly coupled and constrained in imitating device mobility. ECSNeT++ [19] is another simulator developed by Amarasinghe et al that mimics the execution of distributed stream processing (DSP) applications in Edge/Fog computing environments. It extends OMNeT++/INET and provides multiple configurations for two real DSP applications with calibration and deployment management policy.…”

“…All nodes but the proxy server within a block act as the gateway for the IoT devices. As a means of notations, proxy servers are specified as the tier-1 nodes [10] FogNetSim++ [11] IoTSim-Edge [12] MobFogSim [13] PureEdgeSim [14] STEP-ONE [15] YAFS [16] IoTNetSim [17] SatEdgeSim [18] ECSNeT++ [19] IoTSim-Osmosis [20] However, while parsing the location information of both Edge/Fog nodes and users/IoT devices, the DataParser class creates separate Location objects for each coordinate. The block-wise information of the respective entities (servers and mobile objects) can also be included in a Location object.…”

“…The latency of the current node to each CM will be estimated and stored in mapCMToLatency (lines [17][18]. If the communication latency of CMs is a clustering factor (which is checked by the lf flag), the list of current CMs will be pruned to find the CMs satisfying latency constraint (lines [19][20][21][22][23][24][25]. Finally, the list of CMs list cm f of the current Fog node alongside their latency mappings mapCMToLatency will be returned as the outputs (line 26).…”

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

“…Most of these analytics solutions are focused on using batch processing techniques and technologies like the Hadoop, for the batch processing of large data sets across distributed clusters using the MapReduce programming model [44]. However, considering the advantages of big data analytics and technologies, the application rate in the smart environment and environmental monitoring domain is slow due to underlying fundamental challenges of data and systems heterogeneity [45,46]. While a comprehensive review of existing application scenarios is beyond the scope of this paper, some notable examples are discussed and investigated.…”

PerficientCloudSim: a tool to simulate large-scale computation in heterogeneous clouds

“…Prominent features presented by the Nutshell include: provision of IaaS resources; creating new datacenter architectures; communication protocols; and VM schedulers. Similarly, ECSNeT++ [38] is an extension of the most widely used OMNeT++ toolkit that could simulate distributed stream processing applications on edge and IaaS clouds (hybrid platforms i.e. mobile edge clouds).…”