Survey of Real-time Processing Technologies of IoT Data Streams

Yasumoto, Keiichi; Yamaguchi, Hirozumi; Shigeno, Hiroshi

doi:10.2197/ipsjjip.24.195

Cited by 107 publications

(56 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…By changing the granularity of the operator graph, Nephele-Streaming enables pipeline parallelism in the operators. Nakamura et al present in [107] an SP middleware that is based on their earlier system called "Information Flow of ings" (IFoT) [151]. ey designed a layer fog-computing architecture that uses Raspberry Pi-nodes.…”

Section: Parallelization For General Streammentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

Stream Processing (SP) has evolved as the leading paradigm to process and gain value from the high volume of streaming data produced e.g. in the domain of the Internet of ings. An SP system is a middleware that deploys a network of operators between data sources, such as sensors, and the consuming applications. SP systems typically face intense and highly dynamic data streams. Parallelization and elasticity enables SP systems to process these streams with continuously high quality of service. e current research landscape provides a broad spectrum of methods for parallelization and elasticity in SP. Each method makes speci c assumptions and focuses on particular aspects of the problem. However, the literature lacks a comprehensive overview and categorization of the state of the art in SP parallelization and elasticity, which is necessary to consolidate the state of the research and to plan future research directions on this basis. erefore, in this survey, we study the literature and develop a classi cation of current methods for both parallelization and elasticity in SP systems. or even di erent processing nodes in a shared-nothing cloud-based infrastructure. Frequent state synchronization must not hamper parallel processing, while the processing results have to remain consistent. Research proposes di erent approaches for parallel, stateless and stateful SP. ey di er in assumptions about the operator functions and state externalization mechanisms an SP system supports.is led to the development of a broad range of parallelization approaches tackling di erent problem cases.Second, how to continuously adapt the level of parallelization when the conditions of the SP operators, e.g. the workload or resources available, change at runtime. On the one hand, an SP system always needs enough resources to process the input data streams with a satisfying quality of service (QoS), e.g. latency or throughput. On the other hand, continuous provisioning of computing resources for peak workloads wastes resources at o -peak hours. us, an elastic SP system scales its resources according to the current need. Cloud computing provides on-demand resources to realize such elasticity [9]. e pay-as-you-go business model of cloud computing allows to cut costs by dynamically adapting the resource reservations to the needs of the SP system. It is challenging to strive the right balance between resource over-provisioning-which is costly, but is robust to workload uctuations-and on-demand scaling-which is cheap, but is vulnerable to sudden workload peaks. To this end, academia and industry developed elasticity methods. Again, they di er in their optimization objectives and assumptions about the operator parallelization model employed, the target system architecture, state management as well as timing and methodology.While there are many works that propose methods and solutions for speci c parallelization and elasticity problems in SP systems, there is a severe lack of overview, comparison, and classi cation of these methods. When we investigate...

show abstract

Section: Parallelization For General Streammentioning

confidence: 99%

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Röger

Mayer

2019

ACM Comput. Surv.

View full text Add to dashboard Cite

show abstract

“…The challenge of deriving insights from big data streams is recognized as a key challenge. Various authors proposed various algorithms and techniques for mining big data streams . The proposed algorithms illustrated various techniques to collect, integrate, analyze, and visualize big data streams in real time.…”

Section: Preliminariesmentioning

confidence: 99%

“…Various authors proposed various algorithms and techniques for mining big data streams. 13,14,[33][34][35][36][37][38] The proposed algorithms illustrated various techniques to collect, integrate, analyze, and visualize big data streams in real time. Nevertheless, less attention has been paid to resource scheduling for big data streams.…”

Section: Related Workmentioning

confidence: 99%

Dynamic resource allocation for big data streams based on data characteristics (5Vs)

Kaur

Sood

2017

Int J Network Mgmt

View full text Add to dashboard Cite

Summary Various Internet‐based applications such as social media, business transactions, mobile applications, cyber‐physical systems, and Internet of Things have led to the generation of big data streams in every field. The growing need to extract knowledge from big data streams has pioneered the challenge of selecting appropriate cloud resources. The current techniques allocate resources based on data characteristics. But because of the stochastic nature of data generation, the characteristics of data in big data streams are unknown. This poses difficulty in selecting and allocating appropriate resources to big data stream. Working towards this direction, this paper proposes a system that predicts the data characteristics in terms of volume, velocity, variety, variability, and veracity. The predicted values are expressed in a quadruple called Characteristics of Big data (CoBa). Thereafter, the proposed system uses self‐organizing maps to dynamically create clusters of cloud resources. One of these clusters is allocated to the big data stream based on its CoBa. The proposed system is dynamic in the sense that it changes the cloud cluster allocated to big data stream if its CoBa changes. Experimental results show that the proposed system has a performance edge over other streaming data processing tools such as Storm, Flume, and S4.

show abstract

“…IoT has been distinguished as a developing technology in numerous IT trend reports [1], and the number of IoT devices is proposed to increase [2,3]. It is suggested by some IT trend reports that the worldwide IoT market will be worth billions of dollars by 2022 [4]. The interconnection between the different kinds of IoT devices is a key issue for the achievement of IoT, in light of the fact that the numbers of IoT devices is developing ceaselessly.…”

Section: Introductionmentioning

confidence: 99%

Merging Securely M2M Protocols, Internet of Things and Cloud Computing

Darwish

2019

TNC

View full text Add to dashboard Cite

The Internet of Things provides new ways for communication through the Web world using objectenabled networks. At the same time, M2M devices intercommunication and their communication through the web if they were connected to the Internet, presents new challenges, especially in security, that traditional communication models have not yet fully solved. Because of their inborn un-watched, minimal effort and mass-sent nature, M2M devices, and remote communication architectures and solutions for these devices, would encapsulate new dangers in security. These threats are not fully faced by use of security technologies and methods implemented in existing wireless devices, cellular networks or WLANs. The use of cloud computing gives a convenient, on demand and scalable network access to a shared pool of configurable computing resources and devices. This paper concentrates on a secure method to integrate the M2M protocols with the Internet of Things (IoT) and Cloud Computing under the name of Secure Machine-to-Internet Clouding (SM2IC) architecture. The secure design for integrating M2M protocols, along with IoT and cloud computing is proposed. To apply this design, an IoT enabled smart home scenario was examined to analyze secure communication between M2M devices and IoT applications. Also, the cloud computing is used to include different cloud applications, such as, IaaS, PaaS, and SaaS for monitoring the quality of service of M2M devices through IoT applications. Then, simulations were performed to test the proposed security technique, followed by conclusions and future work. executed over the previous decade. In any case, most platforms were applied in light of particular solutions or created to address certain domain issues.To interconnect different proprietary platforms and deliver common IoT services to customers, there is a strong need to create a standardized IoT platform. To face interoperability issues, seven standard development organizations (SDOs) started a global standards project named oneM2M [5], by providing scalable and interoperable IoT standards for communication of devices and services. The goal of oneM2M is to present a single horizontal service platform, that can be implemented in different industries to deliver smarter IoT services to users and to exchange and share data among IoT applications. Machine-tomachine (M2M) communication is one of the next frontiers in wireless communication. There exist a large number of possibilities, in terms of new use cases, services and applications, that is suggested to result from communication between M2M devices. M2M can present benefits for the production and market opportunities for various manufacturers of M2M devices and components, service providers, and network operators.Due to the huge number of M2M devices expected to be used, in a highly distributed network, enforcement of traditional security methods will not be practical because of the high cost of implementation of these devices. Also, deploying the conventional centralized IT security network mod...

show abstract

Survey of Real-time Processing Technologies of IoT Data Streams

Cited by 107 publications

References 17 publications

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

Dynamic resource allocation for big data streams based on data characteristics (5Vs)

Merging Securely M2M Protocols, Internet of Things and Cloud Computing

Contact Info

Product

Resources

About