Efficient Streaming Mass Spatio-Temporal Vehicle Data Access in Urban Sensor Networks Based on Apache Storm

Abstract

The efficient data access of streaming vehicle data is the foundation of analyzing, using and mining vehicle data in smart cities, which is an approach to understand traffic environments. However, the number of vehicles in urban cities has grown rapidly, reaching hundreds of thousands in number. Accessing the mass streaming data of vehicles is hard and takes a long time due to limited computation capability and backward modes. We propose an efficient streaming spatio-temporal data access based on Apache Storm (ESDAS) to achieve real-time streaming data access and data cleaning. As a popular streaming data processing tool, Apache Storm can be applied to streaming mass data access and real time data cleaning. By designing the Spout/bolt workflow of topology in ESDAS and by developing the speeding bolt and other bolts, Apache Storm can achieve the prospective aim. In our experiments, Taiyuan BeiDou bus location data is selected as the mass spatio-temporal data source. In the experiments, the data access results with different bolts are shown in map form, and the filtered buses’ aggregation forms are different. In terms of performance evaluation, the consumption time in ESDAS for ten thousand records per second for a speeding bolt is approximately 300 milliseconds, and that for MongoDB is approximately 1300 milliseconds. The efficiency of ESDAS is approximately three times higher than that of MongoDB.

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

“…Previous studies showed significant advantages from the integration of big data technologies such as reducing the processing time for home automation systems [ 38 ], providing effective and efficient solutions for processing IoT-generated data for smart cities [ 39 ], and handling large amounts of smart environmental data in real-time [ 40 ]. The aforementioned big data technologies have been integrated in data processing systems, resulting in significant advantages due to processing large amounts of streaming spatiotemporal data [ 41 ] as well as processing massive amounts of manufacturing sensor data efficiently [ 42 ]. Therefore, it is necessary to integrate Apache Kafka, Apache Storm, and MongoDB in big data processing systems for the manufacturing industry so that large amounts of streaming manufacturing sensor data can be promptly processed, stored, and presented in real-time.…”

A Scalable Architecture for Real-Time Stream Processing of Spatiotemporal IoT Stream Data—Performance Analysis on the Example of Map Matching

“…Zhou et al [12] propose a system for efficient streaming access and cleaning of spatiotemporal data, which is also based on Apache Storm. It uses Apache Kafka, a distributed, partitioned, replicated commit log service in order to integrate the data into the Storm cluster.…”

A Real-Time and Open Geographic Information System and Its Application for Smart Rivers: A Case Study of the Yangtze River

“…Since Spark has outstanding processing capabilities for massive real-time data, Spark has attracted greater attention among developers. Meanwhile, the current trend is to use real-time computation technologies in a GIS [27]. Therefore, Spark also supports real-time and open GIS.…”