Real-time processing of IoT events with historic data using Apache Kafka and Apache Spark with dashing framework

D’silva, Godson Michael; Khan, Azharuddin; Gaurav, .; Bari, Siddhesh

doi:10.1109/rteict.2017.8256910

Cited by 28 publications

(9 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The proposed system was capable of efficiently handling massive amounts of sensor data when the amount of data and the number of clients increased. D’silva et al proposed a framework for handling real-time IoT event data [ 22 ]. The proposed framework utilized Apache Kafka as a message queue system and was efficient enough to process real-time IoT events data.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Big data analysis has led to significant improvements in the manufacturing industry, such as reducing energy consumption [ 17 ], improving production scheduling and logistics planning [ 18 ], mitigating social risks [ 19 ], and facilitating better decision making [ 20 ]. Previous studies have shown significant benefits from several big data technologies in processing and storing large volumes of data quickly, such as with the application of Apache Kafka [ 21 , 22 , 23 , 24 , 25 , 26 ], Apache Storm [ 27 , 28 , 29 , 30 , 31 ], and NoSQL MongoDB [ 32 , 33 , 34 , 35 , 36 , 37 ]. Previous studies showed significant advantages from the integration of big data technologies such as reducing the processing time for home automation systems [ 38 ], providing effective and efficient solutions for processing IoT-generated data for smart cities [ 39 ], and handling large amounts of smart environmental data in real-time [ 40 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Syafrudin

Alfian

Fitriyani

et al. 2018

Sensors

280

103

View full text Add to dashboard Cite

With the increase in the amount of data captured during the manufacturing process, monitoring systems are becoming important factors in decision making for management. Current technologies such as Internet of Things (IoT)-based sensors can be considered a solution to provide efficient monitoring of the manufacturing process. In this study, a real-time monitoring system that utilizes IoT-based sensors, big data processing, and a hybrid prediction model is proposed. Firstly, an IoT-based sensor that collects temperature, humidity, accelerometer, and gyroscope data was developed. The characteristics of IoT-generated sensor data from the manufacturing process are: real-time, large amounts, and unstructured type. The proposed big data processing platform utilizes Apache Kafka as a message queue, Apache Storm as a real-time processing engine and MongoDB to store the sensor data from the manufacturing process. Secondly, for the proposed hybrid prediction model, Density-Based Spatial Clustering of Applications with Noise (DBSCAN)-based outlier detection and Random Forest classification were used to remove outlier sensor data and provide fault detection during the manufacturing process, respectively. The proposed model was evaluated and tested at an automotive manufacturing assembly line in Korea. The results showed that IoT-based sensors and the proposed big data processing system are sufficiently efficient to monitor the manufacturing process. Furthermore, the proposed hybrid prediction model has better fault prediction accuracy than other models given the sensor data as input. The proposed system is expected to support management by improving decision-making and will help prevent unexpected losses caused by faults during the manufacturing process.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Syafrudin

Alfian

Fitriyani

et al. 2018

Sensors

280

103

View full text Add to dashboard Cite

show abstract

“…The rapid improvement in industrial operations and technology, the internet of things, smart gadgets, and social media, digital data has grown in volume and complexity at a rapid rate [1][2][3][4]. Big data is a collection of records with a large volume and a rate of exponential growth over time [5].…”

Section: Introductionmentioning

confidence: 99%

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Junaid

Ali

Siddiqui

et al. 2022

Wireless Pers Commun

View full text Add to dashboard Cite

Artificial intelligence, specifically machine learning, has been applied in a variety of methods by the research group to transform several data sources into valuable facts and understanding, allowing for superior pattern identification skills. Machine learning algorithms on huge and complicated data sets, computationally expensive on the other hand, processing requires hardware and logical resources, such as space, CPU, and memory. As the amount of data created daily reaches quintillion bytes, A complex big data infrastructure becomes more and more relevant. Apache Spark Machine learning library (ML-lib) is a famous platform used for big data analysis, it includes several useful features for machine learning applications, involving regression, classification, and dimension reduction, as well as clustering and features extraction. In this contribution, we consider Apache Spark ML-lib as a computationally independent machine learning library, which is open-source, distributed, scalable, and platform. We have evaluated and compared several ML algorithms to analyze the platform’s qualities, compared Apache Spark ML-lib against Rapid Miner and Sklearn, which are two additional Big data and machine learning processing platforms. Logistic Classifier (LC), Decision Tree Classifier (DTc), Random Forest Classifier (RFC), and Gradient Boosted Tree Classifier (GBTC) are four machine learning algorithms that are compared across platforms. In addition, we have tested general regression methods such as Linear Regressor (LR), Decision Tree Regressor (DTR), Random Forest Regressor (RFR), and Gradient Boosted Tree Regressor (GBTR) on SUSY and Higgs datasets. Moreover, We have evaluated the unsupervised learning methods like K-means and Gaussian Mixer Models on the data set SUSY and Hepmass to determine the robustness of PySpark, in comparison with the classification and regression models. We used ”SUSY,” ”HIGGS,” ”BANK,” and ”HEPMASS” dataset from the UCI data repository. We also talk about recent developments in the research into Big Data machines and provide future research directions.

show abstract

“…Thirdly, in a Big Data platform, data transmission is a necessary technology for sending data from the factory to other locations (cross-platform) [20]. Many researchers have implemented these technologies, some of them including ZeroMQ [21], ActiveMQ [22], RabbitMQ [23], and Apache Kafka [24].…”

Section: Related Studiesmentioning

confidence: 99%

“…In order to build upon the performance of his proposed method, Apache Spark was employed for video data processing. Similarly, machine log data was generated rapidly, so transmitting the sensing data requires reliable technology, D’silva et al [20] proposed a method for transmitting and processing the IoT historic sensing data, that integrated with Dashing Framework, to visualize the historical data on a graph. In the data transmission phase, security is of concern during sending and receiving the data from the Kafka server.…”

Section: Related Studiesmentioning

confidence: 99%

Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment

Leang

Ean

Ryu

et al. 2019

Sensors

View full text Add to dashboard Cite

The large amount of programmable logic controller (PLC) sensing data has rapidly increased in the manufacturing environment. Therefore, a large data store is necessary for Big Data platforms. In this paper, we propose a Hadoop ecosystem for the support of many features in the manufacturing industry. In this ecosystem, Apache Hadoop and HBase are used as Big Data storage and handle large scale data. In addition, Apache Kafka is used as a data streaming pipeline which contains many configurations and properties that are used to make a better-designed environment and a reliable system, such as Kafka offset and partition, which is used for program scaling purposes. Moreover, Apache Spark closely works with Kafka consumers to create a real-time processing and analysis of the data. Meanwhile, data security is applied in the data transmission phase between the Kafka producers and consumers. Public-key cryptography is performed as a security method which contains public and private keys. Additionally, the public-key is located in the Kafka producer, and the private-key is stored in the Kafka consumer. The integration of these above technologies will enhance the performance and accuracy of data storing, processing, and securing in the manufacturing environment.

show abstract

Real-time processing of IoT events with historic data using Apache Kafka and Apache Spark with dashing framework

Cited by 28 publications

References 6 publications

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Performance Analysis of IoT-Based Sensor, Big Data Processing, and Machine Learning Model for Real-Time Monitoring System in Automotive Manufacturing

Performance Evaluation of Data-driven Intelligent Algorithms for Big data Ecosystem

Improvement of Kafka Streaming Using Partition and Multi-Threading in Big Data Environment

Contact Info

Product

Resources

About