Big Data Processing Frameworks for Handling Huge Data Efficiencies and Challenges: A Survey

Abstract

The increasing expansion of digital data collected from many sources renders traditional storage, processing, and analysis methods obsolete. For these restrictions, new technologies for processing and storing very massive datasets have been developed. Big data processing is required to extract relevant information from it. Transforming data into information and knowledge is what processing implies. Big data processing is the process of dealing with massive amounts of data and changing it from its raw form into useable information in a more understandable manner. As a result, numerous big data processing execution frameworks have emerged, but determining and selecting the appropriate framework for processing your big data applications is a significant challenge. Therefore, this paper investigates the possible influence of big data challenges and discusses in depth the most well-known approaches to big data processing, which are divided into five classes: batch processing, streaming processing, realtime processing, interactive processing, and hybrid processing, as well as the variety of the most popular frameworks associated with them such as Apache Hadood, Dryad, Samza, IBM Infosphere, Storm, Amazon Kinesis, Drill, Impala, Flink, and Spark. Furthermore, this study presents a comparison among the several features of the frameworks by highlighting their drawbacks and strengths. Thus, it can be used as a guideline for picking the best application framework in IT analytics and will help busi-ness users make faster decisions.

Design and Implementation of a Malfunction Detection System for Livestock Ventilation Devices in Smart Poultry Farms

“…Pre-registered rules are used to judge the validity of malfunction detections, and the ontology is tasked with sending cloud notifications for future malfunction events. Such events can be saved and managed in RDBMS/HBase [29]. Queries related to the malfunction detection history are directly sent to RDBMS and HBase through the API, which allows users to inspect previous malfunction events.…”