Task Scheduling for Spark Applications With Data Affinity on Heterogeneous Clusters

Xiaodong, Zhang; Li, Xiaoping; Du, Houan; Ruíz, Rubén

doi:10.1109/jiot.2022.3181997

Cited by 2 publications

(1 citation statement)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The system was built on a big data framework to establish a data processing platform for the statistical analysis of data. The platform was built upon HDFS (Hadoop Distributed File System) [36] and Spark [37] to establish an offline data warehouse, which is utilized for processing parsed sow production data. The data platform constructs hierarchical dimension tables, including fields such as pig farm ID, house number, stall number, entry time, exit time, and farrowing date.…”

Section: Data Processing Programmentioning

confidence: 99%

An Internet of Things-Based Cluster System for Monitoring Lactating Sows’ Feed and Water Intake

He,

Zeng,

Liu

et al. 2024

Agriculture

View full text Add to dashboard Cite

Acquiring real-time feeding information for monitoring lactating sows and their feeding requirements is a challenging task. Real-time data represent an important input for numerous tasks, such as disease monitoring, nutritional regulation, and feeding modeling. However, concurrently monitoring large numbers of sows and processing the real-time information for modeling is challenging using existing platforms. In this paper, we describe the design and development of a system that monitors and processes sows’ feed and water consumption in real time. The system was custom-developed using open-source networking technologies. The system consists of three components: an electronic sow feeder connected to a central controller via a CAN network, an MQTT service cluster, and a data processing program. The MQTT service cluster uses Netty to develop a single service node, and it uses Zookeeper and Redis to complete node registration, discovery, and scheduling. The data processing program is based on Spark and Flink. We conducted comparative testing of three common codecs (Java Serializer, Marshalling, and Protostuff) to further speed up data transmission. The results of the experiment show that, with three service nodes, the system can concurrently monitor up to 20,000 sows. Moreover, the system achieves optimal performance when monitoring 10,000 sows at the same time, with a TPS of 6399 pcs/s and an RT of 643 ms.

show abstract

Section: Data Processing Programmentioning

confidence: 99%

An Internet of Things-Based Cluster System for Monitoring Lactating Sows’ Feed and Water Intake

He,

Zeng,

Liu

et al. 2024

Agriculture

View full text Add to dashboard Cite

show abstract

AMORA: An Advanced Malleable and Operational Framework for Performance Prediction of Big Data Systems

Lin,

Xu,

Zhong

et al. 2024

Softw Pract Exp

View full text Add to dashboard Cite

BackgroundIn the data era, big data systems have emerged as pivotal tools, underscoring the importance of performance prediction in enhancing the efficiency of big data clusters. Numerous performance models have been proposed, often grounded in artificial intelligence or simulation methodologies. While the bulk of research focuses on refining prediction precision and minimizing overhead, limited attention has been given to the consignation and standardization of these models.ObjectivesTo bridge this gap between model developers and end‐users, this paper introduces AMORA—a novel versatile framework tailored for predicting the performance of big data systems.MethodsLeveraging the identified behavior descriptions‐computation submodels (BD‐CS) pattern that is prevalent among various big data job performance models, AMORA allows access to different plugins accommodating different performance models' implementations. This framework also integrates a novel mutable computation graph technique to facilitate backtracking computation. Furthermore, AMORA's functionality extends to comprehensive end‐to‐end usability by enabling the acceptance of origin configuration files from diverse big data systems and presenting easily interpretable prediction reports.ResultsThis work demonstrates AMORA's efficacy in producing an accurate trace of Hadoop job through the selection of appropriate performance model plugins and parameter adjustments and showcasing the application of the proposed mutable computation graph technique in calculating the starting moment of an early‐start reducer. Additionally, two validation experiments are conducted, involving the implementation of various Hadoop and Spark performance models, respectively. The experiment results manifest the prediction precision and overheads of these performance models.ConclusionThese experiments exhibit AMORA's role as a benchmark platform for implementing various types of big data job performance models catered to diverse big data systems.

show abstract

Task Scheduling for Spark Applications With Data Affinity on Heterogeneous Clusters

Cited by 2 publications

References 29 publications

An Internet of Things-Based Cluster System for Monitoring Lactating Sows’ Feed and Water Intake

An Internet of Things-Based Cluster System for Monitoring Lactating Sows’ Feed and Water Intake

AMORA: An Advanced Malleable and Operational Framework for Performance Prediction of Big Data Systems

Contact Info

Product

Resources

About