MR-Advisor: A comprehensive tuning, profiling, and prediction tool for MapReduce execution frameworks on HPC clusters

Rahman, Md. Wasi-ur; Islam, Nusrat Sharmin; Lu, Xiaoyi; Shankar, Dipti; Panda, Dhabaleswar K.

doi:10.1016/j.jpdc.2017.11.004

Cited by 19 publications

(1 citation statement)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Each server is independent with the capability to provide independent data services. This method solves the two inevitable problems of centralized metadata management but also makes the system complicated with a performance overhead and consistency problems [29,30]. Figure 8 shows that the server and client make up the GlusterFS; Remote Direct Memory Access (RDMA) or TCP is used to link the cluster server; the server can contain one or multiple nodes, and each node may contain multiple blocks; additionally, each block can be any directory in the LFS, and then, multiple blocks are combined to form various types of volume.…”

Section: Metadata Managementmentioning

confidence: 99%

Optimized Design of Multilines Center of Subway AFC System via Distributed File System and Bayesian Network Model

Fang

Jiang

Lin³

et al. 2021

Journal of Sensors

View full text Add to dashboard Cite

Automatic fare collection system (AFCS) is a modern, automatic, networked toll collection system for rail transit ticket sales, collection, billing, charging, statistics, sorting, and management. To realize the subway transit networking operation, this paper designs the subway AFCS based on a distributed file system (DFS), namely, Gluster File System (GlusterFS). Firstly, the multiline center (MLC) in the subway AFCS is designed to analyze the status and current situation of distributed file processing in subway MLC system; secondly, the relevant technical theories are summarized, the Bayesian Network (BN) theoretical model and DFS are explored, and the principles of four DFS are comparatively analyzed; thirdly, the architecture and cluster mode of GlusterFS is expounded, and then based on GlusterFS, the architecture of subway AFCS is discussed. This paper presents several innovation points: first, the subway AFCS is designed based on GlusterFS by analyzing and aiming at the functional requirements, performance requirements, and safety requirements of the MLC subway system; second, the safety risk analysis (SRA) of AFCS is conducted from six security requirements, and a Web scanning system is designed to ensure the system data security. Finally, the design scheme is used to analyze the subway passenger flow and power consumption. The results demonstrate that the design scheme can competently adapt to three different application scenarios. Through comparison of two deployment modes of the Web scanning system, the data security Web scanning system can ensure the safe operation of the AFCS. Furthermore, the statistical analysis of subway passenger flow and power supply data shows that the proposed scheme can support the smooth operation of the subway system, which has significant practical value.

show abstract

Section: Metadata Managementmentioning

confidence: 99%