aHDFS: An Erasure-Coded Data Archival System for Hadoop Clusters

Chen, Yuanqi; Yi, Zhou; Taneja, Shubbhi; Qin, Xiao; Huang, Jianzhong

doi:10.1109/tpds.2017.2706686

Cited by 15 publications

(5 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MapReduce algorithms have a wide range of applications, as mentioned repeatedly. They have emerged and continued to do so for the open-source implementations of Hadoop for millions of users handled at once [21] . The performance of Hadoop clusters is closely related to the short response times as processed by MapReduce [11] .…”

Section: Distributed Processing and Mapreducementioning

confidence: 99%

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Kumar,

Varshney,

Bhatiya

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

We live in an age where everything around us is being created. Data generation rates are so scary, creating pressure to implement costly and straightforward data storage and recovery processes. MapReduce model functionality is used for creating a cluster parallel, distributed algorithm, and large datasets. The MapReduce strategy from Hadoop helps develop a community of non-commercial use to offer a new algorithm for resolving such problems for commercial applications as expected from this working algorithm with insights as a result of disproportionate or discriminatory Hadoop cluster results. Expected results are obtained in the work and the exam conducted under this job; many of them are scheduled to set schedules, match matrices' data positions, clustering before determining to click, and accurate mapping and internal reliability to be closed together to avoid running and execution times. Mapper output and proponents have been implemented, and the map has been used to reduce the function. The execution input key/value pair and output key/value pair have been set. This paper focuses on evaluating this technique for the efficient retrieval of large volumes of data. The technique allows for capabilities to inform a massive database of information, from storage and indexing techniques to the distribution of queries, scalability, and performance in heterogeneous environments. The results show that the proposed work reduces the data processing time by 30%.

show abstract

Section: Distributed Processing and Mapreducementioning

confidence: 99%

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Kumar,

Varshney,

Bhatiya

et al. 2023

Big Data Min. Anal.

View full text Add to dashboard Cite

show abstract

“…HDFS is divided into metadata of the storage file system and application data. As presented in Figure 7, HDFS can store all files in the system as a series of blocks of the same size [27,28].…”

Section: Typical Dfs Figure 4 Illustrates the Categories Of Dfsmentioning

confidence: 99%

Optimized Design of Multilines Center of Subway AFC System via Distributed File System and Bayesian Network Model

Fang

Jiang

Lin³

et al. 2021

Journal of Sensors

View full text Add to dashboard Cite

Automatic fare collection system (AFCS) is a modern, automatic, networked toll collection system for rail transit ticket sales, collection, billing, charging, statistics, sorting, and management. To realize the subway transit networking operation, this paper designs the subway AFCS based on a distributed file system (DFS), namely, Gluster File System (GlusterFS). Firstly, the multiline center (MLC) in the subway AFCS is designed to analyze the status and current situation of distributed file processing in subway MLC system; secondly, the relevant technical theories are summarized, the Bayesian Network (BN) theoretical model and DFS are explored, and the principles of four DFS are comparatively analyzed; thirdly, the architecture and cluster mode of GlusterFS is expounded, and then based on GlusterFS, the architecture of subway AFCS is discussed. This paper presents several innovation points: first, the subway AFCS is designed based on GlusterFS by analyzing and aiming at the functional requirements, performance requirements, and safety requirements of the MLC subway system; second, the safety risk analysis (SRA) of AFCS is conducted from six security requirements, and a Web scanning system is designed to ensure the system data security. Finally, the design scheme is used to analyze the subway passenger flow and power consumption. The results demonstrate that the design scheme can competently adapt to three different application scenarios. Through comparison of two deployment modes of the Web scanning system, the data security Web scanning system can ensure the safe operation of the AFCS. Furthermore, the statistical analysis of subway passenger flow and power supply data shows that the proposed scheme can support the smooth operation of the subway system, which has significant practical value.

show abstract

“…In Hadoop, users can not only use the data in the default format of the system but also store and analyze the data in a customized format, so they can store and analyze all kinds of data [9]. is paper analyses the shortcomings of setting delay time interval in delay scheduling algorithm, clarifies the importance of delay time interval to localized scheduling, and gives a reasonable setting scheme of delay time on the basis of retaining the conciseness and efficiency of the delay scheduling algorithm, so as to make the delay scheduling algorithm play its optimal efficiency [10]. When small jobs are submitted late, the actual execution time may be much less than the waiting time, and if there are several large jobs that need to be run for a long time before, the waiting time will be more.…”

Section: Introductionmentioning

confidence: 99%

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Wang

2022

Computational Intelligence and Neuroscience

View full text Add to dashboard Cite

Considering that in the process of job scheduling, the cluster load should be prebalanced rather than remedied when the load is seriously unbalanced; therefore, in this paper, the task scheduling flow of the Hadoop cluster is analyzed deeply. On the Hadoop platform, a self-dividing algorithm is proposed for load balancing. An intelligent optimization algorithm is used to solve load balance. A dynamic feedback load balancing scheduling method is proposed from the point of view of task scheduling. In order to solve the shortcoming of the fair scheduling algorithm, this paper proposes two ways to improve the resource utilization and overall performance of Hadoop. When the mapping task is completed and the tasks to be reduced are assigned, the task assignment is based on the performance of the nodes to be reduced. It gives full play to the advantages of the ant colony algorithm and the hive colony algorithm so that the fusion algorithm can better deal with load balance. Then, three existing scheduling algorithms are introduced in detail: single queue scheduling, capacity scheduling, and fair scheduling. On this basis, an improved task scheduling strategy based on genetic algorithm is proposed to allocate and execute application tasks to reduce task completion time. The experiment verifies the effectiveness of the algorithm. The LBNP algorithm greatly improves the efficiency of reducing task execution and job execution. The delay capacity scheduling algorithm can ensure that most tasks can achieve localization scheduling, improve resource utilization, improve load balance, and speed up job completion time.

show abstract

aHDFS: An Erasure-Coded Data Archival System for Hadoop Clusters

Cited by 15 publications

References 21 publications

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Replication-Based Query Management for Resource Allocation Using Hadoop and MapReduce over Big Data

Optimized Design of Multilines Center of Subway AFC System via Distributed File System and Bayesian Network Model

Load Balancing Algorithms for Hadoop Cluster in Unbalanced Environment

Contact Info

Product

Resources

About