Performance models and dynamic characteristics analysis for HDFS write and read operations: A systematic view

Dong, Bo; Zheng, Qinghua; Tian, Feng; Chao, Kuo‐Ming; Godwin, Nick; Ma, Tao; Xu, Hui

doi:10.1016/j.jss.2014.02.038

Cited by 18 publications

(14 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To study the relationship between the file size and HDFS W/R performance, a file set containing 78 files with different sizes is generated. Based on prior knowledge of HDFS [2], [4], its W/R performance changes quickly and sharply for small files, and relatively turn some kind of stable as the increase of file size. Therefore, the range of size of the 78 files is from 0.25 MB to 320 MB.…”

Section: Experiments and Analysismentioning

confidence: 99%

“…Thirdly, analytical modeling typically depends on some simplifying assumptions on the system or workload behaviors, thus the accuracy of analytical models may be seriously challenged in scenarios when such assumptions are no longer satisfied [17]. The other measurement-based methodology (e.g., [4], [5], [18], [19]), which is to implement experimentation upon the system by running benchmarks, application programs, or specially designed data set, is a promising way to deal with the randomness since it requires no expertise of the dynamic behaviors of system or workloads. Evidently, the measurement-based methodology relies on the observed system behaviors which are always subject to various kinds of uncertainties due to external disturbance or modeling error.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Ordinal Optimization-Based Performance Model Estimation Method for HDFS

Tian

Dong

2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

Modeling and analyzing the performance of distributed file systems (DFSs) benefit the reliability and quality of data processing in data-intensive applications. Hadoop Distributed File System (HDFS) is a typical representative of DFSs. Its internal heterogeneity and complexity as well as external disturbance contribute to HDFS's built-in features of nonlinearity as well as randomness in system level, which raises a great challenge in modeling these features. Particularly, the randomness results in the uncertainty of HDFS performance model. Due to the complex mathematical structure and parameters hardly estimated of analytical models, it is highly complicated and computationally impossible to build an explicit and precise analytical model of the randomness. The measurement-based methodology is a promising way to model HDFS performance in terms of randomness since it requires no knowledge of system's internal behaviors. In this paper, the estimation of HDFS performance models on account of the randomness is transformed to an optimization problem of finding out the real best design of performance model structure with large design space. Core ideas of ordinal optimization (OO) are introduced to solve this problem with a limited computing budget. Piecewise linear (PL) model is applied to approximate the nonlinear characteristics and randomness of HDFS performance. The experimental results show that the proposed method is effective and practical to estimate the optimal design of the PL-based performance model structure for HDFS. It not only provides a globally consistent evaluation of the design space but also guarantees the goodness of the solution with high probability. Moreover, it improves the accuracy of system model-based HDFS performance models.

show abstract

Section: Experiments and Analysismentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Ordinal Optimization-Based Performance Model Estimation Method for HDFS

Tian

Dong

2020

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…A similar research work was conducted by Dong et al In the aforementioned work, the authors presented a performance characterization and modeling for read and write operations on the Hadoop Distributed File System (HDFS). The HDFS is a file system designed for huge data analysis problems, often implemented through map reduce programming models.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Saini et al 8 On previous papers, 42,43 we have approached the performance and workload characterization problem from a different perspective, focusing on understanding the impact of different parameters. In a previous work, 42 we presented an analysis on the effect of multiple parameters on the I/O performance of write operations on a PFS, whereas in another work, 43 A similar research work was conducted by Dong et al 44 In the aforementioned work, 44 the authors presented a performance characterization and modeling for read and write operations on the Hadoop Distributed File System (HDFS). The HDFS is a file system designed for huge data analysis problems, often implemented through map reduce programming models.…”

Section: Performance and Workload Characterizationmentioning

confidence: 99%

A coarse‐grained page cache aware multivariate analytical model for the storage performance of a parallel file system

Inacio

Dantas

2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Huge requirements of emerging Big Data applications combined with the performance impairment of current I/O subsystems pose a great challenge on data storage, management, and access performance. In order to design efficient storage systems, a clear understanding about how factors in the I/O path affect the performance of a data‐intensive application is of utmost importance. This paper reports our ongoing research toward addressing this issue, presenting a coarse‐grained page cache aware multivariate analytical model for the performance of write operations in a parallel file system. The proposed model was developed to reflect the performance behavior observed in an extensive experimental effort, in which the impact of 14 parameters in the response time and throughput of the OrangeFS was investigated. More than one million experiments were carried out using four distinct computing infrastructures, providing a detailed performance characterization. Additionally, a thorough evaluation of the proposed model, covering more than 14 000 scenarios, is reported, discussing both qualitative and quantitative aspects. Evaluation results indicate that the model succeeded in representing the behavior of the parallel file system performance, achieving a Mean Absolute Percentage Error of 39.94%.

show abstract

“…Hadoop is a representative open-source software framework for the distributed storage and processing of large amounts of data. As Hadoop stores and processes large amounts of data in the disks of distributed nodes, continuous disk input and output occur, making real-time processing impossible [9][10][11]. Furthermore, when input and output are concentrated on a specific node, a bottleneck occurs, and the overall processing speed is lowered.…”

Section: Introductionmentioning

confidence: 99%

Load Balancing Scheme for Effectively Supporting Distributed In-Memory Based Computing

Bok

Choi

et al. 2019

Electronics

View full text Add to dashboard Cite

As digital data have increased exponentially due to an increasing number of information channels that create and distribute the data, distributed in-memory systems were introduced to process big data in real-time. However, when the load is concentrated on a specific node in a distributed in-memory environment, the data access performance is degraded, resulting in an overall degradation in the processing performance. In this paper, we propose a new load balancing scheme that performs data migration or replication according to the loading status in heterogeneous distributed in-memory environments. The proposed scheme replicates hot data when the hot data occurs on the node where a load occurs. If the load of the node increases in the absence of hot data, the data is migrated through a hash space adjustment. In addition, when nodes are added or removed, data distribution is performed by adjusting the hash space with the adjacent nodes. The clients store the metadata of the hot data and reduce the access of the load balancer through periodic synchronization. It is confirmed through various performance evaluations that the proposed load balancing scheme improves the overall load balancing performance.

show abstract

Performance models and dynamic characteristics analysis for HDFS write and read operations: A systematic view

Cited by 18 publications

References 18 publications

Ordinal Optimization-Based Performance Model Estimation Method for HDFS

Ordinal Optimization-Based Performance Model Estimation Method for HDFS

A coarse‐grained page cache aware multivariate analytical model for the storage performance of a parallel file system

Load Balancing Scheme for Effectively Supporting Distributed In-Memory Based Computing

Contact Info

Product

Resources

About