Ordinal Optimization-Based Performance Model Estimation Method for HDFS

Ma, Tao; Tian, Feng; Dong, Bo

doi:10.1109/access.2019.2962724

Cited by 9 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This not only reduces data access latency, but also provides load balancing of data access requests. This motivated the authors of [18], [19], [20] to investigate the performance of HDFS in remote data access.…”

Section: Related Workmentioning

confidence: 99%

Experimental Analysis of WebHDFS API Throughput

Kalmukov¹,

Marinov²

2023

IJACSA

View full text Add to dashboard Cite

Data analysis is very important for the success of any business today. It helps to optimize business processes, analyze users' behavior, demands etc. There are powerful data analytics tools, such as the ones of the Hadoop ecosystem, but they require multiple high-performance servers to run and highqualified experts to install, configure and support them. In most cases, small companies and start-ups could not afford such expenses. However, they can use them as web services, on demand, and pay much lower fees per request. To do that, companies should somehow share their data with an existing, already deployed, Hadoop cluster. The most common way of uploading their files to the Hadoop's Distributed File System (HDFS) is through the WebHDFS API (Application Programming Interface) that allows remote access to HDFS. For that reason, the API's throughput is very important for the efficient integration of a company's data to the Hadoop cluster. This paper performs a series of experimental analyses aiming to determine the WebHDFS API's throughput, if it is a bottleneck in integration of a company's data to existing Hadoop infrastructure and to detect all possible factors that influence the speed of data transmission between the clients' software and the Hadoop' file system.

show abstract

Section: Related Workmentioning

confidence: 99%

Experimental Analysis of WebHDFS API Throughput

Kalmukov¹,

Marinov²

2023

IJACSA

View full text Add to dashboard Cite

show abstract

“…These assumptions frame the architecture of HDFS and allow the optimization of several aspects [7], especially data security and processing speed.…”

Section: B Hadoop Distributed File Systemmentioning

confidence: 99%

“…Many works have tried to improve the strategy of placing chunks in DFS. Some works have proposed to change the storage method by reducing the number of small files by grouping them into larger files [5], or to optimize the replication factor and the size of chunks to improve internal exchanges inside the cluster [6], [7]. Another approach is to try to assign the data to the best node according to the known physical capacities of each node.…”

Section: ) Related Workmentioning

confidence: 99%

New Data Placement Strategy in the HADOOP Framework

Elomari¹,

Hassouni²,

Maizate³

2021

IJACSA

View full text Add to dashboard Cite

Today, the data quantities generated and exchanged between information systems continues to increase. Storing and exploiting such quantities require can't be done without bigdata systems with mechanisms capable of meeting technological challenges commonly grouped under the four Vs (Volume, Velocity, Variety and Veracity). These technologies include mainly the Distributed File System (DFS). Like Hadoop, which is based on HDFS, the main Big Data systems use a data distributed storage where a subsystem is responsible for subdividing data (data striping) and replicating it on a network of nodes called Grid. In the typical case of Hadoop, a Grid generally consists of many nodes, grouped in multiple Racks. The logic of distributing the stored data through the Grid respects a simple strategy that guarantees the durability of the data and a certain speed of writing. This strategy does not take into consideration neither the technical characteristics of nodes, nor the number of requests on the data, which means a considerable loss in processing capacity of the grid. In this work we proposed a new placement strategy based on exploitation analysis of new information integrated into the HDFS metadata model. A significant 20% improvement in overall processing time was reached through the simulations we conducted on Hadoop.

show abstract

“…22 OO has been widely-used in many complex optimization problems. Ma et al 23 incorporated the OO into piecewise linear algorithm to improve the performance optimization of distributed ﬁle systems. Ni et al 24 applied it to maximize core utilization in parallel computing.…”

Section: Introductionmentioning

confidence: 99%

Redundancy optimization for multi-state series-parallel systems using ordinal optimization-based-genetic algorithm

Ding

Zeng

2021

Proceedings of the Institution of Mechanical Engineers, Part O:

View full text Add to dashboard Cite

Multi-state series-parallel systems (MSSPSs) are widely-used for representing engineering systems. In real-life cases, engineers need to design an optimal MSSPS structure by combining different versions and number of redundant components. The objective of the design is to ensure reliability requirements using the least costs, which could be formulated as a redundancy optimization problem under reliability constraints. The genetic algorithm is one of the most frequently used method for solving redundancy optimization problems. In traditional genetic algorithms, the population size needs to be determined based on the experience of the modeler. Often, this ends up creating a large number of unnecessary samples. As a result, the computational burden can be huge, especially for large-scale MSSPS structures. To solve these problems, this paper proposes an optimal structure designing method named as redundancy ordinal optimization. The universal generating function technique is applied to evaluate the reliabilities of the MSSPSs. Based on the reliabilities, an ordinal optimization algorithm is adapted to update the parent populations and the stopping criterion of genetic algorithm, so that the unnecessary structure designs can be eliminated. Numerical examples show that the proposed method improves the computational efficiency while remaining satisfactorily accurate.

show abstract

Ordinal Optimization-Based Performance Model Estimation Method for HDFS

Cited by 9 publications

References 35 publications

Experimental Analysis of WebHDFS API Throughput

Experimental Analysis of WebHDFS API Throughput

New Data Placement Strategy in the HADOOP Framework

Redundancy optimization for multi-state series-parallel systems using ordinal optimization-based-genetic algorithm

Contact Info

Product

Resources

About