Heterogeneous Job Allocation Scheduler for Hadoop MapReduce Using Dynamic Grouping Integrated Neighboring Search

Chen, Chi-Ting; Hung, Ling-Ju; Hsieh, Sun Yuan; Buyya, Rajkumar; Zomaya, Albert Y.

doi:10.1109/tcc.2017.2748586

Cited by 31 publications

(13 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An AWS account was created to use Amazon EC2 and S3. Next, we compare the performance evaluation results of our algorithm with those of previous Hadoop improvement algorithms, including DGNS [27]. The experimental results are used to validate…”

Section: Resultsmentioning

confidence: 99%

“…The drawback of this optimization method is the high processing time, which is largely due to the convergence of the initial guess [9]. A heterogeneous job allocation scheduler with a dynamic grouping integrated neighboring search algorithm was proposed in [27]. This algorithm processes tasks based on (1) the job classification (the job type, such as CPU bound or I/O bound), (2) the ratio table (a capability ratio table is created for task trackers and data nodes), (3) data block allocation and grouping (grouping with CPU slot numbers), and (4) neighboring searches (CPU task allocation and I/O task allocation).…”

Section: Problem Statementmentioning

confidence: 99%

See 1 more Smart Citation

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

et al. 2020

View full text Add to dashboard Cite

The improvement of Hadoop performance has received considerable attention from researchers in cloud computing fields. Most studies have focused on improving the performance of a Hadoop cluster. Notably, various parameters are required to configure Hadoop and must be adjusted to improve performance. This paper proposes a mechanism to improve Hadoop, schedule jobs, and allocate and utilize resources. Specifically, we present an improved ant colony optimization method to schedule jobs according to the job size and the time expected for execution. Priority is given to the job with the minimum data size and minimum response time. The resource usage and running jobs by data node are predicted using an artificial neural network, and job activity and resource usage are monitored using the resource manager. Moreover, we enhance the Hadoop Name node performance by adding an aggregator node to the default HDFS framework architecture. The changes involve four entities: the name node, secondary name node, aggregator nodes, and data nodes, where the aggregator node is responsible for assigning the jobs among the data node, and the Name node keeps tracking only the aggregator nodes. We test the overall scheme among Amazon EC2 and S3, and show the results of throughput and CPU response time for different data sizes. Finally, we show that the proposed approach shows significant improvement compare to native Hadoop and other approaches.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Problem Statementmentioning

confidence: 99%

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Chi-Ting Chen et al categorizes a batch of MapReduce jobs into two groups (CPU-intensive and IO-intensive) using "dynamic grouping integrated neighbouring search strategy" [12] to improve resource utilization and number of data-local executions in heterogeneous computing environment. There are four phases in this proposed method.…”

Section: Literature Surveymentioning

confidence: 99%

Handling Non-Local Executions to Improve MapReduce Performance Using Ant Colony Optimization

et al. 2021

View full text Add to dashboard Cite

Improving the performance of the MapReduce scheduler is a primary objective, especially in a heterogeneous virtual cloud environment. A map task is assigned with an input split(IS) which consists of one or more data blocks. When a map task is assigned to more than one data block, non-local execution is performed. In classical MapReduce scheduling schemes, data blocks are copied over the network to a node in where the map task is running. This increases job latency and consumes more network bandwidth within and between racks in the cloud data-center. Considering this situation, we propose a methodology "improving data locality using ant colony optimization (IDLACO)" to minimize the number of non-local executions and virtual network bandwidth consumption when IS are assigned to more than one data block. First IDLACO determines a list of an optimal number of data blocks for each map task of a job to perform a non-local execution reducing the job latency and virtual network consumption. Then, the target virtual machine to execute the map task is determined on the basis of its heterogeneous performance. Finally, if a set of data blocks is transferred to the same node for repeated job execution, it is decided to temporarily cache those data block in the target virtual machine. The performance of IDLACO is analysed and compared with fair scheduler and Holistic scheduler based on the parameters, such as the number of non-local executions, average map task latency, job latency, and amount of bandwidth consumed for a MapReduce job. Results show that our proposed IDLACO significantly outperforms the classical fair scheduler and Holistic scheduler.

show abstract

“…The ability to model such behaviors and pick the correct execution model 78 is important for achieving optimum performance. It has been observed that using a single task executor for both these applications would bring inferior performance 79 .…”

Section: Task Executionmentioning

confidence: 99%

Twister2: Design of a big data toolkit

Kamburugamuve

Govindarajan

Wickramasinghe

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Data-driven applications are essential to handle the ever-increasing volume, velocity, and veracity of data generated by sources such as the Web and Internet of Things (IoT) devices. Simultaneously, an event-driven computational paradigm is emerging as the core of modern systems designed for database queries, data analytics, and on-demand applications. Modern big data processing runtimes and asynchronous many task (AMT) systems from high performance computing (HPC) community have adopted dataflow event-driven model. The services are increasingly moving to an event-driven model in the form of Function as a Service (FaaS) to compose services. An event-driven runtime designed for data processing consists of well-understood components such as communication, scheduling, and fault tolerance. Different design choices adopted by these components determine the type of applications a system can support efficiently. We find that modern systems are limited to specific sets of applications because they have been designed with fixed choices that cannot be changed easily. In this paper, we present a loosely coupled component-based design of a big data toolkit where each component can have different implementations to support various applications. Such a polymorphic design would allow services and data analytics to be integrated seamlessly and expand from edge to cloud to HPC environments.

show abstract

Heterogeneous Job Allocation Scheduler for Hadoop MapReduce Using Dynamic Grouping Integrated Neighboring Search

Cited by 31 publications

References 31 publications

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

Handling Non-Local Executions to Improve MapReduce Performance Using Ant Colony Optimization

Twister2: Design of a big data toolkit

Contact Info

Product

Resources

About