Network-aware scheduling of mapreduce framework ondistributed clusters over high speed networks

Kondikoppa, Praveenkumar; Chiu, Chui-Hui; Xue, Lin; Park, Seung-Jong

doi:10.1145/2378975.2378985

Cited by 22 publications

(9 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A number of task schedulers have been developed for the Map-Reduce framework including performance-driven schedulers that dynamically adjust resource allocation to maximize each job's chance of meeting its runtime performance goal (46), network-aware schedulers that try to reduce network traffic by enforcing local computations (47), and resource-aware schedulers that try to improve resource utilization across machines (48). The scheduler in PARAMO differs from these in that the prioritization scheme is based on higher level application-specific objective functions that can be specified by the user instead of lower level infrastructure and application independent metrics.…”

Section: Methodsmentioning

confidence: 99%

PARAMO: A PARAllel predictive MOdeling platform for healthcare analytic research using electronic health records

Ng¹,

Ghoting²,

Steinhubl

et al. 2014

Journal of Biomedical Informatics

View full text Add to dashboard Cite

Objective Healthcare analytics research increasingly involves the construction of predictive models for disease targets across varying patient cohorts using electronic health records (EHRs). To facilitate this process, it is critical to support a pipeline of tasks: 1) cohort construction, 2) feature construction, 3) cross-validation, 4) feature selection, and 5) classification. To develop an appropriate model, it is necessary to compare and refine models derived from a diversity of cohorts, patient-specific features, and statistical frameworks. The goal of this work is to develop and evaluate a predictive modeling platform that can be used to simplify and expedite this process for health data. Methods To support this goal, we developed a PARAllel predictive MOdeling (PARAMO) platform which 1) constructs a dependency graph of tasks from specifications of predictive modeling pipelines, 2) schedules the tasks in a topological ordering of the graph, and 3) executes those tasks in parallel. We implemented this platform using Map-Reduce to enable independent tasks to run in parallel in a cluster computing environment. Different task scheduling preferences are also supported. Results We assess the performance of PARAMO on various workloads using three datasets derived from the EHR systems in place at Geisinger Health System and Vanderbilt University Medical Center and an anonymous longitudinal claims database. We demonstrate significant gains in computational efficiency against a standard approach. In particular, PARAMO can build 800 different models on a 300,000 patient data set in 3 hours in parallel compared to 9 days if running sequentially. Conclusion This work demonstrates that an efficient parallel predictive modeling platform can be developed for EHR data. This platform can facilitate large-scale modeling endeavors and speed-up the research workflow and reuse of health information. This platform is only a first step and provides the foundation for our ultimate goal of building analytic pipelines that are specialized for health data researchers.

show abstract

Section: Methodsmentioning

confidence: 99%

PARAMO: A PARAllel predictive MOdeling platform for healthcare analytic research using electronic health records

Ng¹,

Ghoting²,

Steinhubl

et al. 2014

Journal of Biomedical Informatics

View full text Add to dashboard Cite

show abstract

“…It increases performance by a factor of 2 on 200 VMs on EC2. Similarly, there are several approaches focused on enhancing the task scheduler of the Hadoop framework to reduce power consumption or network cost . Kim et al defined an intercloud as a federated environment of public cloud and private clusters and proposed a task scheduler to enhance performance.…”

Section: Related Workmentioning

confidence: 99%

Enhancing network I/o performance for a virtualized Hadoop cluster

Jeong

Choi

2016

Concurrency and Computation

View full text Add to dashboard Cite

Summary A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware‐based single root I/O virtualization network architecture.

show abstract

“…In [23], the authors propose the method to make Hadoop scheduler aware of network topology is to extend the rack aware feature of the existing Hadoop scheduler to provide one more level of caching. An administrator controlled script will hold the information about which cluster the TaskTracker is associated with.…”

Section: Network Aware Schedulingmentioning

confidence: 99%

A Comprehensive View of MapReduce Aware Scheduling Algorithms in Cloud Environments

Yazdanpanah¹,

Shouraki²,

Abshirini³

2015

IJCA

View full text Add to dashboard Cite

Cloud computing has emerged as a model that harnesses massive capacities of data centers to host services in a costeffective manner. MapReduce has been widely used as a Big Data processing platform, proposed by Google in 2004 and has become a popular parallel computing framework for large-scale data processing since then. It is best suited for embarrassingly parallel and data-intensive tasks. It is designed to read large amount of data stored in a distributed file system such as Google File System (GFS), process the data in parallel, aggregate and store the results back to the distributed file system. Scheduling is one of the most critical aspects of MapReduce. Also three important scheduling issues in MapReduce such as locality, synchronization and fairness exist. This paper tries to illustrate and analyze the overview of thirteen different aware scheduling algorithms with different techniques and approaches for MapReduce in Hadoop and their scheduling issues and problems. At the end, Advantages and disadvantages of these algorithms are identified.

show abstract

Network-aware scheduling of mapreduce framework ondistributed clusters over high speed networks

Cited by 22 publications

References 3 publications

PARAMO: A PARAllel predictive MOdeling platform for healthcare analytic research using electronic health records

PARAMO: A PARAllel predictive MOdeling platform for healthcare analytic research using electronic health records

Enhancing network I/o performance for a virtualized Hadoop cluster

A Comprehensive View of MapReduce Aware Scheduling Algorithms in Cloud Environments

Contact Info

Product

Resources

About