An Analysis of Traces from a Production MapReduce Cluster

Kavulya, Soila; Tan, Jiaqi; Gandhi, Rajeev; Narasimhan, Priya

doi:10.1109/ccgrid.2010.112

Cited by 255 publications

(185 citation statements)

References 15 publications

Supporting

Mentioning

178

Contrasting

Unclassified

Order By: Relevance

“…The trace analyzed in [18] shows 9% jobs are reducemostly. In this test, we experimented with two reduce-mostly applications: ts-reduce and matrix multiplication.…”

Section: Results For Reduce-mostly Jobsmentioning

confidence: 99%

“…According to the study in [18], a large portion of MapReduce jobs (over 71%) are map-only. In this section, we ran two map-mostly applications: ts-map and PSA-SWG.…”

Section: A Results For Map-mostly Compute-intensive Workloadmentioning

confidence: 99%

“…Usually, the utilization of real data centers is low. For example, according to the real traces of production clusters, CPU utilization was 5%-10% in Yahoo's M45 cluster [18] and below 50% mostly in a multithousand node Google cluster [19]. The low utilization may be caused by several factors.…”

Section: A Resource Stealing (Rs)mentioning

confidence: 99%

See 2 more Smart Citations

Improving Resource Utilization in MapReduce

Guo

Fox

Zhou

et al. 2012

2012 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Abstract-MapReduce has been adopted widely in both academia and industry to run large-scale data parallel applications. In MapReduce, each slave node hosts a number of task slots to which tasks can be assigned. So they limit the maximum number of tasks that can execute concurrently on each node. When all task slots of a node are not used, the resources "reserved" for idle slots are unutilized. To improve resource utilization, we propose resource stealing to enable running tasks to steal resources reserved for idle slots and give them back proportionally whenever new tasks are assigned. Resource stealing makes the otherwise wasted resources get fully utilized without interfering with normal job scheduling. MapReduce uses speculative execution to improve fault tolerance. Current Hadoop implementation decides whether to run speculative tasks based on the progress rates of running tasks, which does not take into consideration the absolute progress of each task. We propose Benefit Aware Speculative Execution which evaluates the potential benefit of speculative tasks and eliminates unnecessary runs. We implement the proposed algorithms in Hadoop, and our experiments show that our algorithms can significantly shorten job execution time and reduce the number of non-beneficial speculative tasks.

show abstract

“…The trace analyzed in [18] shows 9% jobs are reducemostly. In this test, we experimented with two reduce-mostly applications: ts-reduce and matrix multiplication.…”

Section: Results For Reduce-mostly Jobsmentioning

confidence: 99%

“…According to the study in [18], a large portion of MapReduce jobs (over 71%) are map-only. In this section, we ran two map-mostly applications: ts-map and PSA-SWG.…”

Section: A Results For Map-mostly Compute-intensive Workloadmentioning

confidence: 99%

See 1 more Smart Citation

Improving Resource Utilization in MapReduce

Guo

Fox

Zhou

et al. 2012

2012 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

show abstract

“…Очевидно, редко используемые контейнеры можно группировать плотнее и изолировать их от более активных контейнеров, тем самым не вызывая заметных пользователям замедлений в работе приложений. Использование данных о поведении пользователей в распределенных вычислительных системах неоднократно рассматривалось разными авторами, например, для оптимизации потребления энергии [6], для задачи размещения данных [5], оптимизации расписаний в Grid [7], снижения латентности тонких клиентов [8], запуска виртуальных машин [4] и др. Во всех работах подтверждается периодичность, характерная для создаваемой пользователями нагрузки.…”

Section: управление вычислительными ресурсами в системе Unihubunclassified

Load balancing in Unihub SaaS system based on user behavior prediction

Грушин

Kuzyurin

2015

Proceedings of ISP RAS

View full text Add to dashboard Cite

“…In fact, for common MapReduce jobs, most of the time is spent in the map phase and shuffle phases, e.g., according to [22,40], only around 7% of the workload in a production MapReduce cluster are reduce heavy jobs.…”

Section: Introductionmentioning

confidence: 99%

Joint optimization of overlapping phases in MapReduce

Lin

Zhang

Wierman

et al. 2013

Performance Evaluation

View full text Add to dashboard Cite

MapReduce is a scalable parallel computing framework for big data processing. It exhibits multiple processing phases, and thus an efficient job scheduling mechanism is crucial for ensuring efficient resource utilization. This paper studies the scheduling challenge that results from the overlapping of the "map" and "shuffle" phases in MapReduce. We propose a new, general model for this scheduling problem, and validate this model using cluster experiments. Further, we prove that scheduling to minimize average response time in this model is strongly NP-hard in the offline case and that no online algorithm can be constant-competitive. However, we provide two online algorithms that match the performance of the offline optimal when given a slightly faster service rate (i.e., in the resource augmentation framework). Finally, we validate the algorithms using a workload trace from a Google cluster and show that the algorithms are near optimal in practical settings.

show abstract

An Analysis of Traces from a Production MapReduce Cluster

Cited by 255 publications

References 15 publications

Improving Resource Utilization in MapReduce

Improving Resource Utilization in MapReduce

Load balancing in Unihub SaaS system based on user behavior prediction

Joint optimization of overlapping phases in MapReduce

Contact Info

Product

Resources

About