Towards A Better Understanding of Workload Dynamics on Data-Intensive Clusters and Grids

Abstract

This paper presents a comprehensive statistical analysis of workloads collected on data-intensive clusters and Grids. The analysis is conducted at different levels, including Virtual Organization (VO) and user behavior. The aggregation procedure and scaling analysis are applied to job arrival processes, leading to the identification of several basic patterns, namely, pseudo-periodicity, long range dependence (LRD), and (multi)fractals. It is shown that statistical measures based on interarrivals are of limited usefulness and count based measures should be trusted instead when it comes to correlations. We also study workload characteristics like job run time, memory consumption, and cross correlations between these characteristics. A "bag-of-tasks" behavior is empirically proved, strongly indicating temporal locality. We argue that pseudo-periodicity, LRD, and "bag-of-tasks" behavior are important workload properties on data-intensive clusters and Grids, which are not present in traditional parallel workloads. This study has important implications on workload modeling and performance predictions in data-intensive Grid environments.

Model-Driven Simulation of Grid Scheduling Strategies

“…However, it is not as flexible as models in case that many traces have to be generated to enable a Grid scheduling study. The traces available from parallel workloads can also have significantly different characteristics compared to Grid workloads, which has been empirically proved [12]. Such differences, in turn, may lead to considerably different performance evaluation results.…”

“…These traces or models, however, exhibit significantly different characteristics than the traces on production Grids. As has been studied and reported in [12], pseudo-periodicity, long range dependence (LRD), and "bag-of-tasks" behavior with strong temporal locality are the main properties that characterize production Grid workloads. Therefore, it is important that representative models be developed to capture the salient properties of Grid workloads.…”

“…Job arrival processes exhibit long range dependence at many levels, including VO, cluster, and the Grid [12]. LRD is closely related to temporal burstiness, in which jobs tend to arrive within bursty periods.…”

“…A study of workload dynamics on clusters and Grids has been conducted in [12]. It is shown that workload characteristics on clusters and Grids, particularly in data-intensive environments, are significantly different than those on conven-tional supercomputers.…”

A Realistic Integrated Model of Parallel System Workloads

“…To overcome this difficulty, many researchers decide to use randomly generated workloads in their work. These workloads are usually unrealistic because several statistical studies [7], [8], [13], [14], [15], [31] have shown that the characteristics of parallel system workloads 1 are far from independently and identically distributed. Instead, they have several important and correlated characteristics such as long range dependence, burstiness, and bag-of-tasks (BoT) behaviour.…”