The Statistical Properties of Host Load

Abstract

Understanding how host load changes over time is instrumental in predicting the execution time of tasks or jobs, such as in dynamic load balancing and distributed soft real‐time systems. To improve this understanding, we collected week‐long, 1 Hz resolution traces of the Digital Unix 5 second exponential load average on over 35 different machines including production and research cluster machines, compute servers, and desktop workstations. Separate sets of traces were collected at two different times of the year. The traces capture all of the dynamic load information available to user‐level programs on these machines. We present a detailed statistical analysis of these traces here, including summary statistics, distributions, and time series analysis results. Two significant new results are that load is self‐similar and that it displays epochal behavior. All of the traces exhibit a high degree of self‐similarity with Hurst parameters ranging from 0.73 to 0.99, strongly biased toward the top of that range. The traces also display epochal behavior in that the local frequency content of the load signal remains quite stable for long periods of time (150–450 s mean) and changes abruptly at epoch boundaries. Despite these complex behaviors, we have found that relatively simple linear models are sufficient for short‐range host load prediction.

Developing an Efficient Pattern Discovery Method for CPU Utilizations of Computers

“…In [10], Dinda analysed several statistical properties of host load data. In [7], Di et al also analysed some statistical properties of CPU host load data and demonstrated their probability distribution.…”

Towards Autonomic Virtual Applications in the In-VIGO System

“…[5][6] [7] and [21]). In our case, one very important factor that must be considered in choosing the prediction method is its overhead since the jobs of interest are relatively short lived.…”

“…For CPU-intensive jobs, the goal is to find the resource on which the given job can run fast. However, runtime does not only depend on the machine's CPU speed, but also is strongly related to the machine's load [7] [8]. Therefore, the following equations are used to estimate the job's execution on each machine by assuming that the current load of the machine does not change in a relatively short period of time.…”

“…Topics on resource-usage prediction have been examined by [6] [7][8] [9]. Most of these approahches rely on the use of past-execution-time knowledge.…”

An evaluation of linear models for host load prediction

“…In this paper, we describe and use the August 1997 set. A more detailed description of our analysis can be found in [8,9] and the traces themselves can be found on the web. 1 All of the hosts in the August 1997 set were DEC Alpha DUX machines, and they form four classes: Production Cluster, 13 hosts of the PSC's "Supercluster", including two front-end machines (axpfea, axpfeb), four interactive machines (axp0 through axp3), and seven batch machines); Research Cluster, eight machines in an experimental cluster in our lab (manchester-1 through manchester-8)); Compute servers, two high performance large memory machines used by our group as compute servers for simulations and the like (mojave and sahara); and Desktops, 15 desktop workstations owned by members of our research group (aphrodite through zeno).…”