Automatic modeling of file system workloads using two-level arrival processes

Abstract

This article describes a method for analyzing, modeling, and simulating a two-level arrivalcounting process. This method is particularly appropriate when the number of independent processes is large, as is the case in our motivating application which requires analyzing and representing computer file system trace data for activity on nearly 8,000 files. The method is also applicable to network trace data characterizing communication patterns between pairs of computers. We apply cluster analysis to separate the arrival process into groups or bursts of activity on a file. We then characterize the arrival process in terms of the time between bursts of activity on a file, the time between file events within bursts, and the number of events in a burst. Finally, we model these three components individually, then reassemble the results to produce a synthetic trace generator. In order to gauge the effectiveness of this method, we use synthetically generated (simulated) trace data produced in this way to drive a discrete-event simulation of a distributed replicated file system. We compare the results of the simulation driven by the synthetic trace with the same simulation driven by the original trace data, and conclude that the synthetic data capture the essential characteristics of the empirical trace.

On extracting session data from activity logs

“…This motivates the idea of adjusting the thresholds and using a customized value for each individual user. The idea of using user-specific thresholds has been proposed before by Murray et al [18] (and essentially the same idea was suggested even earlier by Ware et al in the context of file-access patterns [26]). Given data about a single user's intervals between queries, they suggested the following algorithm to find a threshold to distinguish between short and long intervals.…”

Workload Modeling for Computer Systems Performance Evaluation

“…The I/O operations on a file have also been shown to be bursty, and are well modeled by a two-level process [721]. The top level defines the arrivals of bursts and their durations.…”

“…A completely different approach is to use clustering, in which jobs that are adjacent to each other form clusters that represent sessions [721,74,508]. This technique also has the advantage of tailoring the threshold for each user separately, rather than assuming that a single threshold is suitable for all users [482].…”

Autonomic Storage System Based on Automatic Learning

“…With regard to application model, some works have been proposed in characterizations and modeling of I/O access pattern [9][10][11][12][13][14]. Most of these studies were made at the file system level.…”