Analysis of Workload Behavior in Scientific and Historical Long-Term Data Repositories

Abstract

The scope of archival systems is expanding beyond cheap tertiary storage: scientific and medical data is increasingly digital, and the public has a growing desire to digitally record their personal histories. Driven by the increased cost efficiency of hard drives compared to tape, and the rise of the Internet, content archives have become a means of providing the public with fast, cheap access to long-term data. Unfortunately, designers of purpose-built archival systems are either forced to rely on workload behavior obtained from a narrow, anachronistic view of archives as simply cheap tertiary storage, or extrapolate from marginally related enterprise workload data and traditional library access patterns.To provide relevant input for the design of effective long-term data storage systems, we examined the workload behavior of several scientific and historical archives, covering a mixture of purposes, media types, and access models. Our findings show that, for scientific archival storage, files have become larger, but update rates have remained largely unchanged. However, in public content archives, we observed behavior that diverges from the traditional "write-once, read-maybe" behavior of tertiary storage. Our study shows that the majority of such data is modified relatively frequently, and that indexing services such as Google and internal data management processes may routinely access large portions of an archive, accounting for most of the accesses. Based on these observations, we identify areas for improving the efficiency and performance of archival storage systems.

Can We Group Storage? Statistical Techniques to Identify Predictive Groupings in Storage System Accesses

“…Our file-archive dataset is a database of vital records from the Washington state digital archives, where records are labeled with one of many type identifiers (e.g., "Birth Records," "Marriage Records") [Adams et al 2012]. We examined 5,321,692 accesses from 2007 through 2010 that were made to a 16.5TB database.…”

“…Categorical grouping by definition requires some functional knowledge of the data that relies on human curation, either in manual labeling or metadata upkeep [Adams et al 2012]. Example categorical attributes include size, name, type, owner, path, or even whole file content.…”

“…Both metadata and domain knowledge require upkeep by a local expert in the system workload. Over time, the logging methodology can change, leading to inconsistent interpretations for metadata fields [Adams et al 2012]. Also, if administrators leave, their terminology and understanding of the system must be accurately transferred to their successor.…”

“…After well-documented cases of failed or insufficient data anonymization, such as the infamous AOL data leak [Barbaro and Jr. 2006], companies are very aggressively defending internal privacy to the detriment of well-meaning researchers. This concern is part of the reason much modern research in predictive grouping uses data that is 5 to 10 years out of date, if they use real data at all [Adams et al 2012].…”

Usage behavior of a large-scale scientific archive

“…Since 1993, there have been only two studies that have looked explicitly at day-to-day activities on a large-scale scientific archive, each limited in their application to modern scientific archive design. Adams et al were stymied by the coarse granularity of their data, preventing them from examining user and file level behaviors [1]. Frank et al focused on evolutionary trends over time rather than a detailed analysis of a modern system's behavior [2].…”