DataMill

Abstract

Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science?This paper introduces DataMill, a community-based easyto-use services-oriented open benchmarking infrastructure for performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors. Multiple research groups already participate in DataMill.DataMill is also of interest for research on performance evaluation. The infrastructure supports quantifying the effect of hidden factors, disseminating the research results beyond mere reporting. It provides a platform for investigating interactions and composition of hidden factors.

DataMill: a distributed heterogeneous infrastructure forrobust experimentation

“…The DataMill infrastructure is composed of a master node , responsible for the distribution of experiment trials and the collection of results, and several worker nodes , which execute the experiment packages provided by the users. The original design and implementation of DataMill is reported in , but a great deal of the architecture has changed in response to several challenges that we have encountered through heavy use of the infrastructure. This section includes the highlights of the changes since the previous iteration of DataMill.…”

“…This is because with each additional factor and factor level, the size of the experiment space increases combinatorially. DataMill attempts to and minimize the length of user's experiments by using an optimization solver (see for implementation details) to reduce the set of trials down to what is necessary for estimating the factor effect sizes of interest to the user, as in typical factorial experiment design .…”

“…Several new factors are supported by DataMill since the original report on the infrastructure . New factors in DataMill version 2.0 include support for altering the underlying filesystem type, several compiler wrapper enhancements, support for the autogroup kernel parameter, support for CPU frequency scaling, and support for altering system time.…”

“…To improve the current state of experimentation, we proposed DataMill . DataMill is a distributed infrastructure for computer performance experimentation targeted at scientists, researchers, and aspiring researchers.…”

“…In order to illustrate the state of experimental methodology in computer science, we conducted a survey of the recent publications to SOSP 2011, ASPLOS 2012, OSDI 2012, ATC 2012, EuroSys 2011, and PLDI 2012 in . We have updated the survey with more recent conferences, including HiPC 2013, and ASPLOS 2014; the results are shown in Table .…”