Wake up and smell the coffee

Abstract

Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate experimental design, and rigorous analysis. Unfortunately, methodology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and Ruby in the past decade and the imminent rise of commodity multicore architectures for the next decade pose new methodological challenges that are not yet widely understood. This paper explores the consequences of our collective inattention to methodology on innovation, makes recommendations for addressing this problem in one domain, and provides guidelines for other domains. We describe benchmark suite design, experimental design, and analysis for evaluating Java applications. For example, we introduce new criteria for measuring and selecting diverse applications for a benchmark suite. We show that the complexity and nondeterminism of the Java runtime system make experimental design a first-order consideration, and we recommend mechanisms for addressing complexity and nondeterminism. Drawing on these results, we suggest how to adapt methodology more broadly. To continue to deliver innovations, our field needs to significantly increase participation in and funding for developing sound methodological foundations.

Concept analysis in programming language research: done well it is all right

“…Here, the reality of interest is the software reality, and knowledge is generated by the means of examining or running programs. The most prominent method here is computational experiments-the study of algorithms by exposing their implementations to a wide variety of automatically generated stimuli and measuring the effort expended by the implementation as a function of stimulus parameters [6,28,29,37,48,67,72]; it is relevant to programming language research mostly in the study of implementation techniques.…”

Fast conservative garbage collection

“…We then measure and report the subsequent iteration. This methodology greatly reduces non-determinism due to the adaptive optimizing compiler and improves underlying performance by about 5% compared to the prior replay methodology [14]. We run each benchmark 20 times (20 invocations) and in Table 4 we report the average and 95% confidence intervals using Student's t-distribution.…”

Multitudes of Objects: First Implementation and Case Study for Java.

“…13, no. 5,2014 which differs only in the multiplicity, is roughly equivalent to Account account; meaning that account can hold either no or one account (see Section 3.4 for the important difference). Note that using multiplicities, both account and accounts have the same type Account; they differ only in their declared multiplicities.…”

“…To measure on steady state, we used the multi-iteration determinism method for benchmarking from Blackburn et al [5], which includes the following steps: JUnit 4.0 255 † excluding 10 that we had to remove because they could not be compiled with JUnit 4.0 § 25 of these tests fail both with and without multiplicities because they should normally be run with JUnit 4.7 (see text) $ excluding 2 that were removed because they contain an infinite loop (see text) Table 3 -Subject programs used in the evaluation.…”