A Multi-Site Joint Replication of a Design Patterns Experiment Using Moderator Variables to Generalize across Contexts

Abstract

Context. Several empirical studies have explored the benefits of software design patterns, but their collective results are highly inconsistent. Resolving the inconsistencies requires investigating moderators-i.e., variables that cause an effect to differ across contexts. Objectives. Replicate a design patterns experiment at multiple sites and identify sufficient moderators to generalize the results across prior studies. Methods. We perform a close replication of an experiment investigating the impact (in terms of time and quality) of design patterns (Decorator and Abstract Factory) on software maintenance. The experiment was replicated once previously, with divergent results. We execute our replication at four universities-spanning two continents and three countries-using a new method for performing distributed replications based on closely coordinated, small-scale instances ("joint replication"). We perform two analyses: 1) a post-hoc analysis of moderators, based on frequentist and Bayesian statistics; 2) an a priori analysis of the original hypotheses, based on frequentist statistics. Results. The main effect differs across the previous instances of the experiment and across the sites in our distributed replication. Our analysis of moderators (including developer experience and pattern knowledge) resolves the differences sufficiently to allow for cross-context (and cross-study) conclusions. The final conclusions represent 126 participants from five universities and twelve software companies, spanning two continents and at least four countries. Conclusions. The Decorator pattern is found to be preferable to a simpler solution during maintenance, as long as the developer has at least some prior knowledge of the pattern. For Abstract Factory, the simpler solution is found to be mostly equivalent to the pattern solution. Abstract Factory is shown to require a higher level of knowledge and/or experience than Decorator for the pattern to be beneficial.

Four commentaries on the use of students and professionals in empirical software engineering experiments

“…In another experiment, only the best performing developers benefitted from a "proper" object-oriented control style, whereas the others did not (Arisholm and Sjøberg 2004). Also, in a study by Krein et al (2016), the purported benefit of the investigated design patterns was mostly negative for the students but mostly positive for the most experienced and knowledgeable professionals. Such experiments show that one technology (method, technique, tool, etc.)…”

A Procedure and Guidelines for Analyzing Groups of Software Engineering Replications

“…On the contrary, if the replications have identical response variable operationalizations and access to the raw data is guaranteed-as is typically the case in groups of SE replications [17]-, it may be possible to apply IPD-S [13]- [15] and, thus, interpret results in natural units. This practice has already been applied in SE [26], [75], [76] and can lead to more informative joint conclusions.…”

“…• Use the following four-step procedure to analyze a stereotypical group of SE replications: (1) describe the characteristics of the participants using appropriate descriptive statistics and visualizations; (2) use consistent statistical techniques to pre-process, describe and analyze the data of each replication; (3) select suitable aggregation techniques to provide 1. Variables that cause an effect to differ across contexts [26]. joint conclusions; and (4) conduct exploratory analyses to identify experiment-level moderators 2 and participant-level moderators 3 .…”

A Cloud Storage Monitoring System using Deduplication and File Access Pattern

“…Cost related to cloud storage services based on the amount of storage capacity which was consumed by users and the number of IOPs accomplished with respect of the amount of consumed bandwidth. It may reduce cloud storage costs through optimization on de-duplication and also taking advantage of pay-as-you-grow options on choosing the respective cloud storage service provider [12,13]. It meets all other challenge on the principles.…”