Evaluating guidelines for reporting empirical software engineering studies

Abstract

Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. The reviews detected 44 issues where the guidelines would benefit from amendment or clarification and 8 defects. Reporting guidelines need to specify what information goes into what section and avoid excessive duplication. The current guidelines need to be revised and then subjected to further theoretical and empirical validation. Perspective-based checklists are a useful validation method but the practitioner/consultant perspective presents difficulties

Reporting Empirical Research in Open Source Software: The State of Practice

“…The Software Engineering (SE) community has been emphasising the need for guidelines for reporting empirical research [5,[12][13][14]. Such guidelines allow for a systematic and standardised way of presenting empirical findings.…”

Guidelines for conducting and reporting case study research in software engineering

“…Guidelines for reporting experiments have been proposed by Jedlitschka and Pfahl (2005) and evaluated by Kitchenham et al (2008). Their work aims at defining a standardized reporting of experiments that enables cross-study comparisons through e.g.…”

“…The high level structure for reporting experiments in software engineering proposed by Jedlitschka and Pfahl (2005) therefore also fits the purpose of case study reporting. However, some changes are needed, based on specific characteristics of case studies and other issues based on an evaluation conducted by Kitchenham et al (2008). The resulting structure is presented in Table 9.…”

“…it reports the planning of the case study including the measures taken to ensure the validity of the study. Table 9 Proposed reporting structure by Jedlitschka and Pfahl (2005) and modification proposed by Kitchenham et al (2008) Since the case study is of flexible design, and data collection and analysis are more intertwined, these sections may be combined into one. Consequently, the contents at the lower level must be adjusted, as proposed in Table 9.…”

“…by Seaman (1999). There exist guidelines for experiments' conduct (Kitchenham et al 2002;Wohlin et al 2000) and reporting (Jedlitschka and Pfahl 2005), measurements (Basili and Weiss 1984;Fenton and Pfleeger 1996;van Solingen and Berghout 1999), and systematic reviews (Kitchenham 2007), while only little is written on case studies in software engineering (Höst and Runeson 2007;Kitchenham et al 1995;Wohlin et al 2003) and qualitative methods (Dittrich 2007;Seaman 1999;Sim et al 2001). Recently, a comprehensive view of empirical research issues for software engineering has been presented, edited by .…”