A (Updated) Review of Empiricism at the SIGCSE Technical Symposium

Abstract

The computer science education (CSEd) research community consists of a large group of passionate CS educators who often contribute to other disciplines of CS research. There has been a trend in other disciplines toward more rigorous and empirical evaluation of various hypotheses. Prior investigations of the thencurrent state of CSEd research showed a distinct lack of rigor in the top research publication venues, with most papers falling in the general category of experience reports. In this paper, we present our examination of the two most recent proceedings of the SIGCSE Technical Symposium, providing a snapshot of the current state of empiricism at the largest CSEd venue. Our goal to categorize the current state of empiricism in the SIGCSE Technical Symposium and identify where the community might benefit from increased empiricism when conducting CSEd research. We found an increase in empirical validation of CSEd research to over 70%; however, our findings suggest that current CSEd research minimizes replication precluding meta-analysis and theory building.

Methodological Rigor and Theoretical Foundations of CS Education Research

“…The comparison with Randolph et al's results shows that significantly fewer articles in our sample presented anecdotal or attitudes only results [16]. This could be the result of our limited breadth of analysis relative to Randolph et al, but results reported in Al-Zubidy, comparing their categorization SIGCSE papers to earlier reviews of SIGCSE papers, support the conclusion that CS education research is moving away from superficial results and more towards empirical results [1]. However, the results on the 'positive' methodological indicators (experimental and qualitative), as well as the 'negative' indicator of single group posttest only designs, show no change from Randolph et al's analysis.…”

“…Lastly, we compared our results to those described in Al-Zubidy et al [1], which categorized articles from the proceedings of SIGCSE with respect to whether or not they reported on an empirical study. Al-Zubidy et al's analysis used the same definition of experimental as used by Randolph et al [16], so the comparison between our results and theirs are shown as well.…”

“…The investigation of lack of empirical rigor has shown a consistently improving trend. A review of SIGCSE studies from 1984-2003 found that 21% of studies were 'experimental' (their definition of this term is explained further below) [21], a similar review of studies from multiple CS education research venues found 40% to be experimental [16], whereas a later review of SIGCSE research from 2014-2015 found over 70% of studies to be empirical [1]. This increasing trend is heartening to those concerned about rigor in CS education research, but it is not without caveats.…”

“…In the most recent review, although 70% of studies were classified as empirical, the quality of such empirical evidence was questionable. Of the 70% of studies classified as empirical, the most common source of data was student feedback surveys [1].…”

“…The rigor of CS education research has been previously examined on a number of dimensions. The degree to which research relies on empirical results [1], the methodological rigor of those empirical studies [16], the diversity of research methods employed [2], and the degree to which relevant theory from the behavioral sciences and other education fields is used to ground and guide research [6]. With respect to the last of these dimensions, Mark Guzdial made the statement at a 2005 SIGCSE panel that "Too much of the research in computing education ignores the hundreds of years of education, cognitive science, and learning sciences research that have gone before us" [2].…”