Metrics and Methods Used To Compare Student Performance Data in Chemistry Education Research Articles

Abstract

Quasi-experiments are common in studies that estimate the effect of instructional interventions on student performance outcomes. In this type of research, the nature of the experimental design, the choice in assessment, the selection of comparison groups, and the statistical methods used to analyze the comparison data dictate the validity of causal inferences. Therefore, gathering and reporting validity evidence in causal studies is of utmost importance, especially when conclusions have real policy implications for students and faculty, among other stakeholders. This review examines 24 articles that reported quantitative investigations of the effect of instructional interventions on performance-based outcomes conducted within undergraduate chemistry courses. Specifically, we examined four aspects of conducting such evaluations, including: (1) the type of quasi-experimental design used to study the relationship between interventions, students, outcomes, and settings; (2) the metrics used to measure performance outcomes; (3) the type of groups used to contrast outcomes across experimental conditions; and (4) the statistical methods used to analyze the comparison data. Through the examination of these four aspects of causal studies, together with a validity typology for quasi-experimental designs, we catalogued the metrics and methods used to compare student performance outcomes across varied instructional contexts. Recommendations for researchers and practitioners planning quasi-experimental investigations and for interpreting results from causal studies in chemistry education are provided.

Assessing Affective Differences between a Virtual General Chemistry Experiment and a Similar Hands-On Experiment

“…Irby and colleagues focused more narrowly within a hybrid curriculum to better understand if students who completed an electrical conductivity experiment in a virtual environment engaged with the chemistry triplet, 24 which models the different levels of understanding in chemistry, submicroscopic, macro- scopic, and symbolic, to a different degree than students who completed a similar hands-on experiment. 25 Their study utilized a pretest−posttest alternative treatment with a control group study design 26 where there was a control section that did not use the hybrid curriculum and two treatment sections that did. The two treatment sections were staggered so that, in any given week, one section was completing a hands-on experiment while the other section was completing a virtual experiment on a different topic.…”

Evaluating the evidence base for evidence‐based instructional practices in chemistry through meta‐analysis

“…Additionally, the nature of a meta‐analytical approach requires combining evidence bases that arise from similar research designs. This investigation focuses on quasi‐experimental and experimental comparisons given their frequency in the research literature (Mack, Hensen, & Barbera, ). Other investigative approaches such as qualitative investigations into the quality of students' written responses or quantitative measures of growth over time, generate compelling evidence in support of instructional practices but cannot be synthesized with a corpus of studies enacting comparative designs.…”

A Review of Research on Process Oriented Guided Inquiry Learning: Implications for Research and Practice

“…To answer this guiding question, we focused on the experimental designs and data collected across the studies in our sample. In order to provide an overview of the studies, we utilized the conceptual framework outlined by Mack, Hensen, and Barbera, 113 which provides operationalized definitions of different types of quasiexperimental study designs; Table 1 provides descriptions of the design types observed in the sample. Since Mack et al 113 described only quantitative studies, we added additional categories (interview, classroom observation, open-ended survey) in order to classify qualitative studies in terms of the type of data collected.…”

“…In order to provide an overview of the studies, we utilized the conceptual framework outlined by Mack, Hensen, and Barbera, 113 which provides operationalized definitions of different types of quasiexperimental study designs; Table 1 provides descriptions of the design types observed in the sample. Since Mack et al 113 described only quantitative studies, we added additional categories (interview, classroom observation, open-ended survey) in order to classify qualitative studies in terms of the type of data collected. In addition, we used the category other for studies that did not fit in any of the qualitative or quantitative categories; for example, the development of an instrument to measure process skills did not fit neatly into either category, because it did not report student data.…”

“…39 In addition, some studies in our sample used university-developed course evaluation surveys to assess students' perceptions of courses incorporating POGIL; these studies highlighted Table 1. Experimental Designs Used in Quantitative Studies (Adapted from Mack et al 113 ) One Group Two Group (Treatment and Control) Post-test A group receives a treatment and is then assessed on the outcome measure. One group experiences a treatment, and a second group experiences an alternative treatment (or the absence of that treatment).…”