Missing data and bias in physics education research: A case for using multiple imputation

Nissen, Jayson; Donatello, Robin A.; Dusen, Ben Van

doi:10.1103/physrevphyseducres.15.020106

Cited by 58 publications

(50 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The size of the dataset after each step in the filtering process is shown in Table . After filtering, 58% of the students had matched pretest and posttest scores, which fell in the range of typical participation rates in the literature (Nissen, Donatello, & Van Dusen, in press).…”

Section: Methodsmentioning

confidence: 86%

Equity in college physics student learning: A critical quantitative intersectionality investigation

Dusen

Nissen

2019

J Res Sci Teach

Self Cite

View full text Add to dashboard Cite

We investigated the intersectional nature of race/racism and gender/sexism in broad scale inequities in physics student learning using a critical quantitative intersectionality. To provide transparency and create a nuanced picture of learning, we problematized the measurement of equity by using two competing operationalizations of equity: Equity of Individuality and Equality of Learning. These two models led to conflicting conclusions. The analyses used hierarchical linear models to examine student's conceptual learning as measured by gains in scores on research-based assessments administered as pretests and posttests. The data came from the Learning About STEM Student Outcomes' (LASSO) national database and included data from 13,857 students in 187 firstsemester college physics courses. Findings showed differences in student gains across gender and race. Large gender differences existed for White and Hispanic students but not for Asian, Black, and Pacific Islander students. The models predicted larger gains for students in collaborative learning than in lecture-based courses. The Equity of Individuality operationalization indicated that collaborative instruction improved equity because all groups learned more with collaborative learning. The Equality of Learning operationalization indicated that collaborative instruction did not improve equity because differences between groups were unaffected. We discuss the implications of these mixed findings and identify

show abstract

Section: Methodsmentioning

confidence: 86%

Equity in college physics student learning: A critical quantitative intersectionality investigation

Dusen

Nissen

2019

J Res Sci Teach

Self Cite

View full text Add to dashboard Cite

show abstract

“…The final dataset included data from 5959 students in 112 courses at 17 institutions with missing data for 15% of the pretests and 30% of the post-tests. This resulted in 55% of the responses having matched pretest and post-test which falls in the middle of the 30% to 80% range of matched data reported in the PER literature [37]. We calculated pretest and post-test scores using the total percentage correct of all the items on the assessment.…”

Section: Data Collection and Preparationmentioning

confidence: 98%

Modernizing use of regression models in physics education research: A review of hierarchical linear modeling

Dusen

Nissen

2019

Phys. Rev. Phys. Educ. Res.

Self Cite

View full text Add to dashboard Cite

This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] Physics education researchers (PER) often analyze student data with single-level regression models (e.g., linear and logistic regression). However, education datasets can have hierarchical structures, such as students nested within courses, that single-level models fail to account for. The improper use of single-level models to analyze hierarchical datasets can lead to biased findings. Hierarchical models (also known as multilevel models) account for this hierarchical nested structure in the data. In this publication, we outline the theoretical differences between how single-level and multilevel models handle hierarchical datasets. We then present analysis of a dataset from 112 introductory physics courses using both multiple linear regression and hierarchical linear modeling to illustrate the potential impact of using an inappropriate analytical method on PER findings and implications. Research can leverage multi-institutional datasets to improve the field's understanding of how to support student success in physics. There is no post hoc fix, however, if researchers use inappropriate single-level models to analyze multilevel datasets. To continue developing reliable and generalizable knowledge, PER should use hierarchical models when analyzing hierarchical datasets. The Supplemental Material includes a sample dataset, R code to model the building and analysis presented in the paper, and an HTML output from the R code.

show abstract

“…We used Multivariate Imputations by Chained Equa-tions (MICE) [61] with predictive mean matching (pmm) to impute the missing closed-response data. These methods have been discussed previously in [62,63] and so we do not elaborate on them here. For each respondent, we used the levels of the lab they were enrolled in (FY or BFY), the type of lab they were enrolled in (CTLabs or other), and the score on the closed-response survey they completed to estimate their missing score.…”

Section: Appendix A: Multiple Imputation Analysismentioning

confidence: 99%

“…MICE operates by imputing our dataset M times, creating M complete datasets, each containing data from 4084 students. Each of these M datasets will have somewhat different values for the imputed data [62,63]. If the calculation is not prohibitive, it has been recommended that M be set to the average percentage of missing data [63], which in our case is 27.…”

Section: Appendix A: Multiple Imputation Analysismentioning

confidence: 99%

“…The main limitation to imputing our data in this way stems from the reliability of the imputed values. As briefly mentioned above, we lack information about students' grades or their scores on other standardized assessments, which have been shown to be useful predictors of student scores on diagnostic assessments like the PLIC [62]. Without this information, the reliability of the imputed dataset is limited.…”

Section: Summary and Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking

Walsh

Quinn

Wieman

et al. 2019

Phys. Rev. Phys. Educ. Res.

View full text Add to dashboard Cite

Introductory physics lab instruction is undergoing a transformation, with increasing emphasis on developing experimentation and critical thinking skills. These changes present a need for standardized assessment instruments to determine the degree to which students develop these skills through instructional labs. In this article, we present the development and validation of the Physics Lab Inventory of Critical thinking (PLIC). We define critical thinking as the ability to use data and evidence to decide what to trust and what to do. The PLIC is a 10-question, closed-response assessment that probes student critical thinking skills in the context of physics experimentation. Using interviews and data from 5584 students at 29 institutions, we demonstrate, through qualitative and quantitative means, the validity and reliability of the instrument at measuring student critical thinking skills. This establishes a valuable new assessment instrument for instructional labs.

show abstract

Missing data and bias in physics education research: A case for using multiple imputation

Cited by 58 publications

References 30 publications

Equity in college physics student learning: A critical quantitative intersectionality investigation

Equity in college physics student learning: A critical quantitative intersectionality investigation

Modernizing use of regression models in physics education research: A review of hierarchical linear modeling

Quantifying critical thinking: Development and validation of the physics lab inventory of critical thinking

Contact Info

Product

Resources

About