Two-phase study examining perspectives and use of quantitative methods in physics education research

Abstract

While other fields such as statistics and education have examined various issues with quantitative work, few studies in physics education research (PER) have done so. We conducted a two-phase study to identify and to understand the extent of these issues in quantitative PER . During Phase 1, we conducted a focus group of three experts in this area, followed by six interviews. Subsequent interviews refined our plan. Both the focus group and interviews revealed issues regarding the lack of details in sample descriptions, lack of institutional/course contextual information, lack of reporting on limitation, and overgeneralization or overstatement of conclusions. During Phase 2, we examined 72 manuscripts that used four conceptual or attitudinal assessments (Force Concept Inventory, Conceptual Survey of Electricity and Magnetism, Brief Electricity and Magnetism Assessment, and Colorado Learning Attitudes about Science Survey). Manuscripts were coded on whether they featured various sample descriptions, institutional/course context information, limitations, and whether they overgeneralized conclusions. We also analyzed the data to see if reporting has changed from the earlier periods to more recent times. We found that not much has changed regarding sample descriptions and institutional/course context information, but reporting and overgeneralizing conclusions has improved over time. We offer some questions for researchers, reviewers, and readers in PER to consider when conducting or using quantitative work.

Predictive and explanatory models might miss informative features in educational data

“…Some examples include sample size, the number of features, the number of numerical features, the number of categorical features, and the percentage of observation of the majority class or outcome balance (Couronné et al, 2018). Just as there have been calls for increased reporting of demographics in the DBER and EDM communities to understand how results might depend on the sample population or generalize (Kanim and Cid, 2020;Paquette et al, 2020), we are calling for the same with the explanatory and predictive models we create, partially addressing some of the questions raised by Knaub et al (2019) in their analysis of physics education research quantitative work. By doing so, we hope for greater acknowledgement of possible sources of bias or false negatives in feature selection as a result of the data or algorithms used in DBER and EDM studies.…”

A Framework for Evaluating Statistical Models in Physics Education Research

“…Finally, communication of data context has also been observed to be an issue in PER. Knaub et al [32] found that approximately half of papers published in Physical Review: PER do not describe basic descriptive information about the students being studied such as their demographics, average grades, or institutional descriptors [33].…”

“…It also means discussing how the data was collected (e.g., in-class, online, for credit, optional, etc.). We recommend following the questions presented in Knaub et al [32] such as "what information regarding the sample is useful for the audience," as well as emphasizing explicit sample descriptions and limitations [33].…”

“…1. Describe the data set, how it was collected, the types of data that are utilized in the study, the context of the data whether it's from a specific institution or many, and any processes used to impute the data or otherwise alter it [27,32].…”

Editorial: Focused Collection: Quantitative Methods in PER: A Critical Examination

“…This Focused Collection offers the PER community with a variety of ideas to support and strengthen the work that we conduct. A number of authors pushed on our ideas of data and analysis providing an opportunity for an ongoing conversation about what researchers need to consider when designing, executing, and reporting a quantitative study [11][12][13][14]. Other authors highlighted analytical approaches that push the boundaries of quantitative PER or that solve persistent and common issues in our analyses [15][16][17][18][19].…”