Methods for Addressing Missing Data with Applications from ACS Exams

Abstract

As part of the ACS Examinations Institute (ACS-EI) national norming process, student performance data sets are collected from professors at colleges and universities from around the United States. Because the data sets are collected on a volunteer basis, the ACS-EI often receives data sets with only students' total scores and without the students' responses to individual exam questions. Nonetheless, several national norming statistics require students' item responses. This data return leads to missing data and potentially biased results when inferences are made based on that data set. This work uses student performance data sets from ACS-EI to consider how methods for replacing missing data, such as hot-deck imputation and simulating data, affect the nature of the analysis of quantitative data. ABSTRACT: As part of the ACS Examinations Institute (ACS-EI) national norming process, student performance data sets are collected from professors at colleges and universities from around the United States. Because the data sets are collected on a volunteer basis, the ACS-EI often receives data sets with only students' total scores and without the students' responses to individual exam questions. Nonetheless, several national norming statistics require students' item responses. This data return leads to missing data and potentially biased results when inferences are made based on that data set. This work uses student performance data sets from ACS-EI to consider how methods for replacing missing data, such as hot-deck imputation and simulating data, affect the nature of the analysis of quantitative data.

Development of a Method for Imputation of Missing Data Using ACS Exams as a Prototype

“…Our use of the term "known" is analogous to the use of the term "observed" in previous work. 14 It is also important to note that difficulty and discrimination values can only be calculated using student data from the known data subset. The goal of the work we report herein is to develop an imputation method to populate the missing student responses so that they can be included within the individual item statistics.…”

“…Therefore, methods to fill in incomplete data sets are needed and have been previously investigated and reported in this Journal. 14 These methods, however, allow for variation between inputted and observed total scores, a constraint that we will add to our imputation method; in addition, the ability to input data is dependent on the amount of missing data, which can be upward of 50% for "real" data. We have, therefore, further refined and extended methods reported in this Journal to smaller data sets that might be encountered in a typical course at an institution or in multi-institution assessments with a small combined number of students.…”

Development of the Exams Data Analysis Spreadsheet as a Tool To Help Instructors Conduct Customizable Analyses of Student ACS Exam Data

“…Because the ACS-EI collects national normative data sets on a volunteer basis, professors sometimes send students' overall test level performances but not their responses on an individual item basis. As a result, methods to estimate the missing data were used by the ACS-EI, 20 and the EDAS uses these augmented data sets to provide the national normative comparisons. In order to assess its utility and usability, the EDAS went through trial testing with 10 chemistry instructors from across the country and most of the instructors stated that the EDAS was a useful tool with several implications for the classroom.…”

Evaluation of Subset Norm Stability in ACS General Chemistry Exams

“…5−8 Additionally, a tool was developed by the ACS-EI to aid instructors as they conducted analyses of student performance on standardized, norm-referenced ACS General Chemistry Exams. 9,10 This tool, the Exams Data Analysis Spreadsheet (EDAS), allows instructors to perform unique analyses of performance data by allowing instructor selection of exam items to include in the analyses. The EDAS then generates new norms and class performance comparisons upon the basis of the subset of items the user has selected for analysis.…”

“…Therefore, the ACS-EI has made efforts to translate research endeavors into pragmatic approaches for classroom assessment. For example, development of the Anchoring Concepts Content Map (ACCM) for various chemistry subdisciplines stemmed from desired instructional interest to analyze students’ content knowledge and provide the opportunity for programmatic assessment. − Additionally, a tool was developed by the ACS-EI to aid instructors as they conducted analyses of student performance on standardized, norm-referenced ACS General Chemistry Exams. , This tool, the Exams Data Analysis Spreadsheet (EDAS), allows instructors to perform unique analyses of performance data by allowing instructor selection of exam items to include in the analyses. The EDAS then generates new norms and class performance comparisons upon the basis of the subset of items the user has selected for analysis.…”