Multilevel Rasch modeling of two-tier multiple choice test: A case study using Lawson’s classroom test of scientific reasoning

Abstract

Assessment instruments composed of two-tier multiple choice (TTMC) items are widely used in science education as an effective method to evaluate students' sophisticated understanding. In practice, however, there are often concerns regarding the common scoring methods of TTMC items, which include pair scoring and individual scoring schemes. The pair-scoring method is effective in suppressing "false positives" at the cost of missing possible middle states of progression of student understanding. On the other hand, the individual scoring method captures an undistinguished middle level but is prone to rewarding guessing, which leads to "false positives". In addition, this middle level does not discriminate the progression between knowing the result and explaining the reason, which limits the capacity of drawing meaningful implications from the assessment outcomes. To address the concerns with the current scoring methods, it is valuable to explore new scoring method(s) that can fully utilize the information measured with TTMC items. In this study, a number of scoring models are studied using Rasch analysis on data of a popular TTMC test, the Lawson classroom test of scientific reasoning (LCTSR), collected from four considerably different populations. The results show that the model fit quality of the scoring methods varies with student population and item design. In general, there is no one-fits-all solution; however, given the new information obtained in this study, a three-step process is suggested that can guide the development of new mixed scoring models tailored for a particular population and or test. The evaluation results show that the mixed models produce the most reliable model fitting and better than average goodness of fit. Furthermore, the results in this study also confirm previous studies, which suggest that it is harder to come up with a correct explanation than to just know the answer.

Validity evaluation of the Lawson classroom test of scientific reasoning

“…Students might implicitly know that the red light was not a factor, and then primarily focused on the influential factor (the gravity) in their explanations. This also agrees with many two-tier assessment studies which have demonstrated that knowing is easier than reasoning [34,41].…”

“…While the two-tier test was originally developed as an effective alternative of the traditional multiple-choice test to suppress the possible false positive [34,42,57], it has been unclear as to how to quantify the likelihood of false positives in TTMC tests. The method of analyzing inconsistent response patterns described in this paper provides a practical means to quantitatively evaluate the probabilities of possible false positives and negatives, which can be used as evidence for establishing the validity of two-tier tests.…”

“…Understanding the relation between knowing and reasoning can provide important cognitive and education insights into how teaching and learning can be improved. Recent studies have started to show a possible progression from knowing to reasoning, which suggests that reasoning is harder than knowing [34,41,[53][54][55][56]. However, the actual difficulty reflected within a given test can also be affected by a number of factors including test design, content, and population [34,41], and therefore needs to be evaluated with all the possible factors considered and or controlled.…”

“…The scoring of the LCTSR has evolved into two forms: one is the pair scoring method, which assigns one point when both questions in a pair are answered correctly, and the other is the individual scoring method, which simply grades each question independently. Both methods have been frequently used by researchers [4,11,16,[32][33][34].…”

Linking and comparing short and full-length concept inventories of electricity and magnetism using item response theory

“…Meanwhile, nearly all FCI items showed reasonable item parameters. In recent research and development, the use of the IRT family of analysis methods has gained popularity in PER, especially in the area of assessment development and validation [22][23][24][25]. For example, following the Rasch modeling procedures of assessment development [20], Mešić et al applied the Rasch model to develop the wave optics item bank [22].…”