Some Paths Toward Making Praxis Scores More Useful

Abstract

Xiaohui WangIn this study we describe an analytic method for aiding in the generation of subscales that characterize the deep structure of tests. In addition we also derive a procedure for estimating scores for these scales that are much more statistically stable than subscores computed solely from the items that are contained on that scale. These scores achieve their stability through augmentation with information from other related information on the test. These methods were used to complement each other on a data set obtained from a Praxis administration. We found that the deep structure of the test yielded ten subscales, and that because the test was essentially unidimensional ten subscores could be computed; all with very high reliability. This result was contrasted with the calculation of six traditional subscales based on surface features of the items. These subscales also yielded augmented subscores of high reliability.

Subscores Based on Classical Test Theory: To Report or Not to Report

“…Wainer, Sheehan, and Wang (2000) and Wainer et al (2001) suggested the idea of an augmented subscore, that is, an estimated subscore based on all the available observed subscores so that, for example, an estimated algebra subscore of a student will be based on not only the observed algebra score of the student, but also on his/her other observed subscores, like those on arithmetic and geometry. Augmented subscores are more stable than the observed subscores themselves, especially when the observed subscores are based on a few items.…”

“…Augmented subscores are more stable than the observed subscores themselves, especially when the observed subscores are based on a few items. Wainer et al (2000) declared that “all subscores are measuring the same thing” (in other words, the subscores do not have any added value) if the reliabilities of the augmented subscores are all equal and are equal to that of the total score. They found subscores from an 150‐item (all multiple choice) Education in the Elementary School Assessment (EES), one of the PRAXIS™ examinations for teacher licensure, of no added value and commented that “We learned that the test's items were fiercely unidimensional, and so any set of subscales that were chosen would yield essentially the same information.” They found that augmented subscores were reliable enough for diagnostic purposes; however, they also found (see their figure 8) that for any examinee, the standardized augmented subscores were all virtually the same, in which case the subscores did not serve any useful diagnostic purpose.…”

Relating Unidimensional IRT Parameters to a Multidimensional Response Space: A Review of Two Alternative Projection IRT Models for Scoring Subscales

“…The first approach focuses on improving efficiency of the estimation processes of MIRT models for better diagnostic information. Research on the topic is quite active and includes a wide range of promising applications that incorporate additional hierarchical group and domain variables into the item parameter estimation processes (de la Torre & Patz, 2005; Mislevy, 1987; Kahraman, De Boeck, & Janssen, 2009; Rijmen, 2010; Yen, 1987; Wainer, Sheehan, & Wang, 2000). The second approach focuses on obtaining useful unidimensional (i.e.…”

Book Review

“…Would that be worthless? This ignores all the work on the estimation of stable subscores using empirical Bayes technology (Thissen & Wainer, 2001; Wainer, Sheehan, & Wang, 2000), with which even a few items can still yield stable subscores. “Multiple Choice items … primarily provide information on how students or schools compare” (p. 136)—actually they provide information on whatever they are designed to. This silliness is akin to claiming, “Rulers primarily provide information on who is tallest in the class.” “… the items from the National Assessment for Educational Progress are not released” (p. 145)—Sure they are.…”

“…Would that be worthless? This ignores all the work on the estimation of stable subscores using empirical Bayes technology (Thissen & Wainer, 2001; Wainer, Sheehan, & Wang, 2000), with which even a few items can still yield stable subscores.…”