Subscores Based on Classical Test Theory: To Report or Not to Report

Sinharay, Sandip; Haberman, Shelby J.; Puhan, Gautam

doi:10.1111/j.1745-3992.2007.00105.x

Cited by 77 publications

(83 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, Haberman (2005Haberman ( , 2007Haberman, Sinharay, & Puhan, 2006;Sinharay, Haberman, & Puhan, 2007) found that reliability-based analysis is helpful in informing the decision of reporting subscores. His approach compares the root mean square error (RMSE) of the true subscore when it is predicted (or approximated) by the observed subscale score, the observed total test score, and the two scores conjointly.…”

Section: Methods and Analysesmentioning

confidence: 99%

Why the Major Field Test in Business Does Not Report Subscores: Reliability and Construct Validity Evidence

Ling

2012

ETS Research Report Series

View full text Add to dashboard Cite

To assess the value of individual students' subscores on the Major Field Test in Business (MFT Business), I examined the test's internal structure with factor analysis and structural equation model methods, and analyzed the subscore reliabilities using the augmented scores method.Analyses of the internal structure suggested that the MFT Business measures a unidimensional construct, which does not support reporting individual students' subscores. Augmented score analyses revealed that the observed total score could approximate the true subscore more accurately than the observed subscores, which do not support reporting individuals' subscores either. The results from these two approaches provided consistent evidence in support of the current practice of not reporting individuals' subscores on the MFT Business. The relationship between the two approaches is discussed, followed by a recommendation of an alternate method for future research.

show abstract

Section: Methods and Analysesmentioning

confidence: 99%

Why the Major Field Test in Business Does Not Report Subscores: Reliability and Construct Validity Evidence

Ling

2012

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…The approach of Haberman (2008) (Haberman, 2008). Sinharay, Haberman, and Puhan (2007) discussed why this strategy is reasonable and how it ensures that a section score satisfies professional standards. A larger value of x PRMSE for a TOEFL section means than the corresponding TOEFL section score does a better job than the TOEFL total score of predicting the corresponding true section score.…”

Section: Analysesmentioning

confidence: 99%

INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

Sawaki

Sinharay

2013

ETS Research Report Series

Self Cite

View full text Add to dashboard Cite

This study investigates the value of reporting the reading, listening, speaking, and writing section scores for the TOEFL iBT ® test, focusing on 4 related aspects of the psychometric quality of the TOEFL iBT section scores: reliability of the section scores, dimensionality of the test, presence of distinct score profiles, and the section scores' generalizability for norm-referenced decisions as well as the dependability of criterion-referenced decisions for international student admission.Four operational TOEFL iBT test forms were analyzed for all examinees as well as for 3 native language (L1) groups (Arabic, Korean, and Spanish).Haberman's (2008) subscore analysis suggested that the speaking section score had added value due to its relative distinctness from the other modalities. Consistent with the subscore analysis results, a series of exploratory factor analyses (EFAs) indicated the possibility of the presence of 2 correlated factors-a reading/listening/writing factor and a speaking factor.In contrast, the CFAs conducted separately for the 3 L1 groups as well as a multiple-group confirmatory factor analyses (CFAs) identified a correlated 4-factor model with reading, listening, speaking, and writing factors as the best representation of the structure of the entire test for all examinees as well as for the 3 L1 groups. Reliability of the observed section scores for norm-referenced score interpretations and the dependability of classification decisions made based on different cut scores were generally satisfactory while they were also found to be relatively low in some circumstances. Based on the mixed results concerning the value-added information the TOEFL iBT section scores provide, recommendations for future research directions and some key issues of consideration for high-stakes decision making based on the section scores were summarized.Key words: dependability, dimensionality, factor analysis, generalizability theory, reliability, score profile, subscore analysis ii TOEFL ® was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizations concerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965, Educational Testing Service (ETS) and the College Board ® assumed joint responsibility for the program. In 1973, a cooperative arrangement for the operation of the program was entered into by ETS, the College Board, and the Graduate Record Examinations ® (GRE ® ) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education. The test is now wholly owned and operated by ETS.ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previo...

show abstract

“…If we are confident that we are successfully measuring 'clinical skills' only at the whole-OSCE level, a narrow psychometric viewpoint might argue that pass/fail decisions must be made based on this. Making decisions that rest, to an extent, on subsets of stations where decisions are subject to greater, possibly unacceptable, error could be construed as less robust (Sinharay, Haberman, & Puhan, 2007;Sinharay, Puhan, & Haberman, 2011).…”

Section: Station Level Effectsmentioning

confidence: 99%