Using a New Statistical Model for Testlets to Score Toefl

Abstract

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimuli (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, and Du, 2000). This parameter, γ, characterizes the amount of local dependence in a testlet.We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters [discrimination (a), difficulty (b), and guessing (c)] with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over-estimated when conditional independence was incorrectly assumed. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government. ✥ ✥ ✥A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Committee of Examiners. Its 11 members include representatives of the TOEFL Board, and distinguished English as a second language specialists from the academic community. The Committee meets twice yearly to review and approve proposals for test-related research and to set guidelines for the entire scope of the TOEFL research program. Members of the Committee of Examiners serve three-year terms at the invitation of the Board; the chair of the committee serves on the Board.Because the studies are specific to the TOEFL test and the testing program, most of the actual research is conducted by ETS staff rather than by outside researchers. Many projects require the cooperation of other institutions, however, particularly those with programs in the teaching of English as a foreign or second language and applied linguistics. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office...

Modeling Local Item Dependence Due to Common Test Format With a Multidimensional Rasch Model

“…The reliability of the test was overestimated when the standard Rasch model was employed. One should note that although the parameter estimates based on the two models are very similar, the information function based on the standard Rasch model could be imprecise since local dependence ensuing from shared item formats is ignored (Ip, 2010;Wainer & Wang, 2000). In general, ignoring LID due to methods can act like LID due to common stimuli and lead to inaccurate estimation of reliability and standard errors of the primary ability dimension (Bradlow, Wainer, & Wang, 1999;Marais & Andrich, 2008).…”

“…Most empirical examinations of this issue, using testlet model have focused on local dependence due to common stimuli. For example, Wainer and Wang (2000) analyzed the reading and listening comprehension sections of the Test of English as a Foreign Language with the 3-PL TRT model. They compared the difficulty, discrimination, and guessing parameters estimated by the TRT model with those obtained from a standard IRT model.…”

“…LID can also lead to premature termination of computer adaptive tests when the criterion for termination is the precision of ability parameter estimates (Wainer & Wang, 2000). Much the same problem exists when classical test theory is employed.…”

The Answer is in the Question: A Guide for Describing and Investigating the Conceptual Foundations and Statistical Properties of Cognitive Psychometric Models

“…Since the constituent measurement model for the rule-space methodology is an IRT model that produces estimated ability values, models that include parameters for testlet dependencies (e.g., Wainer & Wang, 2001;Wang, Bradlow, & Wainer, 2002) could be used. In order to model multiple strategies or stages of development, models are also available (e.g., Wilson, 1989) even though the ability estimates from the different classes would have to be comparable in meaning for the rule-space analysis to be interpretable also.…”

An Examination of Parametric and Nonparametric Dimensionality Assessment Methods with Exploratory and Confirmatory Mode

“…The use of unidimensional IRT models in conditions incompatible with the local independence assumption leads to lower estimations owing to the requirement of standard error values in individuals' ability parameter estimations (Wainer & Wang, 2001). In addition, the traits measured in the fields of education and psychology are generally so complicated that they cannot be grouped under a single dimension.…”