An Exploratory Study of Characteristics Related to Irt Parameter Invariance With the Test of English as a Foreign Language

Abstract

IRT equating methods have been used successfully with the TOEFL® test for many years, and for the most part the observed properties of items have been consistent with model predictions. However, items that do not appear to hold their IRT pretest estimates do exist. If relationships can be found between features of TOEFL items in pretest calibrations and subsequent lack of model‐data fit when these items are used in final forms, steps to eliminate the use of such items in TOEFL final forms can be taken. The purpose of this study was to provide an exploratory investigation of item features that may contribute to a lack of invariance of TOEFL item parameters. The results of the study indicated the following: (1) subjective and quantitative measures developed for the study provided consistent information related to the model‐data fit of TOEFL test items, (2) for Sections 1 and 2, items that were pretested before 1986 exhibited poorer model‐data fit than items that were pretested after 1986, and (3) for Section 3 reading comprehension, model‐data fit appeared to be related to changes in the relative position of items within the sections from the pretest to the final form administrations. Based on the results of the study, it was recommended that (1) the TOEFL program investigate the feasibility of not using pretest IRT statistics for items pretested before 1986 for Sections 1 and 2 and (2) that guidelines be developed for test developers to use with reading comprehension items to limit the change in relative positions of items in the test from pretest to final form administrations.

Item Position and Item Difficulty Change in an IRT-Based Common Item Equating Design

“…Wise, Chia, and Park (1989) found that item order and other item context alterations impacted low-achieving test-takers more than high-achieving test-takers on tests of work knowledge and arithmetic reasoning. In addition, Way, Carey, and Golub-Smith (1992) found that TOEFL reading comprehension items that were pretested toward the beginning of the section but that appeared toward the end of the section on the operational test became more difficult, but items pretested near the end of the section and operationally tested near the beginning of the section became easier. These studies suggest that item position effects can vary depending on factors such as how much the item position changes, the direction of change (i.e., toward the beginning of the test versus toward the end of the test), and the ability levels of the test-takers.…”

Mathematics Performance Assessment: Discovering Why Some Items or Rubrics Don’t Measure Up

“…Design features such as construct specification, content coverage, item format, item representations, fairness, and nature of scoring criteria are discussed by Gitomer (1997), Lane (1993), and Linn, Baker, & Dunbar (1991) for example. Research in educational measurement journals describes studies that examine how well an item "fits" the psychometric model used to scale the student responses (e.g., Tang, 1994;Thissen & Steinberg, 1988;Way, Carey, and Golub-Smith, 1992). An item that fits the model is considered to be a well-functioning item for which the observed student performance matches the expected performance based on the model used.…”

Tolerable Variation in Item Parameter Estimates for Linear and Adaptive Computer‐based Testing

“…Divgi (1986) documented changes in item parameter estimates in an early application of the Armed Services Vocational Aptitude Battery (CAT-ASVAB). Several researchers have investigated causes of item parameter drift in testing programs that utilized IRT in test construction and equating over time (Eignor & Stocking, 1986;Kolen & Harris, 1990;Sykes & Fitzpatrick, 1992;Way, Carey, & Golub-Smith, 1992).…”