An Investigation of Proposed Revisions to Section 3 of the Toefl Test

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimuli (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, and Du, 2000). This parameter, γ, characterizes the amount of local dependence in a testlet.We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters [discrimination (a), difficulty (b), and guessing (c)] with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over-estimated when conditional independence was incorrectly assumed. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government. ✥ ✥ ✥A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Committee of Examiners. Its 11 members include representatives of the TOEFL Board, and distinguished English as a second language specialists from the academic community. The Committee meets twice yearly to review and approve proposals for test-related research and to set guidelines for the entire scope of the TOEFL research program. Members of the Committee of Examiners serve three-year terms at the invitation of the Board; the chair of the committee serves on the Board.Because the studies are specific to the TOEFL test and the testing program, most of the actual research is conducted by ETS staff rather than by outside researchers. Many projects require the cooperation of other institutions, however, particularly those with programs in the teaching of English as a foreign or second language and applied linguistics. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office...

Section: The Resultsmentioning

confidence: 99%

Section: The Resultsmentioning

confidence: 99%

Using a New Statistical Model for Testlets to Score Toefl

Wainer¹,

Wang²

2001

“…Each total form consisted of seven reading passage sets, five of which were common across a set of forms, and two were unique to each form. To avoid the confounding of possible effects that may be related to the timing of the test with dimensionality effects (Oltman, Stricker, & Barrows, 1988;Schedl, Thomas, & Way, 1995), the items in the final, or seventh, passage in each of the TOEFL forms were eliminated from the analyses, leaving the total number of items per modified total form to be analyzed to be 38 or 39. The study design provided 10 replications (i.e., 10 total form combinations) and reasonable power to detect whether the reasoning items measure an ability different from that being assessed by the remaining items in the reading test.…”

Section: Methodsmentioning

confidence: 99%

“…McKinley and Way (1992) explored the feasibility of an item response theory-based method of modeling examinee performance on secondary dimensions present in the test. As part of a feasibility study in preparation for the introduction of a revised TOEFL test in 1995, Schedl, Thomas, and Way (1995) conducted an analysis of the two subparts of Section 3, Vocabulary and Reading Comprehension, and found that departures from essential unidimensionality for the two subsections were primarily due to end-of-test effects that might be related to the timing of the test. While previous research has shown multidimensionality for the test as a whole, none has addressed the constructs implied by the specifications for reading comprehension items.…”

Section: Introductionmentioning

confidence: 99%

An Analysis of the Dimensionality of Toefl Reading Comprehension Items

Schedl

Gordon

Carey

et al. 1995

The issue of exactly what is measured by different types of reading items has been a matter of interest in the field of reading research for many years. Language teaching and testing specialists have raised the question of whether a reading test for foreign students wishing to enter university in the United States should include questions testing abilities beyond linguistic and very general discourse competencies, or indeed whether it is possible to separate these language competencies from other competencies. The purpose of this study was to investigate the dimensionality of the TOEFL® reading test, based on the specifications in use as of April 1991. Of particular interest was whether four item types identified in the test specifications as “reasoning items” could be shown to measure, in addition to general reading ability, any abilities not measured by the other item types in the TOEFL reading test. Two techniques, Stout's procedure and NOHARM analyses, were employed to investigate the hypothesized two‐factor model. In both cases the data failed to fit the model, indicating that TOEFL “reasoning items” cannot be shown to measure a unique construct. However, the follow‐up exploratory analyses indicated that all 10 test forms used in the study violated the assumption of essential unidimensionality, and all of the forms appeared to fit a two‐factor model where the second factor may be related to passage content or position.

“…The study was described in Schedl, Thomas, and Way (1995). Tests with different numbers of items were randomly assigned to examinees during each administration of the experimental section.…”

Section: Toefl Datamentioning

confidence: 99%

Estimating the Effects of Test Length and Test Time on Parameter Estimation Using the Hybrid Model

Yamamoto

1995

117

Educational Testing Serviceis an Equal Opportunity/Affirmative Action Employer.Copyright© 1995 by Educational Testing Service.All rights reserved.No part of this report may bereproduced or transmittedin any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, withoutpermission in writingfrom the publisher. Violatorswill be prosecutedin accordancewithbothU.S. and international copyrightand trademarklaws.EDUCATIONAL TESTING SERVICE, ETS, the ETS logo, GRE, TOEFL, the TOEFL logo, and TWE are registeredtrademarksof EducationalTesting Service. AbstractWhen individuals perform tasks, they differ from each other not only in their ability to perform the tasks correctly, but also in their speed. Even though the traditional indicator of test speededness, missing responses, clearly indicates a lack of time to respond (thereby indicating the speededness of the test), it is inadequate for evaluating speededness in a multiple-choice test scored as number correct and underestimates test speededness. Conventional IRT parameter estimation ignores the mixture of random responses during calibration; consequently, estimated parameters are biased.The HYBRID model (Yamamoto, 1989) was extended (Yamamoto, 1990) to characterize when each examinee switches from an ability-based response strategy to a strategy of responding randomly. The model has allowed us to evaluate test speededness by estimating the proportions of examinees who switch strategies at any possible point in the test. The estimated IRT parameters based on the HYBRID model were more accurate than the ordinary IRT-only analysis.With the extended HYBRID model applied to the data taken from an experimental form of TOEFL~, we found that 1) the test length had a small impact on the proportion of the examinees affected by the speededness of the test, 2) a greater proportion of examinees were affected by speededness of a test with a 50-minute time limit than a test with a 55-or 6O-minute time limit, and 3) the difference in the proportions of examinees affected by speededness of tests under 55-and 6O-minute time limits was small. However, nearly 20 percent of the examinees were affected by speededness after completing 80 percent of the test. In other words, the last 20 percent of the responses of 20 percent of the examinees did not represent their true ability. A continuing programof research relatedto the TOEFL test is carriedout under the direction of the TOEFL Research Committee. Its six members include representatives of the Policy Council, the TOEFLCommittee of Examiners, anddistinguished Englishas a secondlanguagespecialists from the academic community. The Committee meets twice yearlyto reviewand approveproposals for testrelatedresearchand to set guidelines for the entirescopeof theTOEFLresearchprogram. Members of theResearch Committeeservethree-year termsat theinvitation of thePolicy Council; thechairof the committee serves on the Policy Council.The Test of English as a Foreign Language was deve...