The central purpose of this study was to empirically evaluate an in-depth vocabulary knowledge measure in the context of developing the new TOEFL test. The study was carried out with a sample of 207 international students attending an intensive English as a second language (ESL) program in a major Canadian university, in order to determine whether this vocabulary knowledge measure could provide a basis for developing appropriate and useful item types for assessing test-takers’ reading comprehension. The results indicate that, compared with existing TOEFL vocabulary measures, the new measure has a similar difficulty level and provides a similar amount of prediction of test-takers’ reading performance.

Providing information to test takers and test score users about the abilities of test takers at different score levels has been a persistent problem in educational and psychological measurement (Carroll, 1993). Since the 1990s Educational Testing Service has been investigating solutions to this problem through the development of proficiency scaling procedures and questiondifficulty research. In 1997 a proficiency scale was developed for the Test of English as a Foreign Language (TOEFL) Reading Comprehension section using a tree-based regression approach. The current study describes a scale-anchoring study of the new TOEFL iBT reading test and the resulting proficiency descriptors that are now part of the TOEFL iBT score report. The goal was to provide descriptive information about the abilities that test takers need in order to answer questions correctly. These abilities are those articulated in the new TOEFL Reading Framework and in the guidelines for writing test questions. Scale anchoring is a method of creating descriptors of the performance of test takers that is based on both empirical data and judgments by test developers. It has been used with a variety of assessments, including the National Assessment of Educational Progress (NAEP) and the Trends in International Mathematics and Science Study (TIMSS).

The aim of this study was to develop, for the benefit of both test takers and test score users, enhanced TOEFL ITP ® test score reports that go beyond the simple numerical scores that are currently reported. To do so, we applied traditional scale anchoring (proficiency scaling) to item difficulty data in order to develop performance descriptors for multiple levels of each of the three sections of the TOEFL ITP. A (novel) constraint was that these levels should correspond to those established in an earlier study that mapped (i.e., aligned) TOEFL ITP scores to a widely accepted framework for describing language proficiency-the Common European Framework of Reference (CEFR). The data used in the present study came from administrations of five current operational forms of the recently revised TOEFL ITP test. The outcome of the effort is a set of performance descriptors for each of several levels of TOEFL ITP scores for each of the three sections of the test. The contribution, we believe, constitutes (1) an enhancement of the interpretation of scores for one widely used assessment of English language proficiency and (2) a modest contribution to the literature on developing proficiency descriptors-an approach that combines elements of both scale anchoring and test score mapping.

The TOEFL testing program is currently exploring a change in Section 3 of the TOEFL® test that would replace the vocabulary subpart with additional reading comprehension questions. This change has been proposed by internal test development specialists and is supported by external experts in the field of English as a second language. The purpose of this study was to investigate the proposed revision to Section 3 in terms of the length and timing that would be necessary to address concerns of test speededness of the section. The study was carried out using an experimental design with test length and testing time defined as independent variables, and examinee test performance defined as the dependent variable. In addition, several psychometric issues relating to the proposed revision to Section 3 were investigated as part of the study.The results of the study supported the implementation of a revised TOEFL Section 3 consisting of five reading passages with a total of 50 items. The results of the study also suggested that a total testing time of no less than 55 minutes should be allowed for the revised TOEFL Section 3. Additional psychometric analyses indicated that the current TOEFL score scale can be maintained with the revised Section 3, and that the proposed revisions will not appreciably affect the reliability and validity of Section 3 of the TOEFL test. ETS administers the TOEFLprogram under the general direction of a Policy Council that was established by, and is affiliated with, the sponsoring organizations. Members of the PolicyCouncil represent the CollegeBoard,the GRE Board,and suchinstitutions and agenciesas graduateschools of business,junior and community colleges, nonprofiteducational exchangeagencies, and agencies of the United Statesgovernment.. A continuing programof research related to the TOEFL testis carriedout underthe direction of the TOEFL Research Committee. Its six members include representatives of the Policy Council, the TOEFLCommittee of Examiners, anddistinguished Englishasa secondlanguage specialistsfromthe academiccommunity. The Committee meets twice yearly to review and approveproposalsfor testrelatedresearch and to set guidelines for the entirescopeof the TOEFLresearchprogram.Members of theResearch Committee servethree-year termsat the invitation of the PolicyCouncil;thechairof the committeeserveson the Policy Council.Because the studies are specific to the test and the testing program, most of the actual research is conducted by ETS staff rather thanby outsideresearchers. Many projectsrequire the cooperation of other institutions, however, particularly those with programs in the teachingof Englishas a foreign or second language. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office. All TOEFL researchprojectsmust undergo appropriate ETS review to ascertain that data confidentiality will be protected.Current(1994-95) members of the TOEFLResearch Committee are:ii

A conununicative competence orientation wasadopted to study the validity of test score inferences derived from the revised Test of Spoken English (fSE). To implement the approach, a sample of undergraduate students, primarily native speakers of English, provided a variety of reactions to, andjudgments of, the test responses ofa sample ofTSE examinees. The TSE scores ofthese examinees, previously determined by official TSE raters, spanned the full range ofTSE score levels. Undergraduate students were selected as "evaluators" because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants.Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students' understanding of TSE examinees' speech, as represented by their taped responses to tasks on the Test of Spoken English. The objective was to determine the degree to which official TSE scores are predictive of listeners' ability to understand the messages conveyed by TSE examinees. Analyses revealed a strong association between TSE score levels and the judgments, reactions, and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several different kinds of evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in future test development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).The Test ofEnglish as a Foreign Language (TOEFLIll) was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizationsconcerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965,Educational Testing Service (ETSIIl) and the College Board assumed joint responsibility for the program. In 1973,a cooperative arrangement for the operationof the program was entered into by ETS, the College Board, and the Graduate Record Examinations (GRE~) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education.ETS administers the TOEFL program under the general direction of a Policy Council that was established by, and is affiliated with, the sponsoring organizations. Members of the Policy Council represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business,junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government.A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Research Committee. Its six members include representatives of the Policy Council, the TOEFL Comm...