Enhancing the Interpretation of a Norm‐referenced Second‐language Test Through Criterion‐referencing: A Research Assessment of Experience in the Toeic Testing Context

Abstract

This study was undertaken to develop guidelines for making interpretive inferences from scores on the Test of English for International Communication (TOEIC), a norm‐referenced test of English‐language listening comprehension (LC) and reading (R) skills, about level of ability to use English in face‐to‐face conversation, indexed by performance in the Language Proficiency Interview (LPI) situation. LPI performance, rated according to behaviorally defined levels on the LPI/ILR/FSI quasi‐absolute proficiency scale, was treated as a context‐independent criterion, using the familiar regression model in an apparently novel application (for such criterion‐referencing purposes) in the context of a large‐scale ESL‐testing program. The study employed TOEIC/LPI data‐sets generated during operational ESL assessments in representative TOEIC‐use settings (places of work or work‐related ESL training) in Japan, France, Mexico, and Saudia Arabia, involving samples of adult, educated ESL users/learners in or preparing for ESL‐essential positions with companies engaged in international commerce. The pattern of TOEIC/LPI concurrent correlations was consistent across samples and there was relatively close fit between sample LPI means and estimates from TOEIC scores, especially TOEIC‐LC, using combined‐sample regression equations. Theoretical and pragmatic implications of the findings are discussed. General guidelines are provided for making inferences about LPI‐assessed level of oral English proficiency from TOEIC scores. Directions are suggested for further research and development activities in the TOEIC testing context.

Validity of Global Self‐ratings of Esl Speaking Proficiency Based on an Fsi/Ilr‐referenced Scale

“…Instead, attention was focussed on an indirect assessment which involved evaluating "parallelism" in patterns of findings involving TOEIC scores and self rating in the Swiss sample, on the one hand, and patterns involving TOEIC scores and LPI rating in previously studied samples (e.g., Wilson, 1989;Wilson and Stupak, 1998;Wilson and Chavanich, 1989;Wilson, Komarakul & Woodhead, 1997), on the other hand.…”

“…2. What is the extent of agreement between levels ofESL speaking proficiency as indexed, on the one hand, by self-assigned proficiency levels on a scale designed specifically to nominally parallel the ILR-scale used to rate performance in Language Proficiency Interviews and, on the other hand, by ILR-scaled levels that typicially are assigned by professional interviewers/raters as descriptive of LPI performance for individuals with similar scores on the TOEIC--according to empirically validated, regression-based guidelines for inferring LPI rating from TOEIC scores in linguistically diverse samples (e.g., Wilson, 1989;Wilson & Chavanich, 1989;Wilson, Komarakul & Woodhead, 1997;Wilson & Stupak, 1998).…”

“…The SA scale--as written originally in English--was introduced as an exploratory measure in a study (Wilson, 1989) that was concerned primarily with assessing concurrent relationships between formal LPI ratings and scores on the Test of English for International Communication [TOEIC] (e.g., ETS, 1986),C 1 in a sample of native-speakers ofFrench--employees in ESL-essential positions with an international accounting firm, tested on-site in Paris. In 1994, the SA scale was translated into French and used with a sample of students enrolled in ESL training in the Paris area, who also took the TOEIC and the Test of English as a Foreign Language or TOEFL (e.g., ETS, 1991).…”

“…In the initial assessment, the English-language version was administered in a study (Wilson, 1989) concerned primarily with evaluating concurrent relationships between formal Language Proficiency Interview (LPI) rating and scores on the TOEIC in a sample of native-French speaking ESL users/ learners (N = 56) employed by an international accounting firm in Paris. A French-language version of the scale was administered in 1994, also as an exploratory measure, in an operational TOEIC assessment involving a sample (N=114) of students 20 to 24 years of age, enrolled in an intensive ESL program in the Paris area; no language proficiency interviews were conducted for this sample.…”

“…In order to ensure that differences in ESL reading proficiency would not complicate the rating process for the majority of examinees (native-speakers of either French or Ger-man), the original English-language version of the self-rating schedule (Wilson, 1989; also Table 4, above) was translated into French and German versions. Because ESL users/learners taking the TOEIC seldom exhibit speaking proficiency above Level 3+, the self-rating schedules excluded descriptions corresponding nominally to ILR levels above Level 3+.…”