INVESTIGATING THE VALUE OF SECTION SCORES FOR THE TOEFL iBT® TEST

Abstract

This study investigates the value of reporting the reading, listening, speaking, and writing section scores for the TOEFL iBT ® test, focusing on 4 related aspects of the psychometric quality of the TOEFL iBT section scores: reliability of the section scores, dimensionality of the test, presence of distinct score profiles, and the section scores' generalizability for norm-referenced decisions as well as the dependability of criterion-referenced decisions for international student admission.Four operational TOEFL iBT test forms were analyzed for all examinees as well as for 3 native language (L1) groups (Arabic, Korean, and Spanish).Haberman's (2008) subscore analysis suggested that the speaking section score had added value due to its relative distinctness from the other modalities. Consistent with the subscore analysis results, a series of exploratory factor analyses (EFAs) indicated the possibility of the presence of 2 correlated factors-a reading/listening/writing factor and a speaking factor.In contrast, the CFAs conducted separately for the 3 L1 groups as well as a multiple-group confirmatory factor analyses (CFAs) identified a correlated 4-factor model with reading, listening, speaking, and writing factors as the best representation of the structure of the entire test for all examinees as well as for the 3 L1 groups. Reliability of the observed section scores for norm-referenced score interpretations and the dependability of classification decisions made based on different cut scores were generally satisfactory while they were also found to be relatively low in some circumstances. Based on the mixed results concerning the value-added information the TOEFL iBT section scores provide, recommendations for future research directions and some key issues of consideration for high-stakes decision making based on the section scores were summarized.Key words: dependability, dimensionality, factor analysis, generalizability theory, reliability, score profile, subscore analysis ii TOEFL ® was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizations concerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965, Educational Testing Service (ETS) and the College Board ® assumed joint responsibility for the program. In 1973, a cooperative arrangement for the operation of the program was entered into by ETS, the College Board, and the Graduate Record Examinations ® (GRE ® ) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education. The test is now wholly owned and operated by ETS.ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previo...

Investigating the Relationship Between Test‐Taker Background Characteristics and Test Performance in a Heterogeneous English‐as‐a‐Second‐Language (ESL) Test Population: A Factor Analytic Approach

“…This comparison showed that the correlated four-factor model representing the four language modalities of reading, listening, speaking, and writing best accounted for the underlying latent factor structure of the TOEFL iBT test. This result is consistent with a recent investigation by Sawaki and Sinharay (2013) into the factor structure of the TOEFL iBT test. However, these findings are different from that of Sawaki et al (2008), who concluded that the higher-order factor model with a higher-order general factor representing ESL/EFL and four first-order factors for reading, listening, speaking, and writing represented the factor structure of the TOEFL iBT test.…”

“…In the first step, an item-level CFA was used to evaluate several competing models representing the factor structure of the TOEFL iBT test. These models used in this study were similar to those studied in previous CFA analyses of the TOEFL iBT test (Gu, 2014;Sawaki & Sinharay, 2013;Sawaki et al, 2008;Stricker & Rock, 2008). Specifically, the following six factor structures, as described in Sawaki et al (2008), with the exception of 4a, were evaluated and are diagramed in Appendix.…”

“…Thus, the sample used for the study may not be representative of the current test population in terms of educational and other background characteristics. In a more recent study, Sawaki and Sinharay (2013) used data from operational administrations of the TOEFL iBT test and found that the correlated fourfactor structure, corresponding to the four language skills (reading, listening, speaking, and writing), best represented the latent structure of the test. In contrast to Sawaki et al's study, Sawaki and Sinharay used an item-parcel approach (items grouped by content category) rather than item-level approach to the CFA.…”

Evaluating Invariance in Test Performance for Adolescent Learners of English as a Foreign Language

“…Interestingly, the correlated four-factor model has been found to represent English-language proficiency of adult learners using the TOEFL iBT ® test (Manna & Yoo, 2015;Sawaki & Sinharay, 2013), despite the differences in target age groups and assessment instruments. However, unlike adult learners, for whom differential performance across the four latent constructs is not unexpected due to greater variation in background characteristics, the levels of proficiency across the latent constructs in the two groups of adolescent learners studied tended to be similar.…”

“…Currently the general view among researchers is that L2 proficiency is complex and multidimensional in nature (Carroll, 1965;Harsch, 2014;Oller, 1983). As noted earlier, this view has been supported by a number of factor analytic studies that found that language ability comprises one or more components representing the four language skills (reading, listening, speaking, and writing) that may be distinct, closely related, or hierarchically related to a global ability (Bachman et al, 1995;Sawaki & Sinharay, 2013;Sawaki, Stricker, & Oranje, 2009).…”