The Toeic® Speaking and Writing Tests: Relations to Test‐taker Perceptions of Proficiency in English

Abstract

To facilitate the interpretation of test scores from the new TOEIC ® speaking and writing tests as measures of English-language proficiency, we administered a self-assessment inventory to TOEIC examinees in Japan and Korea, to gather their perceptions of their ability to perform a variety of everyday English-language tasks. TOEIC scores related relatively strongly to test taker self-reports for both speaking and writing tasks. The results were extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also suggested modest discriminant validity of the new TOEIC speaking and writing measures, suggesting that both measures contribute uniquely to the assessment of Englishlanguage proficiency. ) was developed to measure the ability to listen and read in English, using a variety of contexts from real-world settings.Recently, ETS added TOEIC speaking and writing tests to the TOEIC product line in order to directly assess the ability to speak and write in English in a workplace setting. This addition was in response to multinational corporations' need for employees with high-level speaking and writing skills. In contrast to the paper-and-pencil TOEIC listening and reading tests, multiplechoice tests requiring test takers to select correct answers, the computer-delivered TOEIC speaking and writing measures require test takers to produce responses that are then scored subjectively by highly trained human raters. The new measures thus complement the TOEIC listening and reading tests. Together, the four components of the TOEIC test battery now provide measurement of all four English-language communication skills.The new tests were developed to align as closely as possible with theories of communicative competence (see, for example, Butler, Eignor, Jones, McNamara, & Suomi, 2000; Cumming, Kantor, Powers, Santos, & Taylor, 2000). To accomplish this, an evidencecentered design (ECD) approach was used (see, for example, Mislevy & Haertel, 2006; Mislevy, Steinberg, Almond, & Lukas, 2006). In short, ECD methodology entails• Looking at the population for which the test is intended and the uses to which the test will be put• Articulating the desired claims to be made about test takers based on their performance on the assessment;• Identifying test-taker behaviors that would allow these claims to be made• Creating (and evaluating) tasks to elicit these behaviors, thus providing evidence to support the claims For the speaking measure, three hierarchical claims were specified-that test takers can:1. Create connected, sustained discourse appropriate to the typical workplace 2. Carry out routine social and occupational interactions such as giving and receiving directions, asking for information, and asking for clarification 3. Produce some language that is intelligible to native and proficient non-native English- Next we invited TOEIC clients in Japan and Korea to r...

The incremental contribution of TOEIC® Listening, Reading, Speaking, and Writing tests to predicting performance on real-life English language tasks

“…These statements were selected from longer selfassessment inventories that had been administered in previous studies of the TOEIC tests (Powers, Kim, & Weng, 2008;Powers, Kim, Yu, Weng, & VanWinkle, 2009). For the current effort, the sample of tasks was selected from the longer assessments so as to represent a range of activities of varying content and degree of difficulty, as estimated from the previous self-assessment studies referenced above.…”

Defining and Operationalizing the Construct of Pragmatic Competence: Review and Recommendations

“…He found the following activities to feature most prominently in business contexts: writing and reading of e-mails and reports, formal, and informal meetings, telephoning, conferences, seminars, and presentations. Similar tasks were identified by Singh and Shamsudin (2009) in their analysis of written workplace communication in Malaysia as well as by Powers et al (2009) who asked 3,814 test takers who took the TOEIC ® assessment which tasks they were usually engaged in and able to do in their respective workplace settings. Moreover, Holmes (2000), who focused on oral means of communication in professional contexts in New Zealand, identified meetings, telephone conversations, and social talk as crucial workplace tasks.…”

“…Second, with regard to business-oriented communication, several studies have identified a range of tasks in business interactions (e.g., Clyne, 1994;Evans, 2013;Flowerdew & Wan, 2006;Holmes, 2000;Koester, 2010;Lan & MacGregor, 2010;Marra, 2012;Powers, Kim, Yu, Weng, & VanWinkle, 2009;Singh & Shamsudin, 2009;Wenger, 1998;Willing, 1992Willing, , 1997. Although it is impossible to provide an exhaustive description, as each workplace has its own genres and communities of practice, several studies have identified and examined tasks that range across a broad variety of professional and business contexts.…”

“…(Holmes, 2000) Hence, it is important for ESL/EFL learners as well as for L1 English speakers who work in international contexts to be aware of these functional-pragmatic phenomena of language use when engaging in tasks at the workplace. With regard to listening and speaking tasks, Powers et al (2009) pointed out that it is essential for an employee to be able to understand a client's request made on the phone, understand the implicature in a business offer made in a presentation, explain to a coworker how to operate a machine (e.g., photocopier, computer, audio player), or have small talk with a guest about topics of general interest before discussing business. Employees need to convey instructions, seek advice, provide feedback, request information, make suggestions, or give compliments (Evans, 2013;Holmes, 2000;Koester, 2010;Riddiford & Joe, 2010;Wigglesworth & Yates, 2007).…”

A Historical Survey of Research Regarding Constructed-Response Formats

“…Automated scoring of speaking as discussed above, could help to reduce costs, but is not yet sufficiently well developed (Bridgeman et al 2012). The TOEIC ® Speaking and Writing test followed the TOEFL (Pearlman 2008b) in using ECD for assessment design (Hines 2010) as well as in the inclusion of speaking (Powers 2010;Powers et al 2009). …”