To facilitate the interpretation of test scores from the new TOEIC ® speaking and writing tests as measures of English-language proficiency, we administered a self-assessment inventory to TOEIC examinees in Japan and Korea, to gather their perceptions of their ability to perform a variety of everyday English-language tasks. TOEIC scores related relatively strongly to test taker self-reports for both speaking and writing tasks. The results were extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also suggested modest discriminant validity of the new TOEIC speaking and writing measures, suggesting that both measures contribute uniquely to the assessment of Englishlanguage proficiency. ) was developed to measure the ability to listen and read in English, using a variety of contexts from real-world settings.Recently, ETS added TOEIC speaking and writing tests to the TOEIC product line in order to directly assess the ability to speak and write in English in a workplace setting. This addition was in response to multinational corporations' need for employees with high-level speaking and writing skills. In contrast to the paper-and-pencil TOEIC listening and reading tests, multiplechoice tests requiring test takers to select correct answers, the computer-delivered TOEIC speaking and writing measures require test takers to produce responses that are then scored subjectively by highly trained human raters. The new measures thus complement the TOEIC listening and reading tests. Together, the four components of the TOEIC test battery now provide measurement of all four English-language communication skills.The new tests were developed to align as closely as possible with theories of communicative competence (see, for example, Butler, Eignor, Jones, McNamara, & Suomi, 2000; Cumming, Kantor, Powers, Santos, & Taylor, 2000). To accomplish this, an evidencecentered design (ECD) approach was used (see, for example, Mislevy & Haertel, 2006; Mislevy, Steinberg, Almond, & Lukas, 2006). In short, ECD methodology entails• Looking at the population for which the test is intended and the uses to which the test will be put• Articulating the desired claims to be made about test takers based on their performance on the assessment;• Identifying test-taker behaviors that would allow these claims to be made• Creating (and evaluating) tasks to elicit these behaviors, thus providing evidence to support the claims For the speaking measure, three hierarchical claims were specified-that test takers can:1. Create connected, sustained discourse appropriate to the typical workplace 2. Carry out routine social and occupational interactions such as giving and receiving directions, asking for information, and asking for clarification 3. Produce some language that is intelligible to native and proficient non-native English- Next we invited TOEIC clients in Japan and Korea to r...

No abstract

The paper applies a natural language computational tool to study a potential construct‐irrelevant response strategy, namely the use of shell language. Although the study is motivated by the impending increase in the volume of scoring of students responses from assessments to be developed in response to the Race to the Top initiative, the data for the study were obtained from the GRE® Analytical Writing measure. The functioning of the shell detection computational tool was first evaluated by applying it to a corpus of over 200,000 issue and argument essays and by means of a study to evaluate whether the shell language score agreed with the characterization of shell by two scoring experts. It was concluded that the computational tool worked well. The tool was then used to select essays for rescoring to determine whether the presence of shell language had had an effect on the operational scores they received. We found no evidence that such an effect was present. However, we did find a leniency effect in the operational scores. That is, the essays that were rescored as part of this project received a lower score compared to the operational score. The validity implications of these results are discussed.

Paper-pencil teacher tests of professional knowledge typically are organized around the conventional dimensions that define state teaching standards, such as classroom management and assessment. We examined if such tests could be partitioned into three other dimensions (general knowledge, academic or textbook knowledge, and functional teaching knowledge), and if these dimensions (1) identified between those who did and did not receive teacher preparation, and (2) predicted teaching competence. A teacher test was given to advanced and beginning education majors, and non-majors. We found that advanced education majors and non-majors differed mostly on academic knowledge, but that functional knowledge predicted competence to the greatest extent. A variety of useful information can be gleaned from teacher tests if subscores of these dimensions are generated.

District‐level policymakers are challenged to use evidence of student achievement to make policy decisions, such as professional development and other school improvement plans. They currently receive reports of student achievement data that are complex, difficult to read, and even harder to interpret. Using the research literature on policymakers' use of data and conducting focus groups and interviews, we elicited information on their roles and responsibilities, as well as questions these people would like to have answered from achievement data. We propose an evidence‐centered reporting framework to help policymakers determine which data they need, in order to design a series of reports that will answer their questions and to help them make sense of the data in support of policy decisions.