Integrated speaking tasks requiring test takers to read and/or listen to stimulus texts and to incorporate their content into oral performances are now used in large-scale, high-stakes tests, including the TOEFL iBT. These tasks require test takers to identify, select, and combine relevant source text information to recognize key relationships between source text ideas, and to organize and transform information. Despite being central to evaluations of validity, relationships between stimulus content, task demands, and the oral discourse produced by test takers are yet to be empirically scrutinized to an adequate degree. In this study, we focus on a TOEFL iBT reading–listening–speaking task, applying discourse analytic measures developed by Frost, Elder and Wigglesworth (2012) to 120 oral performances to examine (a) the integration of source text ideas by test takers across three proficiency levels, and (b) the appropriateness of content-related criteria in the TOEFL integrated speaking rubric. We then combine analyses of these aspects of performances with a qualitative analysis of the generic structure and semantic profiles of stimulus texts to explore relationships between stimulus text properties and oral performances. Findings suggest that the extent to which content-related rating scale criteria distinguish between proficiency levels is contingent on stimulus text properties, with important implications for construct definitions and task design.

No abstract

No abstract

The use of integrated tasks to test English speaking skills raises questions about the impact of comprehension on test score outcomes, and the impact of stimulus materials on test taker strategic behaviours. This study analysed speaking performances and verbal report data to examine the strategies used by test takers at different levels of proficiency to relate stimulus text information in response to a TOEFL-iBT reading-listening-speaking task. 120 speaking performances were analysed to identify how ideas from reading and listening source texts were reproduced, summarised and/or synthesised. Verbal reports from 38 test takers were also collected. Findings showed that test takers, regardless of proficiency, reproduced more single ideas from stimulus texts than summarised or synthesised information, although high proficiency test takers summarised information more than low proficiency test takers. Verbal report data further revealed that high proficiency test takers engage in effective summarisation and synthesising as part of note-taking while reading and listening, suggesting these are aspects of integrated speaking task constructs and supporting the relevance of integrated tasks for assessing entry to tertiary educational domains.