Innovative assessment tasks for academic English proficiency: an integrated listening-speaking task vs. a multimedia-mediated speaking task

Development and evaluation of a spoken dialog system-mediated paired oral task for measuring second language oral communication ability in English

“…The specifications were adapted from Bachman and Palmer's (2010) framework of language task characteristics and Pica et al's (1993) framework of communication tasks. Bachman and Palmer's framework was chosen because of its successful and widespread use in language assessment (e.g., Jia, 2013;Lee, 2015). Pica et al's framework was employed as it is particularly suited to characterize the interactional features of paired/group oral tasks (e.g., Fulcher, 2003a;Nakatsuhara, 2009;Van Moere, 2007), as explained in Chapter 2: Problem Analysis.…”

“…This question was addressed by analyzing holistic scores assigned to the oral task responses of 30 test takers by three raters using the rating scale. Interrater reliability of the scores was estimated using a two-way mixed average measure intraclass correlation coefficient (ICC) for absolute agreement (Shrout & Fleiss, 1979;McGraw & Wong, 1996), as in other studies on L2 oral assessment (e.g., Lee, 2015;Park, 2015). Among different types of ICC, this type was chosen for the study because (a) the same set of raters rated all task responses, (b) raters involved in the analysis were the only raters of interest (i.e., not randomly selected from a large population), (c) the mean of ratings by multiple raters is used as the basis of the assessment, and (d) the absolute agreement among raters is of importance (Shrout & Fleiss, 1979;McGraw & Wong, 1996).…”

Development and validation of spoken dialog system-based oral communication tasks in an ESP context

“…ECD has been successfully used in several research studies that develop and investigate innovative test tasks (e.g., Banerjee, 2019;Choi, 2018;Lee, 2015;Park, 2015). In this dissertation study, an ECD framework was used to provide a solid foundation for test design and development based on Mislevy et al (2003).…”