Development and validation of virtual interactive tasks for an aviation English assessment

Abstract

for their time, support, motivation, and help whenever necessary in every step of my academic journey. My sincere appreciation is extended to the military air traffic controllers in Korea for expressing their enthusiasm for and willingness to participate in this research. Very special thanks go to CSM Jewon Ryu in the 55 th Air Traffic Service Battalion, who helped me in various ways in both the pilot study and dissertation study. I also extend my thanks to Dr. Byeong-Young Cho at University of Pittsburgh for his friendship and insights into the world of strategic competences and Dr. Alistair Van Moere at Pearson for providing me with Versant Aviation English Tests for this dissertation research. I would also like to acknowledge my PhD colleagues and faculty and staff in the English Department at Iowa State University for their continued support, care, and collaboration. I would further like to thank my colleagues in the Intensive English and Orientation Program (IEOP) at ISU and the Center for Language Research at University of Aizu in Japan for their unwavering encouragement and generous support. My heartfelt thanks go to my sister's family-Seoyoung, Joonyoung, and Dongjoofor their love, support, and laughter when I needed it most. I am deeply indebted to my parents who taught me the value of hard work and perseverance, not to mention their love and prayers. This dissertation is dedicated to my two professors at Keimyung University in Korea, Richard Hark and Yongsang Cho, who had been my greatest inspirations and role models since 1996, but sadly passed away soon after I began the doctoral program. I believe they would be most pleased to see the completion of my PhD journey.

Current Uses for Task-Based Language Assessment

“…It should be clear that (a) accomplishing this kind of task is not a reasonable expectation for most language learners, but (b) for those who expect to use language in high-stakes settings like that of air traffic or marine communication, task-based assessments of this sort are probably essential for tapping into the likelihood of a candidate's abilities to use the language on the job (Alderson, 2009). Indeed, according to the International Civil Aviation Organization PRICE Study Group (2007), “Because of the high stakes involved, pilots and controllers deserve to be tested in a context similar to that in which they work and test content should therefore be relevant to their roles in the work-place.” For this reason, task-based assessments have been developed to replicate the demands of these and similar high-stakes professional environments that require language use under very specific circumstances (for one recent example of a task-based assessment for aviation English, see Park, 2015).…”

Development and evaluation of a spoken dialog system-mediated paired oral task for measuring second language oral communication ability in English

“…Over the last two decades, the argument-based approach to validation has been used in many studies in the field of language assessment (e.g., Chapelle et al, 2008Chapelle et al, , 2015Chapelle et al, , 2018Frost, Elder, & Wigglesworth, 2012;LaFlair & Staples, 2017;Li, 2015;Llosa, 2008;Park, 2015;Yan & Staples, 2019;Youn, 2015). These studies indicate the usefulness of the "conceptual tools"…”

“…This question was addressed by analyzing holistic scores assigned to the oral task responses of 30 test takers by three raters using the rating scale. Interrater reliability of the scores was estimated using a two-way mixed average measure intraclass correlation coefficient (ICC) for absolute agreement (Shrout & Fleiss, 1979;McGraw & Wong, 1996), as in other studies on L2 oral assessment (e.g., Lee, 2015;Park, 2015). Among different types of ICC, this type was chosen for the study because (a) the same set of raters rated all task responses, (b) raters involved in the analysis were the only raters of interest (i.e., not randomly selected from a large population), (c) the mean of ratings by multiple raters is used as the basis of the assessment, and (d) the absolute agreement among raters is of importance (Shrout & Fleiss, 1979;McGraw & Wong, 1996).…”

Development and validation of spoken dialog system-based oral communication tasks in an ESP context

“…For example, Kim and Elder (2015) surveyed Korean airline pilots and air traffic controllers through 400 questionnaires and 22 interviews to elicit the underlying construct of both English proficiency tests for pilot and air traffic controller radiotelephony communication. Similarly, benefitting from his military service experience, Park (2015) surveyed and interviewed 20 military air traffic controllers to define the construct of Aviation English and elicit test tasks for his virtual Aviation English assessment. In the domain of English for Health Professionals, researchers recruit the involvement of different stakeholders to help understand the underlying construct in several studies (e.g., Davidson, 2021;Macqueen et al, 2016;Pill & McNamara, 2016).…”

“…ECD has been successfully used in several research studies that develop and investigate innovative test tasks (e.g., Banerjee, 2019;Choi, 2018;Lee, 2015;Park, 2015). In this dissertation study, an ECD framework was used to provide a solid foundation for test design and development based on Mislevy et al (2003).…”