Diagnosing listening and reading skills in the Chinese EFL context: Performance stability and variability across modalities and performance levels

Exploring the relationship between metacognitive awareness and Chinese EFL learners’ listening skills

“…CDAs involve (1) analyzing the test items and identifying cognitive skills involved, and (2) mathematical modeling of test takers’ skill mastery patterns according to their responses to the test and the identified cognitive skills. Recent years have witnessed the development of a variety of cognitive diagnosis models (CDMs) and their application to educational assessment (e.g., Liu et al, 2018 ; Aryadoust, 2021 ; He et al, 2022 ; Min et al, 2022a , b ), including the rule space model, the tree-based regression, the FUSION Model, G-DINA, ACDM, C-RUM, DINO, DINA, RRUM, G-DINA, the deep CDM, the sequential hierarchical CDM, and the semi-supervised learning ANN for diagnostic classification ( Zhang and Wang, 2020 ; Xue and Bradshaw, 2021 ; Gao et al, 2022 ).…”

“…Non-compensatory CDAs, such as the rule space model ( Tatsuoka, 1990 ) and the tree based regression ( Sheehan, 1997 ), were adopted in early listening assessment research for creating and evaluating a Q-matrix, which characterizes the relationship between test items and subskills. Other models are applied to analyze test response data associated with a given Q-matrix (e.g., Lee and Sawaki, 2009 ; Aryadoust, 2011 ; Yi, 2017 ; Min et al, 2022a ; He et al, 2022 ) and examine skill-mastery classification results.…”

“…Using the performance data of two cognitive diagnostic models in an in-house EFL listening test, Min et al (2022b) identified four attributes: listening for words, details, intended meaning, and main idea. He et al (2022) explored the similarities and differences in EFL learners’ use of local and global subskills, based on listening and reading test data from a large-scale English in-house test for students who intend to get their bachelor’s degree. The subskills identified in this study are: linguistic knowledge, details, synthesizing, and inferencing.…”

Cognitive diagnostic assessment of EFL learners’ listening barriers through incorrect responses

“…Though listening is a major language skill (Vandergrift and Goh, 2012;Rost, 2016), it is still the thinly profiled, least understood and particularly under-researched skill (Vandergrift and Goh, 2012;Wolf et al, 2019;He et al, 2022), and the learners' underlying cognitive process is even less addressed (Buck, 2001;Harding et al, 2015). The same is true of Chinese college EFL learners (Xu and Nie, 2016), as a result, understanding their cognitive processes has been a focus in EFL listening research.…”

“…Cognitive Diagnostic Assessment (CDA), as a new generation of measurement, has demonstrated its advantages in combining cognitive process and psychometrics (Leighton and Gierl, 2007). It can not only reveal learners' cognitive processes and mastery or non-mastery of subskills intended by test items (Sawaki et al, 2009), but CDA also helps provide tailored feedback for subsequent remediation and guidance (Yi, 2017;He et al, 2022). This can be achieved by adopting appropriate cognitive diagnostic models (CDMs), which are latent class models used for classifying students based on their skill profiles.…”