A Longitudinal Analysis of the Variability of Cognitive Complexity in IELTS Academic Writing Prompts: a mixed-method study

Abstract

Cognitive complexity of writing prompts may affect the linguistic performance of test takers in writing assessment, as evidenced by previous research. In this mixed method study, we aim to explore if and to what extent the perceived cognitive complexity of IELTS Writing Task II prompts has changed over time. We use official IELTS prompts for this study collected over test administrations years 1996 to 2022. For each test year, two prompts were randomly selected (total of 20 prompts for ten years) and were structured in a survey. Experienced IELTS teachers and assessment experts (n = 19) rated the perceived cognitive complexity of each prompt on 1-8 Likert scale. An intra-class correlation (ICC) coefficient showed a high degree of inter-rater reliability among the raters (r = 0.88, p = 0.00). A time series analysis showed that the perceived year-on-year cognitive complexity of the prompts showed significant negative change (b = -0.073, p=0.00). In the qualitative part of the study, the prompts with the lowest and highest cognitive complexity mean ratings were presented to the raters and inquired what made those prompts easy or difficult for test takers.