Validity of the listening module of international English language testing system: multiple sources of evidence

Abstract

Background: A critical issue in International English Language Testing System (IELTS) lies in the significance of the validity of IELTS listening comprehension test (hereafter IELTS LCM). However IELTS listening validity has been investigated, it has not been investigated with reference to multiple sources of evidence regarding item internal factors. To bridge this gap, we investigated its construct validity with use of structural equation modelling (SEM) and assessed differential item functioning (DIF) through cognitive diagnostic modelling (CDM) and Mantel Haenszel (MH). Methods: In this study, first, the participants signed a consent form for participation in the study; then, 480 participants were administered a proficiency test designed by the university of Cambridge; next, out of 480 participants, 463 participants were administered a 40-item IELTS LCT developed by the University of Cambridge. Finally, the data were analyzed with use of LISREL for probing the construct validity of the test; also, for detecting the potential DIF items, MH and CDM were used to make the results of DIF related findings more reliable. Results: The results of the first study confirmed an appropriate model fit, so that all four constructs, i.e., gap filling, diagram labelling, multiple choice and short answer on IELTS LCT, had a statistically significant contribution to IELTS LCT. However, construct-related evidence may not lead to the whole validity. This given, the second study examined the DIF items to argue the validity of IELTS LCT: MH detected 15 DIF items and CDM detected at least 6 DIF items and at most 12 DIF items. Conclusions: Due to its international nature and worldwide evaluative contribution, IELTS needs to have approximately (not absolutely) a stable factor structure, so that it should be invariant across populations and various cultures. More naturally, a test highly valid in one context might suffer from some degree of invalidity with some related constructs in another context. This in mind, our perspective in this research is not recommended to be taken as a one-size-fits-all model: Neither generalization nor claim is made based on the present study.

Exploring the relationships between various dimensions of receptive vocabulary knowledge and L2 listening and reading comprehension

“…The listening and reading comprehension tests were actual IELTS tests administered in accordance with Cambridge ESOL examination's guidelines. Although the statistics were indeed lower than the average Cronbach' Alpha of 0.88 recorded from the performance of more than 90,000 IELTS examinees by The University of Cambridge Local Examination Syndicate (UCLES) (2007, cited in Hashemi & Daneshfar, 2018), the reliability coefficients were within an acceptable range (0.7 or above) (Alavi et al, 2018;Pallant, 2010;Phakiti, 2016). Moreover, researchers in the field reported even lower reliability coefficients of 0.6 (Staehr, 2009) and 0.7 (Feng, 2016) for standardized international tests provided by Cambridge ESOL examination.…”

Vocabulary Demands of the IELTS Listening Test: An In-Depth Analysis

“…Evidence for English language proficiency could be obtained by taking the International English Language Testing System (IELTS) which was jointly developed and managed by the British Council and IDP Australia (Pearson, 2019). In the four sub-tests of the IELTS, the listening comprehension subtest have been viewed as intensive and overwhelming for test takers due to its one-time and read-listen-write format (Alavi et al, 2018; Field, 2005). Taking the listening part in such format means that test takers have to focus simultaneously on three language skills: listening, reading, and writing, which is really demanding in terms of cognitive processing (Aryadoust, 2012; Alavi et al, 2018).…”

“…In the four sub-tests of the IELTS, the listening comprehension subtest have been viewed as intensive and overwhelming for test takers due to its one-time and read-listen-write format (Alavi et al, 2018; Field, 2005). Taking the listening part in such format means that test takers have to focus simultaneously on three language skills: listening, reading, and writing, which is really demanding in terms of cognitive processing (Aryadoust, 2012; Alavi et al, 2018).…”

A Rasch-based validation of the Vietnamese version of the Listening Vocabulary Levels Test

“…The IELTS was jointly developed by the British Council, The University of Cambridge Local Examination Syndicate (UCLES), and IDP Education Australia (Pearson, 2019;Quaid, 2018). There were four parts in the IELTS listening test: parts 1 and 2 included a conversation and a prompted monologue with transactional purposes and parts 3 and 4 consisted of a discussion dialogue and a monologue in academic contexts (Alavi et al, 2018;Phakiti, 2016). Cronbach's alpha reliability coefficient for the IELTS listening test was .805, which was high and strongly confirmed sound internal consistency.…”