Putting testing researchers to the test: An exploratory study on the TOEFL iBT

DeLuca, Christopher; Cheng, Liying; Fox, Janna; Doe, Christine; Li, Miao

doi:10.1016/j.system.2013.07.010

Cited by 13 publications

(19 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As discussed in the previous section, actual language users in the target domain can be involved for evaluating test items during the validation such as Brown's (1993) study. In addition, including multiple stakeholders' perspectives into validation has been advocated by scholars in language testing (e.g., Bachman & Palmer, 1996Cheng & DeLuca, 2011;DeLuca et al, 2013;Fox & Cheng, 2007Hamp-Lyons, 2000;Moss et al, 2006;Shohamy, 2001).…”

Section: Discussionmentioning

confidence: 99%

“…Recently, scholars in language testing (e.g., Bachman & Palmer, 1996Cheng & DeLuca, 2011;DeLuca, Cheng, Fox, Doe, & Li, 2013;Fox & Cheng, 2007Hamp-Lyons, 2000;Moss, Girard, & Haniford, 2006;Shohamy, 2001) have advocated collecting stakeholders' perspectives for validation. The AUA (Bachman & Palmer, 2010) requires validation for both intended and actual score meaning and score use.…”

Section: Empirical Studies Using An Argument-based Approachmentioning

confidence: 99%

See 1 more Smart Citation

Critical review of validation models and practices in language testing: their limitations and future directions for validation research

Im¹,

Shin

Cheng³

2019

Lang Test Asia

Self Cite

View full text Add to dashboard Cite

Purpose and background: The purpose of this paper is to critically review the traditional and contemporary validation frameworks-the content, criterion, and construct validations; the evidence-gathering; the socio-cognitive model; the test usefulness; and an argument-based approach-as well as empirical studies using an argument-based approach to validation in high-stakes contexts to discuss the applicability of an argument-based approach to validation. Chapelle and Voss (2014) reported that despite the usefulness and advantages of an argument-based approach for test validation, five validation studies using this approach were found in a search from two major journals-Language Testing and Language Assessment Quarterly. We reviewed the validation approaches in language testing and extended the search for empirical studies that used an argument-based approach in five language testing journals including ProQuest Dissertation and Theses. By doing so, this paper aims to provide validation researchers with each approach's conceptual limitations and future directions for validation research. For validity arguments to be defensible, this paper suggests that various validity evidences be required, involving multiple test stakeholders. Implications: By comparing variations of an argument-based approach and reviewing eight representative studies out of 33 empirical validation studies using an argument-based approach, this paper presents the following implications for future researchers to consider: (a) defining test constructs and relevant test tasks through domain analysis; (b) inviting multiple test stakeholders to test validation; (c) investigating the intended and actual interpretations, decisions, and consequences; (d) considering social, cultural, and political values to be embedded; and (e) employing multiple methods beyond statistical analyses using test scores.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Empirical Studies Using An Argument-based Approachmentioning

confidence: 99%

Critical review of validation models and practices in language testing: their limitations and future directions for validation research

Im¹,

Shin

Cheng³

2019

Lang Test Asia

Self Cite

View full text Add to dashboard Cite

show abstract

“…The importance of having testing researchers and practitioners working with language tests has been demonstrated in recent studies (e.g., DeLuca, Cheng, Fox, Doe, & Li, 2013;Fox & Cheng, 2015). For example, the situation regarding the testing environment (e.g., the different pace of test-takers' completion of test sections, as Jack mentioned) would interfere with test-takers' perfor-mance on tasks and also on the test.…”

Section: Discussionmentioning

confidence: 99%

Chinese Students’ Perceptions of the Value of Test Preparation Courses for the TOEFL iBT: Merit, Worth, and Signi cance

Cheng²

2016

TESL Canada

Self Cite

View full text Add to dashboard Cite

Scriven's (1998Scriven's ( , 2007 La recherche dans le domaine de l'évaluation linguistique se penche de plus en plus sur la préparation aux examens d'anglais ā enjeux élevés. On connait mal, par contre, quels aspects de la préparation aux examens les étudiants trouvent importants et sur lesquels ils se concentrent. Dans cette étude, nous avons considéré les perspectives de 12 étudiants chinois inscrits à divers programmes acadé-miques dans une université canadienne et avons examiné leurs perceptions de la valeur des cours de préparation au TOEFL iBT qu'ils avaient passé avant d'être admis à l'université. Le modelé de Scriven (1998Scriven ( , 2007

show abstract

“…The 21st century has witnessed a growth in the use of computer facilitated techniques (Buckingham & Alpaslan, 2017;Stickler & Shi, 2016;Wagner, 2007) and integrative skills items in international English language examinations, such as TOEFL iBT (Deluca, Cheng, Fox, Doe, & Li, 2013) and the Pearson Test of English Academic (PTE Academic). Both features have been widely believed to make listening assessment more valid from various perspectives.…”

Section: Introductionmentioning

confidence: 99%

An investigation of integrative and independent listening test tasks in a computerised academic English test

Wei

Zheng

2017

Computer Assisted Language Learning

View full text Add to dashboard Cite

a computerized academic English testThis research provided a comprehensive evaluation and validation of the listening section of a newly introduced computerized test, Pearson Test of English Academic (PTE Academic). PTE Academic contains 11 item types assessing academic listening skills either alone or in combination with other skills. First, task analysis helped identify skills important for listening comprehension in academic settings. Aspects analyzed included the purpose of assessment tasks, skills/constructs assessed, and task stimuli employed in PTE Academic. The findings indicated that modern technologies enabled PTE Academic, a computer-based test, to assess students' academic listening abilities in real time using the integration of multi-modal sources. The statistical validation consisted of two stages. Exploratory factor analysis was performed first with a sample of over 5,000 students who took PTE Academic, to examine the underlying listening constructs as measured by the scores on the 11 item types item on different listening skills; these scores were subjected to Rasch analysis using CONQUEST. Second, the difficulties of the item types were estimated and the effectiveness of these item types was evaluated by calculating the information function by item type. The study has implications for test developers and test users regarding the interpretation of student performance on listening assessments.

show abstract

Putting testing researchers to the test: An exploratory study on the TOEFL iBT

Cited by 13 publications

References 36 publications

Critical review of validation models and practices in language testing: their limitations and future directions for validation research

Critical review of validation models and practices in language testing: their limitations and future directions for validation research

Chinese Students’ Perceptions of the Value of Test Preparation Courses for the TOEFL iBT: Merit, Worth, and Signi cance

An investigation of integrative and independent listening test tasks in a computerised academic English test

Contact Info

Product

Resources

About