Academic English Proficiency Assessment Using a Computerized Adaptive Test

Abstract

This paper describes the steps to convert a paper-and-pencil English proficiency test for academic purposes, consisting of multiple choice items administered following the Admissible Probability Measurement Procedure [24], adopted by the graduate program at the Institute of Mathematics and Computer Sciences at the University of São Paulo (ICMC-USP), Brazil, to a computerized adaptive test (CAT) based on an Item Response Theory Model (IRT). Despite the fact that the program accepts various internationally recognized tests that attest non-native speakers English proficiency, such as the Test of English as a Foreign Language (TOEFL), the International English Language Testing System (IELTS) and the Cambridge English: Proficiency (CPE), for instance, its requirement is incoherent in public universities in Brazil due to the cost, which ranges from US$ 200.00 to US$ 300.00 per exam. The TAI-PI software (Computerized Adaptive Test for English Proficiency), which was developed in Java language and SQLite, started to be used to assess the English pro?ciency of students on the program from October, 2013. The statistical methodology used was defined considering the history and aims of the test and adopted Samejima's Graded Response Model [21], the Kullback-Leibler information criterion for item selection, the a posteriori estimation method for latent trait [2] and the Shadow Test approach [29] to impose restrictions (content and test length) on the test composition of each individual. A description of the test design, the statistical methods used, and the results of a real application of TAI-PI for graduate students are presented in this paper, as well as the validation studies of the new methodology for pass or fail classification, showing the good quality of the new evaluation system and examination of improvement using the IRT and CAT methods.

Measuring language ability of students with compensatory multidimensional CAT: A post-hoc simulation study

“…Compared to the average test length of the English Proficiency Test (EPT), which ranges from 65 to 75, administering the EPT in MCAT format caused approximately 60% to 65% decrements in test length since the average test length of the CAT version of EPT is equal to 28 with content balancing. This finding is supported by the study conducted by Curi and Silvia (2019) in which a 25-item test was considered sufficient enough to estimate the ability scores of candidates. Similarly, a test with 25 items on average in the context of CAT has been proposed by Van der Linden and Pashley (2010).…”

“…Due to the adaptive nature of the measurement process of CAT designs, the very easy and difficult items are eliminated for each test-taker which decreases the test length and performance times (Curi & Silvia, 2019;Sukamolson, 2002). Thus, CATs are assumed to be advantageous compared to traditional paper-pencil exams.…”

“…The main reason behind this is that the items that provide the highest information at the currently estimated ability are selected due to the adaptive nature of the process. Besides, CATs are more flexible to testing time and have the feature of providing results of the test as soon as the testing process is terminated (Curi & Silvia, 2019;Lin, 2012;Weiss, 1983). Moreover, CATs based on item response theory (IRT) provide comparable ability scores between the test takers answering a different set of items and taking the test at different times (Curi & Silvia, 2019;Kreitzberg et al, 1978;Wainer, 2000).…”

Teste Adaptativo Multiestágio para o ENEM

“…Entretanto, isso pode causar um problema no balanceamento do conteúdo dos itens, conforme o examinado os responde, pode ser que ele tenha um comportamento que leve a responder itens repetidos do mesmo conteúdo, ou pode ser que um determinado conteúdo sequer seja apresentado em sua prova. Para contornar esse problema, seria ideal ter um banco de itens que satisfaça o propósito do teste, com conteúdo variado e com itens com níveis de dificuldades abrangentes, além de critérios e algoritmos de seleção que incorporem restrições baseadas em especificações pedagógicas (como conteúdo e exposição dos itens) (Silva et al, 2019).…”

IRT-based classification analysis of an English language reading proficiency subtest

“…Dimova et al (2020) have argued that locally developed language proficiency tests reflect institutional learning objectives, and may therefore be particularly well suited to fulfill important ancillary placement and diagnostic functions in language programs. However, researchers have commented on the high probability of misclassification involved in traditional PBTs and suggested that computer adaptive testing (CAT) may reduce misclassification by identifying the most informative items in an item bank to increase discrimination around cutoff points, and hence enhance the validity of test-based classification decisions (Curi & Silva, 2019; Mizumoto et al, 2019; Rudner & Guo, 2011; Zhang, 2010). The purpose of the current study is to investigate the potential application of CAT in this context by comparing the classification performance of CAT and paper-based testing (PBT) versions of an English language proficiency reading subtest developed and administered at a Turkish university.…”