The Classical Test or Item Response Measurement Theory: The Status of the Framework at the Examination Council of Lesotho

Abstract

While the Examination Council of Lesotho (ECOL) is burdened with a huge workload of assessment tasks, their procedures for developing tests, analysing items, and compiling scores heavily rely on the classical test theory (CTT) measurement framework. The CTT has been criticised for its flaws, including being test-oriented, sample dependent, and assuming linear relationships between latent variables and observed scores. This article presents an overview of CTT and item response theory (IRT) and how they were applied to standard assessment questions in the ECOL. These theories have addressed measurement issues associated with commonly used assessments, such as multiple-choice, short response, and constructed response tests. Based on three search facets (Item response theory, classical test theory, and examination council of Lesotho), a comprehensive search was conducted across multiple databases (such as Google Scholar, Scopus, Web of Science, and PubMed). The paper was theoretically developed using the electronic databases, keywords, and references identified in the articles. Furthermore, the authors ensure that the keywords are used to identify relevant documents in a wide variety of sources. A general remark was made on the effective application of each model in practice with respect to test development and psychometric activities. In conclusion, the study recommends that ECOL switch from CTT to modern test theory for test development and item analysis, which offers multiple benefits.

Exploring the Accuracy and Consistency of a School Readiness Assessment Tool for Preschoolers: Reliability, Validity and Measurement Invariance Analysis

“…It analyzes the reliability, validity, and sources of measurement error, with the true score representing the individual's actual ability and the measurement error reflecting the variability in observed scores unrelated to the true score (Haw et al 2022). However, CTT does not account for item difficulty or variability in individual differences in ability levels (Ayanwale et al 2022) and MI testing (Siregar and Panjaitan 2022). IRT is a modern approach to psychometric measurement that models the relationship between a person's ability level and their responses to test items (Polat et al 2022).…”

A comparative study of AI-human-made and human-made test forms for a university TESOL theory course

“…This decision is based on the administration of two achievement test forms to the same students at one time. Furthermore, given that classroom teachers can easily utilize standard statistical software for conducting analyses, CTT remains widely used in education and psychology due to its practicality (Ayanwale et al, 2022;De Champlain, 2010).…”

The feasibility of computerized adaptive testing of the national benchmark test: A simulation study

“…The item response theory [IRT] framework is currently used by most CAT systems. IRT is a set of algorithms that measure each item's characteristic on a scale, which corresponds to an individual's traits (Ayanwale, 2019;Ayanwale et al, 2022;van der Linden & Glas, 2010). In IRT models such as one-parameter (which looks at the difficulty -b of the item), two-parameter (which looks at the discrimination-a after item difficulty parameter is computed), three-parameter (which looks at the guessing-c in addition to b and a), and four-parameters (which looks at the carelessness-d in addition to b, a and c) the examinee's behaviour is taken into account at the item level (Ayanwale et al, 2018(Ayanwale et al, , 2019Baker, 2004).…”