Applying adaptive variables in computerised adaptive testing

Abstract

Current research in computerised adaptive testing (CAT) focuses on applications, in small and large scale, that address self assessment, training, employment, teacher professional development for schools, industry, military, assessment of non-cognitive skills, etc. Dynamic item generation tools and automated scoring of complex, constructed response examinations are coming into use. Therefore it is important to extend CAT's functionality to include more variables in its student model that define the examinee as an individual beyond the mastery level, for improved performance and more efficient test delivery. This paper examines variables that can prompt adaptation and discusses their potential use in a hypothetical student model for CAT. The objective of this effort is to provide researchers, designers, and developers of CAT a perspective for exploiting research outcomes from the area of personalised hypermedia applications. IntroductionDue to the advances in communication and information technology, the popularity of computer based testing has increased in recent years. Computer delivery of tests has become feasible for processes such as licensure, certification and admission. Moreover, computers can be used to increase the statistical accuracy of test scores using computerised adaptive testing (CAT). As an alternative to giving each examinee the same fixed test, CAT item selection adapts to the ability level of individual examinees, and after each response the ability estimate is updated and the next item is selected to have optimal properties at the new estimate (van der Linden & Glas, 2003). The computer continuously re-evaluates the ability of the examinee until the accuracy of the estimate reaches a statistically acceptable level or when some limit is reached, such as a maximum number of test items presented. The score is determined from the level of Triantafillou, Georgiadou and Economides 351the difficulty, and as a result, while all examinees may answer the same percentage of questions correctly, high ability examinees will attain a better score as they answer correctly more difficult items. The vast majority of CAT systems rely on Item Response Theory as the underlying model (Lord, 1980;Wainer, 1990). However, Decision Theory provides an alternative underlying model for sequential testing (Rudner, 2002), and Knowledge Space Theory (Doignon & Falmagne, 1985) is a third basis for small scale construction of adaptive tests.Regardless of some disadvantages reported in the literature, for example, high cost of development, item calibration, item exposure control (Eggen, 2001;Boyd, 2003), effect of a flawed item (Abdullah, 2003), or the use of CAT for summative assessment (Lilley & Barker, 2002, CAT has several advantages. Testing on demand can be facilitated, so an examinee can take the test whenever and wherever he or she is ready. Multiple media can be used to create innovative item formats and more realistic testing environments. Other possible advantages are flexibility of test management, immediate avai...

Adaptation criteria for the personalised delivery of learning materials: A multi-stage empirical investigation

“…Here, the paper makes a significant contribution to the initial research of Triantafillou et al (2007) by considering 30 adaptive systems and by validating the set of criteria through experts, in two online surveys. The authors' ratings for each system and the set of criteria provided the primary focus of the first survey.…”

“…The statements differ in the level of detail provided and focus only on the respective perspectives of specific adaptive systems. A more comprehensive presentation can be found in Triantafillou, Georgiadou and Economides (2007), where aspects from five different papers (including two from the same author) are reviewed and described. The aim of the study by Triantafillou et al (2007) was to identify features for adaptation, which can primarily be used in user modelling and adaptive testing.…”

“…In light of the broad spectrum of adaptive systems and the number of authors, an investigation which considers a larger and more diverse sample seems useful. Hence, the main objectives of this paper, following on from Triantafillou et al (2007), are to investigate in greater detail, and empirically, the adaptation needs useful for informing the adaptive delivery of learning objects. …”

“…A more comprehensive presentation can be found in Triantafillou, Georgiadou and Economides (2007), where aspects from five different papers (including two from the same author) are reviewed and described. The aim of the study by Triantafillou et al (2007) was to identify features for adaptation, which can primarily be used in user modelling and adaptive testing. In light of the broad spectrum of adaptive systems and the number of authors, an investigation which considers a larger and more diverse sample seems useful.…”

A recommender framework for the evaluation of end user experience in adaptive technology enhanced learning

“… Measurement criteria (adaptive variables): Adaptive variables refer to features of the user that are used as a source of the adaptation (Triantafillou et al, 2007). In total, 50 variables were mentioned in the studies and were grouped into different categories of variables concerning (i.e.…”