Beyond item analysis: Connecting student behaviour and performance using e‐assessment logs

Abstract

Traditional item analyses such as classical test theory (CTT) use exam‐taker responses to assessment items to approximate their difficulty and discrimination. The increased adoption by educational institutions of electronic assessment platforms (EAPs) provides new avenues for assessment analytics by capturing detailed logs of an exam‐taker's journey through their exam. This paper explores how logs created by EAPs can be employed alongside exam‐taker responses and CTT to gain deeper insights into exam items. In particular, we propose an approach for deriving features from exam logs for approximating item difficulty and discrimination based on exam‐taker behaviour during an exam. Items for which difficulty and discrimination differ significantly between CTT analysis and our approach are flagged through outlier detection for independent academic review. We demonstrate our approach by analysing de‐identified exam logs and responses to assessment items of 463 medical students enrolled in a first‐year biomedical sciences course. The analysis shows that the number of times an exam‐taker visits an item before selecting a final response is a strong indicator of an item's difficulty and discrimination. Scrutiny by the course instructor of the seven items identified as outliers suggests our log‐based analysis can provide insights beyond what is captured by traditional item analyses. What is already known about this topic Traditional item analysis is based on exam‐taker responses to the items using mathematical and statistical models from classical test theory (CTT). The difficulty and discrimination indices thus calculated can be used to determine the effectiveness of each item and consequently the reliability of the entire exam. What this paper adds Data extracted from exam logs can be used to identify exam‐taker behaviours which complement classical test theory in approximating the difficulty and discrimination of an item and identifying items that may require instructor review. Implications for practice and/or policy Identifying the behaviours of successful exam‐takers may allow us to develop effective exam‐taking strategies and personal recommendations for students. Analysing exam logs may also provide an additional tool for identifying struggling students and items in need of revision.

Identification of parameters for electronic distance examinations

“…Additionally, it was shown that MCQ items with a higher character count tend to have a poorer discrimination index, but that the character count has no effect on difficulty. These effects have been described in other studies that also concluded that examinees require more time for poorly discriminating items ( 30 , 46 ) as well as for difficult items ( 46 , 47 ). As a result, to improve discrimination and reduce required response time, items should be kept as clear and concise as possible, which aligns with the formal requirements of multiple-choice items in the literature ( 12 , 28 , 48–52 ).…”

Adoption of learning analytics in higher education institutions: A systematic literature review

“…From a broader perspective, there is a strong match between the critical factor mentioned in the literature and the top three most priority factors raised by LA experts in previous work (see Chevreux et al, 2020 and Figure 3). In this vein, recently published studies such as Cukurova et al (2023) and Lahza et al (2023) also take into account the aforementioned factors, but from a teacher and technical perspective and not from the institutional point of view. For instance, these studies mention teachers' workload, trust and ownership by teachers, guidance, professional development, support, classroom orchestration, privacy concerns, technical infrastructure, data collection protocols and type of data analysis, all of them match with our 14 factors.…”

Learning analytics and assessment: Emerging research trends, promises and future opportunities

“…The authors tested the model using student trace‐data and assessment data collected in an LMS site of an authentic mathematics course. Lahza et al (2022) examined exam‐taker behaviours from exam logs extracted from electronic assessment platforms, thus adding to traditional methods in item analysis. The authors demonstrated that this approach can be used to determine the effectiveness of test items and identify the items that need to be revised to improve the overall reliability of an exam.…”