Building an intelligent recommendation system for personalized test scheduling in computerized assessments: A reinforcement learning approach

Abstract

The introduction of computerized formative assessments in the classroom has opened a new area of effective progress monitoring with more accessible test administrations. With computerized formative assessments, all students could be tested at the same time and with the same number of test administrations within a school year. Alternatively, the decision for the number and frequency of such tests could be made by teachers based on their observations and personal judgments about students. However, this often results in rigid test scheduling that fails to take into account the pace at which students acquire knowledge. To administer computerized formative assessments efficiently, teachers should be provided with systematic guidance regarding effective test scheduling based on each student's level of progress. In this study, we introduce an intelligent recommendation system that can gauge the optimal number and timing of testing for each student. We discuss how to build an intelligent recommendation system using a reinforcement learning approach. Then, we present a case study with a large sample of students' test results in a computerized formative assessment. We show that the intelligent recommendation system can significantly reduce the number of testing for the students by eliminating unnecessary test administrations where students do not show significant progress (i.e., growth). Also, the proposed recommendation system is capable of identifying the optimal test time for students to demonstrate adequate progress from one test administration to another. Implications for future research on personalized assessment scheduling are discussed.

Learning Analytics and Computerized Formative Assessments: An Application of Dijkstra’s Shortest Path Algorithm for Personalized Test Scheduling

“…The authors found that recommender system techniques such as matrix factorization and collaborative filtering outperformed the traditional regression methods in predicting student performance. Recently, more advanced algorithms and approaches such as reinforcement learning [11,19] have been proposed to improve the capacity, architectural flexibility, and performance accuracy of the recommender systems adopted within educational contexts.…”

“…The findings of their study indicated that the IRS could yield a significant reduction in the total number of test administrations required to make accurate decisions on students' academic growth. Similarly, Shin and Bulut [11] also proposed a recommender system using a reinforcement learning algorithm that aimed to optimize test administration schedules for students. The proposed system could identify the critical time points for students to demonstrate their academic growth.…”

“…The training set consisted of students who showed adequate growth in the assessments (hereinafter referred to as the exemplary group). To identify adequate growth, we used 2 rules: the student's Theil-Sen slope must be larger than the median value of all slope estimates in the sample and the student's final Star Reading score must be above the 25th percentile, which is considered a cut-off for identifying at-risk students in reading [11]. After applying these selection criteria, the grade 2 training sample included 276,087 students and the remaining 392,237 students were included in the test sample.…”

“…Results from previous studies suggested that teachers may need systematic guidance to determine an optimal assessment schedule based on each student's unique progress [10,11]. Although several researchers shared practical guidelines regarding the number, timing, and frequency of computerized formative assessments focusing on reading [8,12,13], there is no consensus on the number of test administrations and the testing frequency for computerized formative assessments.…”

“…Recent studies showed that intelligent recommender systems (IRSs) can be used to generate a personalized assessment schedule based on each student's progress [10,11]. The main goal of the IRS is to exploit existing information about items (or users) to provide suggestions of new items for users (e.g., what products to buy, what videos to watch, or what online news to follow) [14].…”