Quality meets Diversity: A Model-Agnostic Framework for Computerized Adaptive Testing

Abstract

Computerized Adaptive Testing (CAT) is emerging as a promising testing application in many scenarios, such as education, game and recruitment, which targets at diagnosing the knowledge mastery levels of examinees on required concepts. It shows the advantage of tailoring a personalized testing procedure for each examinee, which selects questions step by step, depending on her performance. While there are many efforts on developing CAT systems, existing solutions generally follow an inflexible model-specific fashion. That is, they need to observe a specific cognitive model which can estimate examinee's knowledge levels and design the selection strategy according to the model estimation. In this paper, we study a novel model-agnostic CAT problem, where we aim to propose a flexible framework that can adapt to different cognitive models. Meanwhile, this work also figures out CAT solution with addressing the problem of how to generate both high-quality and diverse questions simultaneously, which can give a comprehensive knowledge diagnosis for each examinee. Inspired by Active Learning, we propose a novel framework, namely Model-Agnostic Adaptive Testing (MAAT) for CAT solution, where we design three sophisticated modules including Quality Module, Diversity Module and Importance Module. Specifically, at one CAT selection step, Quality Module first quantifies the informativeness of questions and generates candidate subset with the highest quality. Then, Diversity Module selects one question at each step that maximizes the concept coverage. Additionally, we propose Importance Module to estimate the importance of concepts that optimizes the CAT selection. Under MAAT, we prove that the goal of maximizing both quality and diversity is NP-hard, but we provide efficient algorithms by exploiting the inherent submodular property. Extensive experimental results on two real-world datasets clearly demonstrate that our MAAT can support CAT with guaranteeing both quality and diversity perspectives.