Multi-objective optimization of item selection in computerized adaptive testing

Mujtaba, Dena F.; Mahapatra, Nihar R.

doi:10.1145/3449639.3459334

Cited by 5 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The idea is to lower the burden of students taking tests by only asking them a subset of questions from the entire pool. There have been few main directions of solutions: model-agnostic strategies for selection (Bi et al, 2020), bi-level optimization (Ghosh and Lan, 2021;Zhuang et al, 2022;Feng et al, 2023), multi-objective optimization (Mujtaba and Mahapatra, 2021;Huang et al, 2019;, retrieval-augmented adaptive search .…”

Section: Extended Related Workmentioning

confidence: 99%

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Prabhu

Torr

Dokania

2020

Lecture Notes in Computer Science

347

283

View full text Add to dashboard Cite

Standardized benchmarks drive progress in machine learning. However, with repeated testing, the risk of overfitting grows as algorithms overexploit benchmark idiosyncrasies. In our work, we seek to mitigate this challenge by compiling ever-expanding large-scale benchmarks called Lifelong Benchmarks. As exemplars of our approach, we create Lifelong-CIFAR10 and Lifelong-ImageNet, containing (for now) 1.69M and 1.98M test samples, respectively. While reducing overfitting, lifelong benchmarks introduce a key challenge: the high cost of evaluating a growing number of models across an ever-expanding sample set. To address this challenge, we also introduce an efficient evaluation framework: Sort & Search (S&S), which reuses previously evaluated models by leveraging dynamic programming algorithms to selectively rank and sub-select test samples, enabling cost-effective lifelong benchmarking. Extensive empirical evaluations across ∼31,000 models demonstrate that S&S achieves highly-efficient approximate accuracy measurement, reducing compute cost from 180 GPU days to 5 GPU hours (∼1000x reduction) on a single A100 GPU, with low approximation error. As such, lifelong benchmarks offer a robust, practical solution to the "benchmark exhaustion" problem.

show abstract

Section: Extended Related Workmentioning

confidence: 99%

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Prabhu

Torr

Dokania

2020

Lecture Notes in Computer Science

347

283

View full text Add to dashboard Cite

show abstract

“…To address this, some researchers have tried to approximate the original objective. Mujtaba and Mahapatra [118] use the standard error of measurement (SEM) to approach the optimization objective. The SEM provides a measure of confidence in an estimate from a test.…”

Section: Subset Selection Algorithmsmentioning

confidence: 99%

Geography of Technology Transfer in China

Liu

2023

East China Normal University Scientific Reports

View full text Add to dashboard Cite

Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks (GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily relies on a large amount of task-specific supervision. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt. CCS CONCEPTS• Computing methodologies → Learning latent representations; • Information systems → Data mining.

show abstract

“…Multi-Objective Optimization aims to reach Pareto Optimality while optimizing multiple objectives simultaneously [27]. Multi-objective problems can be solved by various methods, such as genetic algorithms [27], evolutionary algorithms [26] or Multi-Objective RL algorithms [25]. In CAT, Mujtaba and Mahapatra [26] proposed optimizing test length and accuracy by a multi-objective evolutionary algorithm.…”

Section: Multi-objective Optimizationmentioning

confidence: 99%

“…Multi-objective problems can be solved by various methods, such as genetic algorithms [27], evolutionary algorithms [26] or Multi-Objective RL algorithms [25]. In CAT, Mujtaba and Mahapatra [26] proposed optimizing test length and accuracy by a multi-objective evolutionary algorithm. However, this method has not been verified on a real-world dataset.…”

Section: Multi-objective Optimizationmentioning

confidence: 99%

GMOCAT: A Graph-Enhanced Multi-Objective Method for Computerized Adaptive Testing

Wang

Long

Yin

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Computerized Adaptive Testing (CAT) refers to an online system that adaptively selects the best-suited question for students with various abilities based on their historical response records. Compared with traditional CAT methods based on heuristic rules, recent data-driven CAT methods obtain higher performance by learning from large-scale datasets. However, most CAT methods only focus on the quality objective of predicting the student ability accurately, but neglect concept diversity or question exposure control, which are important considerations in ensuring the performance and validity of CAT. Besides, the students' response records contain valuable relational information between questions and knowledge concepts. The previous methods ignore this relational information, resulting in the selection of sub-optimal test questions. To address these challenges, we propose a Graph-Enhanced Multi-Objective method for CAT (GMOCAT). Firstly, three objectives, namely quality, diversity and novelty, are introduced into the Scalarized Multi-Objective Reinforcement Learning framework of CAT, which respectively correspond to improving the prediction accuracy, increasing the concept diversity and reducing the question exposure. We use an Actor-Critic Recommender to select questions and optimize three objectives simultaneously by the scalarization function. Secondly, we utilize the graph neural network to learn relation-aware embeddings of questions and concepts. These embeddings are able

show abstract

Multi-objective optimization of item selection in computerized adaptive testing

Cited by 5 publications

References 21 publications

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

GDumb: A Simple Approach that Questions Our Progress in Continual Learning

Geography of Technology Transfer in China

GMOCAT: A Graph-Enhanced Multi-Objective Method for Computerized Adaptive Testing

Contact Info

Product

Resources

About