A Short Note on Optimizing Cost-Generalizability via a Machine-Learning Approach

Abstract

The costs of an objective structured clinical examination (OSCE) are of concern to health profession educators globally. As OSCEs are usually designed under generalizability theory (G-theory) framework, this article proposes a machine-learning-based approach to optimize the costs, while maintaining the minimum required generalizability coefficient, a reliability-like index in G-theory. The authors adopted G-theory parameters yielded from an OSCE hosted by a medical school, reproduced the generalizability coefficients to prepare for optimizing manipulations, applied simulated annealing algorithm to calculate the number of facet levels minimizing the associated costs, and conducted the analysis in various conditions via computer simulation. With a given generalizability coefficient, the proposed approach, virtually an instrument of decision-making supports, found the optimal solution for the OSCE such that the associated costs were minimized. The computer simulation results showed how the cost reductions varied with different levels of required generalizability coefficients. Machine learning–based approaches can be used in conjunction with psychometric modeling to help planning assessment tasks more scientifically. The proposed approach is easy to adopt into practice and customize in alignment with specific testing designs. While these results are encouraging, the possible pitfalls such as algorithmic convergences’ failure and inadequate cost assumptions should also be avoided.

Exploration of validity evidence for core residency entrustable professional activities in Chinese pediatric residency

“…4. Competence/performance-related investigations through G-theory in the field of medical education have been seen more in the literature (24,25) also conveying that our methodological adoption is a strong fit for the present study, which involves different EPAs, raters, and randomly crossed structure between raters and residents.…”

“…Compared with CTT that simply assumes that observed performance consists of true ability effect and error effect (i.e., the well-known X = T + E and each effect correspond to variance such that ), G-theory is compatible with designs with multiple facets such as raters, items, groups, and occasions ( 24 , 25 ), each of which is an effect affecting the observed scores. For instance, in performance assessment where a is the p × i × r design present (each person p is graded by every rater r on each task/item i ), G-theory can decompose observed response data as , where an observed score, , for person p on item i rated by rater r is made of the grand mean μ, person effect , item effect , rater effect , interaction terms of any two random effects, and error effect .…”

“…, G-theory is compatible with designs with multiple facets such as raters, items, groups, and occasions (24,25), each of which is an effect affecting the observed scores. For instance, in performance assessment where a is the p × i × r design present (each person p is graded by every rater r on each task/item i), G-theory can decompose observed response data as…”