A Comparison of Experimental and Observational Approaches to Assessing the Effects of Time Constraints in a Medical Licensing Examination

Abstract

Test administrators are appropriately concerned about the potential for time constraints to impact the validity of score interpretations; psychometric efforts to evaluate the impact of speededness date back more than half a century. The widespread move to computerized test delivery has led to the development of new approaches to evaluating how examinees use testing time and to new metrics designed to provide evidence about the extent to which time limits impact performance. Much of the existing research is based on these types of observational metrics; relatively few studies use randomized experiments to evaluate the impact time limits on scores. Of those studies that do report on randomized experiments, none directly compare the experimental results to evidence from observational metrics to evaluate the extent to which these metrics are able to sensitively identify conditions in which time constraints actually impact scores. The present study provides such evidence based on data from a medical licensing examination. The results indicate that these observational metrics are useful but provide an imprecise evaluation of the impact of time constraints on test performance. 308

A Multidimensional Partially Compensatory Response Time Model on Basis of the Log-Normal Distribution

“…Possible interpretations of the components depend, of course, on the context. Empirical findings reported by Harik et al (2018) suggest that the response time on a test item is not entirely determined by the time spent on information processing. This implies that the response time can be decomposed into a component that is related to the information processing capabilities of an individual and a component that is not.…”

On the Speed Sensitivity Parameter in the Lognormal Model for Response Times and Implications for High-Stakes Measurement Practice

“…Assuring that even the slowest test takers can work without time pressure would imply a time limit that is far too generous for fast test takers and problematic both from an economical and from a motivational perspective. Furthermore, the results of this study have important implications for determining the speededness of a test: So far, often experimental methods using different time limits or different numbers of items in the same time limit have been used (e.g., Bridgman, Cline, & Hessinger, 2004; Bridgman, Trapani, & Curley, 2004; Harik et al, 2018). But while for the majority of the test takers more generous time limits might only have a small impact on the demonstrated ability, different time limits can still substantially affect the slowest part of the population.…”

Using Natural Language Processing to Predict Item Response Times and Improve Test Construction

“…The sensitivity of scores from Step 2 CK to varying time limits has been studied by Harik et al. (2018). The evidence reported in that paper suggests that examinee performance can be affected by the amount of available time, so increasing the uniformity of the time demand across forms of the examination is a matter of practical importance.…”