On Joining a Signal Detection Choice Model with Response Time Models

Abstract

In a signal detection theory (SDT) approach to multiple choice exams, examinees are viewed as choosing, for each item, the alternative that is perceived as being the most plausible, with perceived plausibility depending in part on whether or not an item is known. The SDT model is a process model and provides measures of item difficulty, item discrimination, and the relative plausibility of each alternative. It is shown how to incorporate information from response times into the model, which has potential benefits for estimation and also offers a way to study underlying processes. The SDT model is joined with a lognormal response time (RT) model in a manner similar to that used in hierarchical models. In addition, a mixture extension of the RT model is joined in a novel way with the SDT choice model, using the idea that the probability of "knowing" in the SDT model might be related to the probability of working in one of two speed states in the mixture RT model. A semiparametric version of the mixture RT model is also used to assess robustness. The fused SDT/RT models are examined with science items from the 2015 Program for International Student Assessment.

Classical Item Analysis from a Signal Detection Perspective

“…These limitations suggest that there might be room for improvement over simple total-score splits. For example, recent research has shown that response times can be used in the SDT choice model in a simple and meaningful way, with the basic idea being that response times provide indirect information about an examinee's latent state of knowledge (DeCarlo, 2021b). This raises the possibility that it might be possible to use response times to help improve the split for a simple item analysis, thus giving improved measures.…”

“…Another option is to use a logistic-normal distribution for λ i (Aitchison & Shen, 1980;DeCarlo, 2021aDeCarlo, , 2021b, which is similar to (but also distinct from) the Beta distribution in that it allows for a variety of shapes. For the logistic normal,…”

“…The distribution arises by taking the logistic transformation of a normal random variable, thus the name logistic-normal (equivalently, a logit transformation of the above probability is normally distributed, and so the term logit-normal has also been used, e.g., Mead, 1965). An advantage of using the logistic-normal is that the model remains within the framework of generalized linear models (McCullagh & Nelder, 1989) and so it can be fit using maximum likelihood or posterior mode estimation, which at this time offer considerable speed advantages over Bayesian estimation (see DeCarlo, 2021aDeCarlo, , 2021b. Another advantage is that it is straightforward with the logistic-normal to allow for correlations in a multidimensional higher-order structure (DeCarlo, 2021b).…”

“…Another option is to use a logistic‐normal distribution for λ i (Aitchison & Shen, 1980; DeCarlo, 2021a, 2021b), which is similar to (but also distinct from) the Beta distribution in that it allows for a variety of shapes. For the logistic normal, $$$$\begin{equation*}p\;\left( {\;{\delta _{ij}} = \;1|\mu ,\sigma ,\theta } \right) = \frac{{{e^{\mu + \sigma \theta }}}}{{1 + {e^{\mu + \sigma \theta }}}}\;,\end{equation*}$$$$where θ ∼ N (0, 1).…”

“…An advantage of using the logistic‐normal is that the model remains within the framework of generalized linear models (McCullagh & Nelder, 1989) and so it can be fit using maximum likelihood or posterior mode estimation, which at this time offer considerable speed advantages over Bayesian estimation (see DeCarlo, 2021a, 2021b). Another advantage is that it is straightforward with the logistic‐normal to allow for correlations in a multidimensional higher‐order structure (DeCarlo, 2021b).…”