Detection of Item Preknowledge Using Response Times

Sinharay, Sandip

doi:10.1177/0146621620909893

Cited by 19 publications

(28 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It can be shown that for this hypothesis‐testing problem, the L T statistic is identical to the Wald test statistic given by

\frac{{\overset{τ}{true}}_{scriptC} - {\overset{τ}{true}}_{\overset{C}{true}}}{\sqrt{Var ({\overset{τ}{true}}_{scriptC}) + Var ({\overset{τ}{true}}_{\overset{C}{true}})}} = \frac{{\overset{τ}{true}}_{scriptC} - {\overset{τ}{true}}_{\overset{C}{true}}}{\sqrt{{false[\sum_{i \in scriptC} α_{i}^{2} false]}^{- 1} + {false[\sum_{i \in \overset{C}{true}} α_{i}^{2} false]}^{- 1}}} .

Sinharay (2019) also showed that under the LNMRT, L T follows the standard normal distribution under the null hypothesis of no item preknowledge, irrespective of the sizes of

scriptC

and

\overset{C}{true}

.…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…The LRT statistic for testing

H_{0} : {normalτ}_{scriptC} = {normalτ}_{\overset{C}{true}}

against

H_{1}^{'} : {normalτ}_{scriptC} \neq {normalτ}_{\overset{C}{true}}

is given by

{normalΛ}_{T} = 2 false[ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC}) + ℓ ({boldy}_{\overset{C}{true}} | {\overset{τ}{true}}_{\overset{C}{true}}) - ℓ (y | \overset{false^}{normalτ}) false],

where, for example,

ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC})

is the log‐likelihood of the log‐response times of the items in

scriptC

, computed at

{\overset{false^}{normalτ}}_{C}

. Sinharay (2019) showed that when one uses the LNMRT (van der Linden, 2006) for the response times under the hierarchical modelling framework of van der Linden (2007),

ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC})

can be expressed as

ℓ false(y_{C} false| {\overset{false^}{normalτ}}_{C} false) = \underset{i \in C}{false\sum} [- \frac{1}{2} \log false[2 normalπ false] + \log false(α_{i} false)] + {\overset{τ}{true}}_{scriptC}^{2} \underset{i \in}{false\sum}

…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…Consequently, to detect item preknowledge based on response times, one can test

H_{0} : {normalτ}_{scriptC} = {normalτ}_{\overset{C}{true}}

against

H_{1} : {normalτ}_{scriptC} > {normalτ}_{\overset{C}{true}}

using the signed LRT given by

L_{T} = \{\begin{matrix} \sqrt{Λ_{T}}, & if {\overset{τ}{true}}_{scriptC} \geq {\overset{τ}{true}}_{\overset{C}{true}}, \\ - \sqrt{Λ_{T}}, & if {\overset{τ}{true}}_{scriptC} < {\overset{τ}{true}}_{\overset{C}{true}} \end{matrix}

(Sinharay, 2019).…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…Ideas from constrained statistical inference (e.g., Silvapulle & Sen, 2001) are used in the construction of the new statistic and the derivation of its asymptotic distribution under the null hypothesis. Given the satisfactory performances of both L S and L T (e.g., Sinharay, 2017a, 2019), the new statistic is expected to have satisfactory Type I error rate and power.…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…As a consequence,

Λ_{ST}

given in equation 8 can be expressed as

\begin{matrix} {normalΛ}_{italicST} = 2 [ℓ false(x_{C} false| {\overset{false^}{normalθ}}_{C} false) + ℓ false(y_{C} false| {\overset{false^}{normalτ}}_{C} false) + ℓ false(x_{\overset{false¯}{scriptC}} false| {\overset{false^}{normalθ}}_{\overset{false¯}{scriptC}} false) + ℓ false(y_{\overset{false¯}{scriptC}} false| {\overset{false^}{normalτ}}_{\overset{false¯}{scriptC}} false) - ℓ false(boldx false| \overset{θ}{true} false) - ℓ false(boldy false| \overset{τ}{true} false)] \\ = {normalΛ}_{S} + {normalΛ}_{T}, \end{matrix}

because of equations 1 and 2. If both

scriptC

and

\overset{C}{true}

are large and the null hypothesis is true, then both L S and L T follow the standard normal distribution (Sinharay, 2017a, 2019) and are independent because of the local independence assumption underlying the model of van der Linden (2007), and hence

Λ_{ST}

is the sum of the squares of two independent standard normal variables and follows the χ 2 distribution with two degrees of freedom. However, the use of

Λ_{ST}

and, for example, the 95th percentile of the

χ_{2}^{2}

distribution as a cut‐off to perform a test at significance level of .05 for detecting preknowledge is inappropriate because the alternative hypothesis of interest consists of the union of two one‐sided hypotheses (rather than the union of two two‐sided hypotheses).…”

Section: Method: a New Statistic Based On Item Scores And Response Timesmentioning

confidence: 99%

See 4 more Smart Citations

The use of item scores and response times to detect examinees who may have benefited from item preknowledge

Sinharay

Johnson

2019

Brit J Math & Statis

Self Cite

View full text Add to dashboard Cite

According to Wollack and Schoenig (2018, The Sage encyclopedia of educational research, measurement, and evaluation. Thousand Oaks, CA: Sage, 260), benefiting from item preknowledge is one of the three broad types of test fraud that occur in educational assessments. We use tools from constrained statistical inference to suggest a new statistic that is based on item scores and response times and can be used to detect examinees who may have benefited from item preknowledge for the case when the set of compromised items is known. The asymptotic distribution of the new statistic under no preknowledge is proved to be a simple mixture of two χ2 distributions. We perform a detailed simulation study to show that the Type I error rate of the new statistic is very close to the nominal level and that the power of the new statistic is satisfactory in comparison to that of the existing statistics for detecting item preknowledge based on both item scores and response times. We also include a real data example to demonstrate the usefulness of the suggested statistic.

show abstract

“…It can be shown that for this hypothesis‐testing problem, the L T statistic is identical to the Wald test statistic given by

\frac{{\overset{τ}{true}}_{scriptC} - {\overset{τ}{true}}_{\overset{C}{true}}}{\sqrt{Var ({\overset{τ}{true}}_{scriptC}) + Var ({\overset{τ}{true}}_{\overset{C}{true}})}} = \frac{{\overset{τ}{true}}_{scriptC} - {\overset{τ}{true}}_{\overset{C}{true}}}{\sqrt{{false[\sum_{i \in scriptC} α_{i}^{2} false]}^{- 1} + {false[\sum_{i \in \overset{C}{true}} α_{i}^{2} false]}^{- 1}}} .

Sinharay (2019) also showed that under the LNMRT, L T follows the standard normal distribution under the null hypothesis of no item preknowledge, irrespective of the sizes of

scriptC

and

\overset{C}{true}

.…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…The LRT statistic for testing

H_{0} : {normalτ}_{scriptC} = {normalτ}_{\overset{C}{true}}

against

H_{1}^{'} : {normalτ}_{scriptC} \neq {normalτ}_{\overset{C}{true}}

is given by

{normalΛ}_{T} = 2 false[ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC}) + ℓ ({boldy}_{\overset{C}{true}} | {\overset{τ}{true}}_{\overset{C}{true}}) - ℓ (y | \overset{false^}{normalτ}) false],

where, for example,

ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC})

is the log‐likelihood of the log‐response times of the items in

scriptC

, computed at

{\overset{false^}{normalτ}}_{C}

. Sinharay (2019) showed that when one uses the LNMRT (van der Linden, 2006) for the response times under the hierarchical modelling framework of van der Linden (2007),

ℓ ({boldy}_{scriptC} | {\overset{τ}{true}}_{scriptC})

can be expressed as

ℓ false(y_{C} false| {\overset{false^}{normalτ}}_{C} false) = \underset{i \in C}{false\sum} [- \frac{1}{2} \log false[2 normalπ false] + \log false(α_{i} false)] + {\overset{τ}{true}}_{scriptC}^{2} \underset{i \in}{false\sum}

…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…Consequently, to detect item preknowledge based on response times, one can test

H_{0} : {normalτ}_{scriptC} = {normalτ}_{\overset{C}{true}}

against

H_{1} : {normalτ}_{scriptC} > {normalτ}_{\overset{C}{true}}

using the signed LRT given by

L_{T} = \{\begin{matrix} \sqrt{Λ_{T}}, & if {\overset{τ}{true}}_{scriptC} \geq {\overset{τ}{true}}_{\overset{C}{true}}, \\ - \sqrt{Λ_{T}}, & if {\overset{τ}{true}}_{scriptC} < {\overset{τ}{true}}_{\overset{C}{true}} \end{matrix}

(Sinharay, 2019).…”

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

Section: Literature Review: a Model And Some Existing Methodsmentioning

confidence: 99%

“…As a consequence,

Λ_{ST}

given in equation 8 can be expressed as

\begin{matrix} {normalΛ}_{italicST} = 2 [ℓ false(x_{C} false| {\overset{false^}{normalθ}}_{C} false) + ℓ false(y_{C} false| {\overset{false^}{normalτ}}_{C} false) + ℓ false(x_{\overset{false¯}{scriptC}} false| {\overset{false^}{normalθ}}_{\overset{false¯}{scriptC}} false) + ℓ false(y_{\overset{false¯}{scriptC}} false| {\overset{false^}{normalτ}}_{\overset{false¯}{scriptC}} false) - ℓ false(boldx false| \overset{θ}{true} false) - ℓ false(boldy false| \overset{τ}{true} false)] \\ = {normalΛ}_{S} + {normalΛ}_{T}, \end{matrix}

because of equations 1 and 2. If both

scriptC

and

\overset{C}{true}

Λ_{ST}

is the sum of the squares of two independent standard normal variables and follows the χ 2 distribution with two degrees of freedom. However, the use of