Dispersion matters: Diagnostics and control data computer simulation in Concealed Information Test studies

Abstract

Binary classification has numerous applications. For one, lie detection methods typically aim to classify each tested person either as "liar" or as "truthteller" based on the given test results. To infer practical implications, as well as to compare different methods, it is essential to assess the diagnostic efficiency, such as demonstrating the number of correctly classified persons. However, this is not always straightforward. In Concealed Information Tests (CITs), the key predictor value (probe-irrelevant difference) for "truthtellers" is always similar (zero on average), and "liars" are always distinguished by a larger value (i.e., a larger number resulting from the CIT test, as compared to the zero baseline). Thereby, in general, the larger predictor values a given CIT method obtains for "liars" on average, the better this method is assumed to be. This has indeed been assumed in countless studies, and therefore, when comparing the classification efficiencies of two different designs, the mean difference of "liar" predictor values in the two designs were simply compared to each other (hence not collecting "truthteller" data to spare resources). We show, based on the metadata of 12 different experimental designs collected in response time-based CIT studies, that differences in dispersion (i.e., variance in the data, e.g. the extent of random deviations from the zero average in case of "truthtellers") can substantially influence classification efficiency-to the point that, in extreme cases, one design may even be superior in classification despite having a larger mean "liar" predictor value. However, we also introduce a computer simulation procedure to estimate classification efficiency in the absence of "truthteller" data, and validate this procedure via a meta-analysis comparing outcomes based on empirical data versus simulated data.

The mechanism of filler items in the response time concealed information test

“…To calculate illustrative areas under curves (AUCs) for probe-irrelevant RT mean differences as predictors, we simulated control groups for the RT data from each of the four possible conditions, using 1,000 normally distributed values with a mean of zero and an SD derived from the real data (from each condition) as SD real × 0.5 + 7 ms (which has been shown to very closely approximate actual AUCs; Lukács & Specker, 2020; the related function is available in the analysis codes uploaded to the OSF repository). We would like to emphasize that these simulated AUCs are just approximations for illustration, and we do not use them for any of our statistical tests.…”

Prolonged Response Time Concealed Information Test decreases probe-control differences but increases classification accuracy

“…The following analysis was performed on a recently published extensive database (Lukács & Specker, 2020) with trial-level results from RT-CIT studies, including 12 different experimental designs with both guilty and innocent participants, from seven different papers (Geven et al, 2018;, 2016Lukács et al, 2017;Noordraven & Verschuere, 2013;; for detailed database description, see Lukács & Specker, 2020, pp. 5-6).…”

“…Likewise, simulated innocent predictor variables can be generated as normally distributed data with the predicted SD of innocent probe-control difference, and a mean of zero (assuming no probe recognition, hence no probe-control RT difference on average). From the simulated guilty and innocent predictor variables (100 values per each) at each given trial, an AUC can be calculated (regarding the reliability of such simulated AUCs, see Lukács & Specker, 2020). Thereby the AUC may be indirectly extrapolated based on the underlying data; see Figure A1, bottom panel (B).…”

Machine learning mega-analysis applied to the Response Time Concealed Information Test: No evidence for advantage of model-based predictors over baseline

“…For all our analysis we used, as a convenience sample, a recently published extensive database with trial-level results from CIT studies, available via https://osf.io/sfbkt/ (Lukács & Specker, 2020). This data includes 12 different datasets with different experimental designs (each including data from guilty as well as innocent participants), from seven different papers (Geven et al, 2018;Lukács, Kleinberg, et al, 2017;Noordraven & Verschuere, 2013;Verschuere et al, 2015; for detailed database description, see Lukács & Specker, 2020, pp.…”