Measurement bias in self-reports of offending: a systematic review of experiments

Abstract

Objectives Self-reported offending is one of the primary measurement methods in criminology. In this article, we aimed to systematically review the experimental evidence regarding measurement bias in self-reports of offending. Methods We carried out a systematic search for studies that (a) included a measure of offending, (b) compared self-reported data on offending between different methods, and (c) used an experimental design. Effect sizes were used to summarize the results. Results The 21 pooled experiments provided evidence regarding 18 different types of measurement manipulations which were grouped into three categories, i.e., Modes of administration, Procedures of data collection, and Questionnaire design. An analysis of the effect sizes for each experimental manipulation revealed, on the one hand, that selfreports are reliable across several ways of collecting data and, on the other hand, selfreports are influenced by a wide array of biasing factors. Within these measurement biases, we found that participants' reports of offending are influenced by modes of administration, characteristics of the interviewer, anonymity, setting, bogus pipeline, response format, and size of the questionnaire. Conclusions This review provides evidence that allows us to better understand and improve crime measurements. However, many of the experiments presented in this review are not replicated and additional research is needed to test further aspects of how asking questions may impact participants' answers.

The impact of survey mode on US national estimates of adolescent drug prevalence: results from a randomized controlled study

“…More recently, Gomes et al . [8] focused more narrowly on reports on offending. Of the 10 studies comparing paper SAQs with CASI, five comparisons favored CASI; only one reached statistical significance at the α = 0.05 level.…”

The Base Rate Study: Developing Base Rates for Risk Factors and Indicators for Engagement in Violent Extremism

“…Second, whilst we suggest that in the present case, direct questioning elicited the best results, we do not suggest that these are not also subject to self‐reporting biases. Gomes et al recently conducted a systematic review of measurement biases in self‐reports of offending behavior and demonstrated the range of potential biases that may occur under varying conditions. No design will be absent of these, however, some may be more suited to certain study designs than others, as in the present study.…”

“…One factor in explaining the degree of misreporting in direct questioning is the mode of survey delivery (see Ref. for a systematic review). Interviewer‐administered surveys, such as pencil‐and‐paper studies, or face‐to‐face interviews, can result in increased misreporting compared to self‐administered surveys.…”

Psychosocial Prevention Programs against Radicalization and Extremism: A Meta-Analysis of Outcome Evaluations

“…In this review, we also noticed that behavioral outcomes were rare, mainly measured through self-report not objective data. While self-reported information on criminal activity is influenced by measurement bias, objective data are limited to official records, disregarding unreported criminal actions (Baier et al, 2016;Gomes et al, 2019). Coupled with the fact that only few people engage in violent extremism at all, assessing such behavior is a difficult task (Clubb & Tapely, 2018;Mastroe & Szmania, 2016).…”