On the validity of non-randomized response techniques: an experimental comparison of the crosswise model and the triangular model

Abstract

Non-randomized response techniques (NRRTs) such as the crosswise model and the triangular model (CWM and TRM; Yu et al. Metrika, 67, 251-263, 2008) have been developed to control for socially desirable responding in surveys on sensitive personal attributes. We present the first study to directly compare the validity of the CWM and TRM and contrast their performance with a conventional direct questioning (DQ) approach. In a paper-pencil survey of 1382 students, we obtained prevalence estimates for two sensitive attributes (xenophobia and rejection of further refugee admissions) and one nonsensitive control attribute with a known prevalence (the first letter of respondents' surnames). Both NRRTs yielded descriptively higher prevalence estimates for the sensitive attributes than DQ; however, only the CWM estimates were significantly higher. We attribute the higher prevalence estimates for the CWM to its response symmetry, which is lacking in the TRM. Only the CWM provides symmetric answer options, meaning that there is no "safe" alternative respondents can choose to distance themselves from being carriers of the sensitive attribute. Prevalence estimates for the nonsensitive control attribute with known prevalence confirmed that neither method suffered from method-specific bias towards over-or underestimation. Exploratory moderator analyses further suggested that the sensitive attributes were perceived as more sensitive among politically left-oriented than among politically right-oriented respondents. Based on our results, we recommend using the CWM over the TRM in future studies on sensitive personal attributes.

Can detailed instructions and comprehension checks increase the validity of crosswise model estimates?

“…Fig 1 shows the CWM as a tree diagram. Significantly higher and thus presumably more valid prevalence estimates have been obtained via the CWM as compared to DQ for sensitive attributes such as xenophobia [10,11], plagiarism [12], tax evasion [13,14], distrust in the Trust Game [14], crossing the street on a red light in plain view of children [15], the use of anabolic steroids among bodybuilders [16], intention to vote for the far-right German party Alternative for Germany [17], and prejudice against female leaders [18]. Moreover, in one strong validation study, the CWM accurately estimated the prevalence of experimentally induced cheating behavior, while DQ led to a severe underestimation [19].…”

Functionality of the Crosswise Model for Assessing Sensitive or Transgressive Behavior: A Systematic Review and Meta-Analysis

“…We conducted a systematic review and meta-analysis as well as quality assessment of empirical applications of CM Coutts et al, 2011;Jann et al, 2012;Korndörfer et al, 2014;Hoffmann et al, 2015Hoffmann et al, , 2017Höglinger et al, 2016;Kundt et al, 2017;Heck et al, 2018;Nasirian et al, 2018;Hopp and Speil, 2019;Lehrer et al, 2019;Safiri et al, 2019;Schnapp, 2019;Walzenbach and Hinz, 2019;Meisters et al, 2020b;Özgül, 2020Özgül, k = 7 Kundt, 2014Höglinger and Diekmann, 2017;Höglinger and Jann, 2018;Atsusaka and Stevenson, 2020;Hoffmann et al, 2020;Meisters et al, 2020a;Jerke et al, 2021 CM prevalence (k =18) Nakhaee et al, 2013;Kazemzadeh et al, 2016k = 9 Eslami et al, 2013Waubert de Puiseau et al, 2017;Banayejeddi et al, 2019;Klimas et al, 2019;Vakilian et al, 2019;Jensen, 2020;Lacker et al, 2020;Mieth et al, 2021k = 7 Shamsipour et al, 2014Vakilian et al, 2014Vakilian et al, , 2016Gingerich et al, 2015;…”

“…Online questionnaires (k = 21) Korndörfer et al, 2014;Kundt, 2014;Hoffmann et al, 2015Hoffmann et al, , 2017Höglinger et al, 2016;Höglinger and Diekmann, 2017;Waubert de Puiseau et al, 2017;Höglinger and Jann, 2018;Hopp and Speil, 2019;Klimas et al, 2019;Lehrer et al, 2019;Schnapp, 2019;Walzenbach and Hinz, 2019;Atsusaka and Stevenson, 2020;Jensen, 2020;Lacker et al, 2020;Meisters et al, 2020a;Canan et al, 2021;Jerke et al, 2021;Mieth et al, 2021 Paper questionnaires (k = 18) Coutts et al, 2011;Jann et al, 2012;Nakhaee et al, 2013;Shamsipour et al, 2014;Vakilian et al, 2014Vakilian et al, , 2016Vakilian et al, , 2019Khosravi et al, 2015;Kazemzadeh et al, 2016;Heck et al, 2018;Mirzazadeh et al, 2018;Nasirian et al, 2018;Banayejeddi et al, 2019;Safiri et al, 2019;Hoffmann et al, 2020;Meisters et al, 2020b;Özgül, 2020 Interviews (k = 4) Gingerich et al, 2015;Kundt et al, 2017;Oliveros and Gingerich, 2020 Interviews and questionnaires (k = 2) Eslami et al, 2013;Jerke e...…”

Controlling social desirability bias: An experimental investigation of the extended crosswise model

“…Alternatively, prejudiced individuals PLOS ONE may become increasingly aware of the growing social undesirability of their views and may therefore be more reluctant to express more blatant forms of prejudice openly [44]. Following this reasoning, estimates for the prevalence of Islamophobia based on direct self-reports are likely underestimates of the true value, and indirect questioning techniques may help to obtain more valid estimates [15,16,31,45]. If the ECWM is suitable as a new means of controlling for social desirability, it should provide higher and thus potentially more valid estimates for the prevalence of Islamophobia than a conventional direct question.…”

“…Moreover, the CWM is easier to provide instructions for, since in contrast to many RRTs it does not require an external randomization device [10]. The CWM has led to significantly higher and thus-according to the "more is better" criterion-presumably more valid prevalence estimates than direct questions in a number of studies investigating sensitive attributes such as xenophobia [15,16], plagiarism [17], tax evasion [18,19], the use of anabolic steroids by bodybuilders [20], the intention to vote for the far-right German party Alternative for Germany [21], distrust in the Trust Game [22] and prejudice against female leaders [23]. Moreover, the CWM was able to accurately estimate the known prevalence of experimentally induced cheating behavior [24].…”