Hidden invalidity among fifteen commonly used measures in social and personality psychology

Abstract

Flake, Pek, and Hehman (2017) recently demonstrated that metrics of structural validity are severely underreported in social and personality psychology. We apply their recommendations for the comprehensive assessment of structural validity to a uniquely large and varied dataset (N = 144496 experimental sessions) to investigate the psychometric properties of some of the most widely used self-report measures (k = 15 questionnaires, 26 subscales) in social and personality psychology. When assessed using the modal practice of considering only their internal consistency, 89% of scales appeared to possess good validity. Yet, when validity was assessed comprehensively (via internal consistency, immediate and delayed test-retest reliability, factor structure, and measurement invariance for median age and gender) only 4% demonstrated good validity. Furthermore, the less commonly a test is reported in the literature, the more likely it was to be failed (e.g., measurement invariance). This suggests that the pattern of under- reporting in the field may represent widespread hidden invalidity of the measures we use, and therefore pose a threat to many research findings. We highlight the degrees of freedom afforded to researchers in the assessment and reporting of structural validity. Similar to the better-known concept of p-hacking, we introduce the concept of validity hacking (v-hacking) and argue that it should be acknowledged and addressed.

Moral Foundations' Methodological Foundations: A Systematic Analysis of Reliability in Research Using the Moral Foundations Questionnaire

“…With the exception of the sanctity-degradation subscale, the MFQ's scales generally do not meet that threshold. In light of Hussey and Hughes's (2018) work and our own findings presented here, one might hypothesize that the MFQ similarly lacks measurement invariance, which is supported by Iurino and Saucier's (2020) inability to demonstrate adequate measurement invariance in a cross-cultural context. Given that measurement validity is a precursor to making valid claims based on observations, our own observations, in line with, for example, Iurino and Saucier (2020) and Davis et al (2016), might serve as a call to revisit the evidentiary basis for MFT more broadly.…”

“…We would be remiss not to contextualize our examination of reliability within a growing body of work on the validity of measures in the field more generally and the MFQ specifically. As Hussey and Hughes (2018) found in their analysis of several popular scales, 89% of the measures met or exceeded α = .70, but only 4% also showed acceptable indicators of validity in more rigorous tests of measurement validity. With the exception of the sanctity-degradation subscale, the MFQ's scales generally do not meet that threshold.…”

Setting the record straight: System justification and rigidity‐of‐the‐right in contemporary Hungarian politics

“…Another difference between our research program and earlier work by Kelemen et al () and Lönnqvist et al () is that we administered complete system justification scales and a complete subscale used to measure personal needs for order and structure, whereas they presented results based on composites of a relatively few selected items after dropping some items based on the results of factor analyses. In light of growing methodological concerns associated with the use of ad hoc measures and individual items (Flake & Fried, ; Flake, Pek, & Hehman, ; Hussey & Hughes, ; Pietryka & Macintosh, ), we believe that it is preferable to administer and analyse data based on complete scales. In this sense, conclusions based on our analyses may be more solid than those reached in previous research on political psychology in Hungary.…”

Neoliberal Ideology and the Justification of Inequality in Capitalist Societies: Why Social and Economic Dimensions of Ideology Are Intertwined

“…Thus, Malka and colleagues’ conclusions were based purely on patterns of correlations involving several unreliable scales that were not designed for the purposes to which they put them (cf. Flake & Fried, ; Flake, Pek, & Hehman, ; Hussey & Hughes, ).…”