Using Multitrait-Multimethod (MTMM) Techniques to Examine the Convergent and Discriminant Validity of Social Disorder

Abstract

Objectives: Disorder has been measured by various data sources; however, little attention has been given to comparing the construct validity of different measures obtained through various methods in capturing social disorder and related phenomena. Methods: The multitrait-multimethod approach was used to triangulate the consistency between social disorder, prostitution and drug activity across resident surveys, systematic social observations, and police calls for service data. Results: Prostitution and drug activity showed convergent validity, while there was little evidence that social disorder was consistently measured across the three methods. None of the three social problem measures showed high discriminant validity. Drug activity seems to have highest trait-specific discriminant validity across measures, and prostitution is the most identifiable measure across data sources. Social disorder was found to have low discriminant validity. However, the agreement between databases varies across the type of social problems. Conclusions: Social disorder appears to the most difficult concept to define and measure consistently. The lack of correspondence across data sources cautions against the use of a single source of information in studying disorder. Future studies should explore the factors that shape perceptions of disorder and how to best measure disorder to test the broken windows thesis and related concepts.

Dissecting Disorder Perceptions: Neighborhood Structure and the Moderating Role of Interethnic Contact and Xenophobic Attitudes

“…However, some studies found weaker correlations, varying between r ¼ .32 (Conley et al, 2014) and r ¼ .10 (Drakulich, 2013). Yang et al (2018) did not find a significant association between observed and perceived social disorder. In no study, the proportion of covariance between perceived and observed disorder exceeded 50%, leaving ample room for other influences that have increasingly found scholarly interest and led to a change of perspective toward the subjective dimensions of urban disorder.…”

Do Prison Administrative and Survey Data Sources Tell the Same Story? A Multitrait, Multimethod Examination With Application to Gangs

“…The final way we expand our understanding of the validity of these measures is by constructing an MTMM matrix (Campbell & Fiske, 1959). Although there are some exceptions (e.g., Decker & Pyrooz, 2010; Rosenfeld & Decker, 1999; Van Voorhis, 1994; Yang et al, 2018), such tests are rare in criminology generally and in institutional corrections specifically. A core requirement for the matrix is that we obtain multiple measures (i.e., traits) from multiple data sources (i.e., methods).…”

“…Finally, we construct a multitrait, multimethod (MTMM) matrix as a final indicator of methodological triangulation. MTMM has a long history in psychology, but is rare in criminology, offering an established test of validity (see Yang, Hinkle, & Wyckoff, 2018). Examining the overlap between primary data sources on prisoners is critically important to the advancement of theory and research in prisons, particularly the study of prison gangs, and has broad implications for measurement in criminology.…”

The impact of measurement error in models using police recorded crime rates

“…We can deduce that the under-recording rates reported in the literature -and throughout this article -are exaggerated, but how much so is currently unclear. Measurement error estimation methods that do not rely on a gold standard, such as multitraitmultimethod latent variable models (Oberski et al, 2017;Yang et al, 2018) or hidden Markov models (Pavlopuolos et al, 2020), offer a particularly promising avenue of research. These can be used to estimate the validity and reliability of variables measuring the same concept, and, to the best of our knowledge, they…”