Internal Consistency, Retest Reliability, and Their Implications for Personality Scale Validity

Abstract

We examined data (N = 34,108) on the differential reliability and validity of facet scales from the NEO Inventories. We evaluated the extent to which (a) psychometric properties of facet scales are generalizable across ages, cultures, and methods of measurement; and (b) validity criteria are associated with different forms of reliability. Composite estimates of facet scale stability, heritability, and cross-observer validity were broadly generalizable. Two estimates of retest reliability were independent predictors of the three validity criteria; none of three estimates of internal consistency was. Available evidence suggests the same pattern of results for other personality inventories. Internal consistency of scales can be useful as a check on data quality, but appears to be of limited utility for evaluating the potential validity of developed scales, and it should not be used as a substitute for retest reliability. Further research on the nature and determinants of retest reliability is needed. KeywordsReliability; validity; cross-national; Five-Factor Model; personality traits Scale reliability is commonly said to limit validity (John & Soto, 2007); in principle, more reliable scales should yield more valid assessments (although of course reliability is not sufficient to guarantee validity). For a given set of scales, such as the 30 facets of the NEO Inventories (McCrae & Costa, in press), there is differential reliability: Some facets are more reliable than others. That fact makes it possible to test the maxim that reliability limits validity, provided that criteria of validity are chosen that are comparable across all facet scales: More reliable facets ought to be more valid. We will argue that three relevant criteria are longitudinal stability, heritability, and cross-observer agreement. Each of the 30 NEO facets is known to be more or less stable (Costa, Herbst, McCrae, & Siegler, 2000) and heritable (Jang, McCrae, Angleitner, Riemann, & Livesley, 1998), and to show evidence of some degree of cross-observer agreement (McCrae et al., 2004); however, other things being equal, more reliable facets should be more stable and heritable, and show stronger evidence NIH-PA Author ManuscriptNIH-PA Author Manuscript NIH-PA Author Manuscript of consensual validity. There are, however, different forms of reliability, of which internal consistency and retest reliability are the most prominent. In the present article we (a) assemble evidence on the stability, heritability, and cross-observer validity of NEO facets from the published literature; and (b) predict these values from estimates of internal consistency and retest reliability. These analyses allow us to assess the relative importance of these two forms of reliability.In this article we construe validity broadly to refer to the quality of a scale as a measure of its intended construct. However, our discussion is limited to a consideration of convergent validity; readers should recall that discriminant validity is also an essential attribute of a good sc...

How functionalist and process approaches to behavior can explain trait covariation.

“…However, there is actually surprisingly little evidence that structural factors are either more heritable or more stable than the narrower traits they summarize. Once a trait measure's instability over very short periods of time (e.g., 1 or 2 weeks) is accounted for, narrow traits appear to show heritability and stability estimates that may be comparable with broader structural factors (McCrae, Kurtz, Yamagata, & Terracciano, 2011, Table 2; Wood & Wortman, 2012). Nor is it clear that higher levels of these characteristics necessarily indicate that they are more "basic."…”

Academic Identity Status Measure: Psychometric Properties when Used among Secondary School Students in Kenya

“…It is argued that coefficient alpha is sensitive to the conditions of testing and the appropriateness of the language to the population tested [13]. This assertion persuaded us to contend that the variation in the AIM's alpha levels for the studies in different cultural contexts may point to its possible differential reliability occasioned by translation; or cultural differences, or even the age differences among of the samples.…”

“…The internal consistency is popularly reported using coefficient alphas (α) and it is widely believed to judge the precision and quality of the data obtained through scales in Psychology [13]. Robust alpha coefficients for AIM have been reported in studies among high school students [8] and college students [2,9,14] in Iran and America (Table 1).…”

“…This mainly involves correlating an instruments scores with those of theoretically relevant outcomes [13]. Was and Isaacson [2] correlated AIM scores with final grades in an introduction to psychology course.…”

Tourism Product Clubs: Theory, Added Value and Practical Implications

“…As usual with new measurement methods, Cronbach's alpha was used to assess the scales' internal consistency (McCrae et al 2011). According to DeVellis (1991) all constructs demonstrate high internal consistency.…”