Using Personality Item Characteristics to Predict Single–Item Internal Reliability, Retest Reliability, and Self–Other Agreement

Abstract

The use of reliability estimates is increasingly scrutinized as scholars become more aware that test–retest stability and self–other agreement provide a better approximation of the theoretical and practical usefulness of an instrument than its internal reliability. In this study, we investigate item characteristics that potentially impact single‐item internal reliability, retest reliability, and self–other agreement. Across two large samples (N = 6690 and N = 4396), two countries (Estonia and The Netherlands), and two personality inventories (the NEO PI‐3 and the HEXACO‐PI‐R), results show that (i) item variance is a strong predictor of self–other agreement and retest reliability but not of single‐item internal reliability; (ii) item variance mediates the relations between evaluativeness and self–other agreement; and (iii) self–other agreement is predicted by observability and item domain. On the whole, weak relations between item length, negations, and item position (indicating effects of questionnaire length) on the one hand, and single‐item internal reliability, retest reliability, and self–other agreement on the other, were observed. In order to increase the predictive validity of personality scales, our findings suggest that during the construction of questionnaire items, researchers are advised to pay close attention especially to item variance, but also to evaluativeness and observability. Copyright © 2016 European Association of Personality Psychology

Effects of personality and gender on self–other agreement in ratings of leadership

“…For example, de Vries et al . () assessed Neuroticism (measured as Emotional Stability) as moderately observable, but stated that visible expressions of high Emotionality (i.e., Neuroticism) may be uncommon. This statement is supported by Digman's () contention that Emotional Stability is a highly socially desirable trait, which can be adapted to conclude that Neuroticism is highly un desirable.…”

“…Regarding external observability, de Vries et al . () assessed Extraversion to be highest among the Big 5 traits in observability. Despite this finding, we posit that the greater effect will be upon self‐ratings due to self‐enhancement.…”

“…‘Fundamentally Openness is a matter of inner experience, a mental phenomenon related to the scope of awareness or the depth and intensity of consciousness’ (McCrae & Costa, , p. 835). Although there have been differences of opinion regarding the degree of observability of Openness, it was rated least observable among the Big 5 NEO PI‐3 traits (de Vries et al ., ). Connelly and Ones () similarly characterize Openness as a trait low in visibility.…”

Estimating the Reliability of Emotion Measures over Very Short Intervals: The Utility of Within- Session Retest Correlations

“…However, the correlates of ( ) at the item-level suggest that within-session retest correlations offer many advantages as reliability estimates (de Vries et al, 2016). Specifically, they provided a rank-ordering of ( ) estimates that was consistent across different samples and more closely tracked item properties associated with reliability and validity related criteria (McCrae et al, 2011;Wood & Wortman, 2012) than internal consistency statistics.…”

The Almost Unbearable Lightness of Personality

“…Social and personality psychology are notorious for using very small samples (Fraley & Vazire, ; Watson, ). In order to overcome limitations imposed by small samples, we recently studied self–other agreement in two large samples (total N = 11,096) of participants who speak two different languages, Dutch and Estonian, and who completed two different personality questionnaires, the NEO PI‐3 and HEXACO‐PI‐R (Allik et al, ; De Vries, Realo, & Allik, ). Our purpose was to investigate how often two people who know each other well can reach an agreement in their judgments of personality that is above random agreement between any arbitrarily paired judgments.…”