Comparing Rater Groups: How To Disentangle Rating Reliability From Construct-Level Disagreements

Abstract

In this commentary, we build on Bracken, Rose, and Church's (2016) definition stating that 360°feedback should involve "the analysis of meaningful comparisons of rater perceptions across multiple ratees, between specific groups of raters" (p. 764). Bracken et al. expand on this component of the definition later by stressing that "the ability to conduct meaningful comparisons of rater perceptions both between (inter) and within (intra) groups is central and, indeed, unique to any true 360°feedback process" (p. 767; italicized in their focal article). Bracken et al. stress that "This element of our definition acknowledges that 360°feedback data represent rater perceptions that may contradict each other while each being true and valid observations" (p. 767).Bracken et al. (p. 768) present six questions, three of which stress intergroup comparisons: Question 2, which reads, "Is the feedback process conducted in a way that formally segments raters into clearly defined and meaningful groups?"; Question 4, which reads, "Is the feedback collected . . . to establish reliability, which can vary by rater group?" [emphasis added]); and Question 5, which reads, does "the feedback process . . . provide the user with sufficiently clear and reliable [emphasis added] insights into interand intragroup perceptions?" The original definition, as well as the three questions, clearly emphasizes the need for delineating distinct groups of raters. Finally, in discussing how we can facilitate evolution of 360°feedback, Bracken et al. call for a more accurate description of how group membership is operationalized.

Supervisory opportunity to observe moderates criterion‐related validity estimates

“…On the surface, these results appear inconsistent with the observation of measurement equivalence across the different levels of opportunity to observe in Study 1. The results might also be viewed as inconsistent with work by Viswesvaran, Schmidt, and Ones () and Viswesvaran, Ones, and Schmidt (), suggesting that job performance ratings are composed of true score variance and rater idiosyncratic error and are not impacted by source of rating (e.g., supervisors vs. peers). One possible explanation could be that at low levels of opportunity to observe, supervisors' ratings are contaminated by other variables (e.g., extroversion) that are a source of shared variance across supervisors but are not related to job performance.…”

A Failed Challenge to Validity Generalization: Addressing a Fundamental Misunderstanding of the Nature of VG

“…In all these cases, statistical and psychometric misunderstandings led to misinformed analyses and/or inaccurate results. Significant resources and subsequent articles had to be devoted to correcting the scientific record (e.g., Harris et al, 2012;Ones, Viswesvaran, & Schmidt, 2012;Roth et al, 2014;Roth, Le, Oh, Van Iddekinge, & Robbins, 2017;Viswesvaran, Ones, & Schmidt, 2016;and Viswesvaran, Ones, Schmidt, Le, & Oh, 2014).…”