Supervisory opportunity to observe moderates criterion‐related validity estimates

Abstract

Supervisors' opportunity to observe incumbents' performance (i.e., how often a supervisor typically sees an employee's performance) has been suggested to be important for accurate performance rating and to be a moderator of criterion-related validity. Here we test these suggestions and present empirical evidence for the effects of opportunity to observe. In Study 1, supervisors in a multi-occupation/ organization criterion-related validation study for a biodata measure indicated the opportunity they had to observe incumbents. The data were split according to different levels of opportunity to observe. Higher validities were found when opportunity to observe was maximal. In Study 2, this finding was replicated using a cognitive ability test. These results suggest that psychologists should consider measuring opportunity to observe in criterion-related validity studies.

Personnel selection: A review of ways to maximize validity, diversity, and the applicant experience

“…Another key issue is how organizations assess the performance of gig and remote workers. In traditional work, there are already issues that supervisors often have limited opportunity to observe employees (MacLane et al., 2020). This seems to be exacerbated in gig and remote work.…”

Rumors of general mental ability’s demise are the next red herring

“…Paradox 5: The folly of selecting for contextual performance but expecting proficiency Sackett et al (2023) state that contextual criteria are not as predictable by GMA as are task-based criteria. Although supervisory ratings are easily obtained and provide an aura of independent authoritative judgment, they are clouded by social/organizational factors (Murphy & Cleveland, 1995), impacted by supervisors' opportunity to observe performance (MacLane et al, 2020), and biased by other factors (e.g., subjectivity, criterion deficiency, poorly developed rating scales; Courtney-Hays et al, 2011). Practices in how ratings are collected impact criterion-related validity (Grubb, 2011).…”