Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Abstract

This study seeks to answer a simple question: Is it possible to develop a scoring key for a situational judgment test (SJT) without a pool of subject matter experts (SMEs)? The SJT method is widely studied and used for selection in both occupational and educational settings (Oswald et al., 2004;Lievens & Sackett, 2012). SJTs are typically designed to measure procedural knowledge about how to behave effectively in a particular job (Motowidlo et al., 2006). Along these lines, SJT items are typically scored using rational keys based on ratings gathered from SMEs (Whetzel et al., 2020). According to the U.S. Office of Personnel Management, SMEs are defined as a "person with bona-fide expert knowledge about what it takes to do a particular job" (Office of Personnel Management, 2016). Their judgments determine how each item will be scored. This poses a challenge when creating a new assessment because test developers typically need an item pool at least twice as large as the desired length of the final test (Hinkin, 1998;Murphy & Davidshofer, 1998). Each of these items needs a scoring key in order to be evaluated during an initial round of testing. This means that SMEs would likely spend most of their time rating items that will not be used in the final version of the test.Researchers and test developers face several challenges when using SMEs. The criteria for determining who is considered a SME can be vague or inconsistent across studies. When SJTs are developed for a specific occupation, SMEs have included job incumbents, supervisors, customers, and even novices (Weekley et al., 2006). In many cases, these SMEs work in prestigious occupations (e.g., experienced physicians or medical professionals, Lievens & Sackett, 2012;Patterson et al., 2009) which can make them expensive and difficult to recruit. For more construct-driven SJTs (e.g., applied social skills), academic researchers or graduate

Do situational characteristics affect the validity of personality situational judgment items?

“…6 A growing body of research suggests that MTurk can be an adequate source of data if researchers are attentive to data quality issues (Cheung et al, 2017;Michel et al, 2018;Paolacci & Chandler, 2014). Specific to SJTs, Brown et al (2021) found that MTurk participants' ratings of SJTs correlated quite strongly (r values ≥ .88) with ratings from subject matter experts, a finding supported by our own comparisons (presented later). Only participants who were US-based, at least 18 years old, and had at least 95% of their previous tasks accepted by other requesters were invited to participate (also, see Table 2 for additional data quality controls).…”

Exploring the Role of Social Desirability in Situational Judgment Tests

“…This is strikingly similar to proportion consensus scoring approaches which have been used to score measures of emotional intelligence (Legree et al, 2005), tacit knowledge (Hedlund et al, 2003) and practical intelligence (Fox & Spector, 2000). Likewise, consensus scores have also been found to converge with expert-based scoring keys for different SJTs (Brown et al, 2021).…”

“…Responses were scored as either 0, 1, 2, or 3 based on effectiveness ratings gathered from SMEs. Scores on this test have been reported to correlate positively with other performance measures of social and emotional intelligence and the ability to evaluate recordings of structured interviews or to identify more effective interview questions (Brown et al, 2021;Speer et al, 2019;Speer et al, 2020). The workplace skill SJT demonstrated weaker internal consistency (α = .36).…”

“…Instead, each potential behavior is evaluated based on human judgment. These judgments may be gathered from select samples of identified experts or based on large-scale, crowdsourcing aggregates (e.g., Brown et al, 2021). During test development, item response options are evaluated based on agreement among SMEs with an emphasis on selecting behaviors that are clearly distinguishable as effective or ineffective (McDaniel et al, 2011;Whetzel et al, 2019).…”

Potential bias when using social media for selection: Differential effects of candidate demographic characteristics, race match, perceived similarity, and profile detail

“…Five hundred working adults recruited through Amazon's Mechanical Turk (“MTurk”) participated in this study. We chose MTurk because it is more generalizable than student samples, is appropriate for workplace research (Highhouse & Zhang, 2015), and has been used when examining questions related to selection (e.g., Brown et al, 2021; Zhang et al, 2018). Upon completion of the survey, participants were paid $2.50.…”