Agree to disagree: Examining the psychometrics of cybervetting

Abstract

PurposeDue to the paucity of research on web-based job applicant screening (i.e. cybervetting), the purpose of the current study was to examine the psychometric properties of cybervetting, including an examination of the impact of adding structure to the rating process.Design/methodology/approachUsing a mixed-factorial design, 122 supervisors conducted cybervetting evaluations of applicant personality, cognitive ability, written communication skills, professionalism, and overall suitability. Cross-method agreement (i.e. the degree of similarity between cybervetting ratings and other assessment methods), as well as interrater reliability and agreement were examined, and unstructured versus structured cybervetting rating formats were compared.FindingsCybervetting assessments demonstrated high interrater reliability and interrater agreement, but only limited evidence of cross-method agreement was provided. In addition, adding structure to the cybervetting process did not enhance the psychometric properties of this assessment technique.Practical implicationsThis study highlighted that whereas cybervetting raters demonstrated a high degree of consensus in cybervetting-based attributions, there may be concerns regarding assessment accuracy, as cybervetting-based ratings generally differed from applicant test scores and self-assessment ratings. Thus, employers should use caution when utilizing this pre-employment screening technique.Originality/valueWhereas previous research has suggested that cybervetting ratings demonstrate convergence with other traditional assessments (albeit with relatively small effects), these correlational links do not provide information regarding cross-method agreement or method interchangeability. Thus, this study bridges a crucial gap in the literature by examining cross-method agreement for a variety of job-relevant constructs, as well as empirically testing the impact of adding structure to the cybervetting rating process.

Social Media Screening and Procedural Justice: Towards Fairer Use of Social Media in Selection

“…Despite the emphasis placed by various authors on the need to structure the process of gathering applicant information via social networks, the study of Schroeder et al (2020) found that adding structure to the process did not lead to the expected improvements in psychometric properties demonstrated in social media screening. In reaction to these findings, Schroeder et al (2020) presume that structuring methods other than those used in their study may be more effective. As a result, future research on the effectiveness of different methods of stucturing the process of social media screening may provide further insights for practitioners.…”

“…The practice of social media screening in selection raises data accuracy concerns in several ways: In their study, Schroeder et al (2020) demonstrate that the assessments of applicants based on information from social media screening were generally inconsistent with the information provided by the applicants. Regarding the quality and reliability of social media information, Davis (2007) has identified the problem of inaccurate or false information about candidates, especially beyond their own control, as being one of the most serious issues in social media screening.…”

“…As screening only certain applicants can be understood as a form of discrimination, it is recommended to perform the screening for all applicants (Bentley 2013;Davison et al 2016). Since employers' screening of social media in selection often occurs ad hoc and in an arbitrary manner (Schroeder et al 2020;Wade et al 2020), organizations should make efforts to standardize the process (Davison et al 2016;Landers and Schmidt 2016;Tews et al 2020;Wade et al 2020) in order to give all participants the chance to obtain positive results. Overall, a considerable number of authors point to the necessity of articulating a clear, standardized policy for carrying out social media screening (Elzweig and Peeples 2009;Davison et al 2012Davison et al , 2016Lam 2016;Landers and Schmidt 2016;Wade et al 2020) and disseminate it to the employees (Elzweig and Peeples 2009).…”

“…This policy should specify the conditions under which social media content should be collected, differentiate job-relevant from irrelevant and discriminatory information, and establish a standardized rating system to account for more objectivity (Tews et al 2020;Wade et al 2020). Despite the emphasis placed by various authors on the need to structure the process of gathering applicant information via social networks, the study of Schroeder et al (2020) found that adding structure to the process did not lead to the expected improvements in psychometric properties demonstrated in social media screening. In reaction to these findings, Schroeder et al (2020) presume that structuring methods other than those used in their study may be more effective.…”

“…The section on accuracy of information has demonstrated that the social media screening process is prone to error in many ways. As discussed earlier, the hiring decision should not be taken by looking at social media in isolation and, as pointed out by Schroeder et al (2020), social media should not be used to replace traditional selection methods. Rather, they should include a number of steps and sources of information traditionally used in the selection process, as proposed by Landers and Schmidt (2016).…”