The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

Abstract

Factorial survey experiments have been widely used to study recruiters' hiring intentions. Respondents are asked to evaluate hypothetical applicant descriptions, which are experimentally manipulated, for hypothetical job descriptions. However, this methodology has been criticized for putting respondents in hypothetical situations that often only partially correspond to real-life hiring situations. It has been proposed that this criticism can be overcome by sampling real-world vacancies and the recruiters responsible for filling them. In such an approach, only the applicants' descriptions are hypothetical; respondents are asked about a real hiring problem, which might increase internal and external validity. In this study, we test whether using real vacancies triggers more valid judgments compared to designs based on hypothetical vacancies. The growing number of factorial survey experiments conducted in employer studies makes addressing this question relevant, both for methodological and practical reasons. However, despite the potential implications for the validity of data, it has been neglected so far. We conducted a factorial survey experiment in Luxembourg, in which respondents evaluated hypothetical applicants referring either to a currently vacant position in their company or to a hypothetical job. Overall, we found little evidence for differences in responses by the design of the survey experiment. However, the use of real vacancies might prove beneficial depending on the research interest. We hope that our comparison of designs using real and hypothetical vacancies contributes to the emerging methodological inquiry on the possibilities and limits of using factorial survey experiments in research on hiring.

Hiring in border regions: experimental and qualitative evidence from a recruiter survey in Luxembourg

“…Overall, FSEs have been criticised for presenting respondents with hypothetical situations that may not depict real hiring situations. Indeed, past studies were based on student samples or samples of recruiters who may or may not have been familiar with recruiting for relevant job types (Gutfleisch et al 2021). The present survey was based partly on samples of real vacancies and the recruiters responsible for filling these vacancies.…”

“…The analyses were based on a factorial survey experiment (FSE) in which respondents were asked to evaluate experimentally manipulated fictitious descriptions of applicants. A vast majority of prior FSEs regarding hiring has focused on descriptions of hypothetical vacancies that may or may not correspond to real-life hiring situations (Gutfleisch et al 2021). Conversely, our recruiter survey was partly based on samples of realworld vacancies and included recruiters responsible for filling these, and this likely increased the external validity of the results.…”

Disclosing the ‘Big C’: what does cancer survivorship signal to employers?

“…More concretely, in the experiment, each participant was randomly assigned one out of nine fictitious vacancies (Gutfleisch et al, 2021). Our selection of nine vacancies varied across three job characteristics: required level of education, required cognitive abilities and required physical abilities.…”

El estudio de viñeta para el análisis de las contrataciones laborales: una perspectiva crítica

“…As the use of vignettes is a relatively recent technique in sociology, few studies have analyzed the effects of presenting hypothetical rather than real situations. In this regard, some studies have found no significant difference between evaluating one type of situation or the other, such as Hainmueller et al (2015) in an experiment on attitudes towards immigration in Switzerland and Gutfleisch et al (2021) in an experiment on labor recruitment in Luxembourg.…”