The Factorial Survey: The Impact of the Presentation Format of Vignettes on Answer Behavior and Processing Time

Abstract

The factorial survey is an experimental design in which the researcher constructs varying descriptions of situations or individual persons (vignettes), which will be judged by respondents with regard to a particular aspect. Some researchers present vignettes in text format as short stories, others present the central information of vignettes in a tabular format. To date, only a few sentences have been published, by Auspurg and Hinz, on the impact of the presentation format (text vs. table) on the answer behavior of students. Empirically, no differences were found between either format. Based on an Internet experiment conducted with a quota sample, we find evidence that ordinary tabular formats outperform text vignettes in terms of total vignette nonresponse but not when it comes to processing time. The former result especially applies in the case of less well-educated people. We further find that tabular format does not perform worse than text format regarding response inconsistency.

Heterogeneous effects of graduates’ international mobility on employers’ hiring intentions—experimental evidence from Germany

“…Too much or too little information varied in vignettes can make respondents tend to use heuristics to form their intentions, rather than bestow serious consideration (cf. Shamon et al 2019). Consequently, we varied graduates' characteristics, which are reported to be the most important in the empirical literature and which are the most crucial in terms of our theoretical considerations.…”

“…As relevant information is presented equally to all respondents in vignettes, the response process is less prone to distortions through individual interpretations (cf. Shamon et al 2019). Accordingly, factorial survey experiments enable a high level of construct validity.…”

Gatekeepers of transformation: private landowners evaluate invasives based on impacts to ecosystem services

“…We considered respondents to be unconditional responders if their ratings of acceptability or management intentions did not vary across the vignettes they evaluated. Although unvarying answer patterns across vignettes are sometimes treated as protest responses (Champ et al 2003) and can be related to cognitive satisficing (Shamon et al 2019), we recognize that unconditional responses can also reflect valid opinions. For instance, a landowner who values native rangelands with historical fidelity to pre-European conditions may reject all new species.…”

“…Additionally, the bulleted list is similar to the format of common plant identification descriptions that agricultural landowners are likely familiar with, which may reduce cognitive burden on respondents (Auspurg and Hinz 2015). We used bold text to support information intake by indicating where variation in each attribute occurred (Shamon et al 2019).…”

The application of factorial surveys to study recruiters’ hiring intentions: comparing designs based on hypothetical and real vacancies

“…A better understanding of the implications of the differences in FSE designs between studies is surely needed to establish best practices in employer studies. Prior research on general methodological questions related to the design of FSEs (e.g., Auspurg et al 2019;Auspurg and Jäckle 2017;Sauer et al 2011;Shamon et al 2019) continues to be highly relevant for the application of FSEs to study employer preferences, as it provides common guidelines for designing vignettes and answer scales. More studies on methodological aspects of FSEs are required, however, to advance our understanding of their possibilities and limits in research on hiring.…”