Are Smileys Valid Answers? Survey Data Quality with Innovative Item Formats

Abstract

Introduction: Survey research is often designed based on multiple-choice questions although many other formats, also referred to as innovative item formats (IIF) exist, such as ranking, sorting, questions with pictures or smileys as response options. Research has suggested that IIF in a broad sense can strengthen data quality, but research is needed on a more specific level. Therefore, the purpose of this article is to present research for separate IIFs about the data quality for that type of item. Method: A literature study was conducted to identify articles that test the data quality of IIF. For each IIF research was discussed regarding aspects of data quality, such as reliability, validity and response rate. Results: A total of 166 research articles were identified with data from 218,532 participants revealing aspects of 22 IIFs with 13 subcategories. The type of evidence on data quality is quite varied and for some IIFs the evidence is supportive, for some it is not and for some it is inconclusive. With 6 IIFs the evidence was estimated in favour hereof, for 11 IIFs the evidence was inconclusive, 1 unfavoured and for 3 there was no evidence. With 6 IIFs potential confounders were identified. Discussion: The study suggests further research is needed where evidence is scarce. The present study could initiate more extensive systematic reviews within specific categories of IIF.