Item Nonresponse in Web Versus Other Survey Modes: A Systematic Review and Meta-Analysis

Abstract

Nonresponse is a fundamental issue in survey research, due to the trend of declining response rates across survey modes. This issue is particularly serious for web surveys: A recent meta-analysis found that unit nonresponse in web surveys is, on average, higher by 12 percentage points than in other modes ( Daikeler et al., 2020 ). Although the issue of unit nonresponse in web and other survey modes has been investigated in several meta-analyses, item nonresponse has not been examined meta-analytically, a gap addressed by this paper. Are web surveys at a disadvantage compared to other survey modes in terms of item nonresponse as well? To address this question, 13 eligible experimental manuscripts reporting 23 effect sizes were identified in a comprehensive literature search. Meta-analytic findings showed that there was no statistically significant difference in the average item nonresponse rate in web versus other survey modes. However, six moderator variables were found to statistically significantly affect the relation between the survey mode and the item nonresponse rate, namely, the target population, the number of contacts, the mode to which the web survey mode was compared, the survey sponsor, the age of the survey, and the baseline item nonresponse rate of the compared mode. The main practical implication is that while web and other survey modes differ in terms of unit nonresponse (on average), item nonresponse rates in web surveys are similar compared to other modes.

Modeling unit non-response and validity of online teaching evaluation in higher education using generalizability theory approach

“…As such, the majority of the students did not participate in this exercise. This trend of result reflects the outcome of recent studies that have reported unit non-response as a common phenomenon in online surveys ( Marcham et al, 2020 ; Čehovin et al, 2022 ; Falk and Thies, 2022 ; Plante et al, 2022 ). Nevertheless, the recorded high unit non-response rates in this context cannot be entirely attributed to the transition from a paper-and-pencil survey to an online survey.…”

“…This transition has also led to decreased rates of student participation, resulting in high rates of non-response ( Dillman et al, 2002 ). Several studies have shown that unit non-response (i.e., individuals not responding to any item on the survey instrument or complete non-participation) in online evaluation surveys is a key challenge for the introduction of technology in students’ appraisal of teaching and learning ( Guder and Malliaris, 2013 ; Marcham et al, 2020 ; Čehovin et al, 2022 ; Falk and Thies, 2022 ; Plante et al, 2022 ).…”

“…These previous studies have stressed that the sources of variation in students’ appraisal of teaching are attributed to raters (i.e., students), items, occasions, course types, and teacher characteristics unrelated to teaching. Meanwhile, the increasing rate of non-response to online teaching evaluation survey in higher education have been argued to also influence the validity of teaching evaluations ( Adams and Umbach, 2012 ; Guder and Malliaris, 2013 ; Marcham et al, 2020 ; Čehovin et al, 2022 ; Falk and Thies, 2022 ; Plante et al, 2022 ).…”

Prospective association of eHealth literacy and health literacy with physical activity among Chinese college students: a multiple mediation analysis

“…Second, the results are only based on the responses among Chinese college students, therefore its application in other groups or areas needs to be examined. Third, online self-report questionnaires were applied, which may result in unreliability and inaccuracy because of the inherent drawbacks of self-reporting (i.e., recall bias, over- or under-reporting, non-response error, and social desirability) ( 118 , 119 ). Fourth, it was valuable that the findings give new insights into the antecedents of behavior intention (>60%), yet the whole model only explained 6% of PA, hinting that more components and covariates need to be identified in the future (e.g., volitional factors between intention and behavior) ( 120 ).…”

Does survey administration mode relate to non-substantive responses? A comparison of email versus phone administration of a residential utility-sponsored energy efficiency program survey

“…Another study found that an interviewer led survey had lower item nonresponse than PC or Webbased survey designs (Lee et al 2019). A more recent meta-analysis of item nonresponse in Web versus other modes found no statistically significant difference in the average item nonresponse rate in Web versus other survey modes (Čehovin, Bosnjak, and Lozar Manfreda 2022).…”