Investigating Cognitive Effort and Response Quality of Question Formats in Web Surveys Using Paradata

Abstract

Measuring attitudes and opinions employing agree/disagree (A/D) questions is a common method in social research because it appears to be possible to measure different constructs with identical response scales. However, theoretical considerations suggest that A/D questions require a considerable cognitive processing. Item-specific (IS) questions, in contrast, offer contentrelated response categories, implying less cognitive processing. To investigate the respective cognitive effort and response quality associated with A/D and IS questions, we conducted a web-based experiment with 1,005 students. Cognitive effort was assessed by response times and answer

Measuring Trust in Medical Researchers: Adding Insights from Cognitive Interviews to Examine Agree-Disagree and Construct-Specific Survey Questions

“…We often interpret longer response latencies and the presence of the behavioral indicators we measured as signs of cognitive response difficulty (e.g., Bassili and Scott 1996;Schaeffer and Dykema 2011). However, recent theorizing and evidence suggests these outcomes may be desirable when comparing AD and CS questions (Höhne and Lenzner 2018;Höhne et al 2017). Because CS questions use construct-specific response categories, which will likely vary on a questionby-question basis in a battery, they encourage deeper processing, which will increase processing time and likely result in respondents producing other behaviors when attempting to respond in an optimal manner.…”

“…In addition, Liu et al (2015) reported extreme response style was present for both AD and CS formats based on latent class factor analysis. Finally, in a web survey comparing questions presented stand-alone or in grids, Höhne et al (2017) reported no differences between AD and CS questions for data quality as indicated by nondifferentiation and dropping out of the survey before completion.…”

“…Response latencies (RLs) capture the length of time participants spend processing while they are formulating answers to survey questions (Bassili and Scott 1996;Draisma and Dijkstra 2004). We predicted longer latencies for the CS questions because they use variable construct-specific response categories, encouraging deeper processing (Höhne and Lenzner 2018;Höhne et al 2017).…”

“…Of course, the model for processing AD questions assumes respondents are optimally engaged with the task of responding and attentively progress through the steps. Höhne and colleagues (Höhne and Krebs 2018;Höhne and Lenzner 2018;Höhne et al 2017) argue this may not be the case. Because AD questions are usually presented in multi-item batteries in which the wording of the statements vary but the response categories remain the same -always some form of agreement to disagreement -they encourage superficial processing.…”

Developing an Approach to Multimodal Quantitative Assessment of Interviewers’ Cognitive Load: First Results of a Field Quasi Experiment

“…Ключевые слова: когнитивная нагрузка; мультимодальная оценка когнитивной нагрузки, связанной с проведением интервью; PAPI и CAPI; качество опросных данных; шкала Пааса; частота сердечных сокращений 1 ВОПРОСЫ ИСТОРИИ, ТЕОРИИ И МЕТОДОЛОГИИ Величина когнитивной нагрузки на интервьюера и респондента при проведении опроса -ключевой компонент оценки затрачиваемых ими когнитивных ресурсов и умственных усилий, который позволяет не только локализовать потенциальные источники ошибок измерения или учесть риски использования интервьюерами или респондентами стратегий, минимизирующих усилия и время заполнения в ущерб качеству данных, но и выбирать оптимальные форматы вопроса [11], управлять эффектами социальной желательности [17] или планировать оптимальное сочетание способов проведения опроса (например, телефонное интервью, онлайн-опрос и т.д.) и технических решений, касающихся, в частности, использования компьютерных технологий в сборе данных [см., напр.…”