Examining Contexts-of-Use for Web-Based and Paper-Based Questionnaires

Abstract

Questionnaire instruments are frequently administered in digital formats, largely webbased, without much systematic investigation of possible effects from these administration methods. Furthermore, little attention has been given to the contextual lack of control for extraneous factors that may influence user responses. In this study, 263 university students were randomly assigned to one of two administration formats, web-based (WBA) or paper-based (PBA), to complete a set of questionnaires in an environment of their choice. Data collection included reporting context characteristics along three parameters: location, companions, and concurrent activities (including help-seeking). Outcomes of interest included location and conditions of user-chosen contexts, instrument performance, generative data quantity and quality, independence of completion, administrative efficiency, and participant affect. Participants did choose and allow distracters in their contexts-of-use, completing the questionnaires while engaged in multiple social and asocial concurrent activities. There were generally small but significant differences in instrument performance and user response characteristics by administration method and contexts-of-use. Participant comfort and data returned were both higher in PBA than WBA. Quantity return of generative data was higher in WBA while overall quality (completeness, coherence, correctness) of generative data was not significantly different. These findings present implications of administrative methods and contextual influences that inform measurement professionals' selection and design of administrative systems and conditions for research and evaluation data collection.

The Problem with Online Data Collection: Predicting Invalid Responding in Undergraduate Samples

“…The results from this study are in-line with a number of extant studies already published that suggest online research does not always result in valid data collection (Aust et al, 2013;Burnham & Hare, 2007;Hardre, Crowson, & Xie, 2012;Ihme et al, 2009;Oppenheimer et al, 2009;Ward & Pond, 2015). Indeed, like previous published work, our results suggest that more than 10% of our participants contributed data that is unlikely to be valid (DeRight & Jorgenson, 2015).…”

“…Although internet-based research clearly benefits researchers, the validity of the data collected is unclear. Online data collection is assumed to provide anonymity and therefore participants are more likely to respond candidly and genuinely, there is evidence to suggest this is not always the case (Aust et al, 2013;Hardre, Crowson, & Xie, 2012;Ihme et al, 2009;Oppenheimer et al, 2009;Ward & Pond, 2015). There are likely a number of factors that influence the quality of data provided.…”

“…There are likely a number of factors that influence the quality of data provided. The physical disconnection from the researcher may increase the likelihood of careless responding (Hardre, Crowson, & Xie, 2012). The presence of a researcher in the room with the participant may also play a role in their performance, as evidenced by data from a study that randomly assigned participants to a room with a researcher present or a room with no researchers present (Burnham & Hare, 2007).…”

Partnership-Level Analysis of African American Women’s Risky Sexual Behavior in Main and Non-Main Partnerships

“…Future research using longitudinal data would provide a stronger basis for such inference, as well as allow for examination of how gendered power dynamics may change over time within the same partnership. Additionally, all data were gathered online, with no control over potentially distracting environments, which may have affected data quality (30). We did not assess the reproductive goals of our participants; and if women were trying to become pregnant, this issue could greatly influence sexual behavior.…”

Uncovering the Evidence: Faculty Perceptions of Distance Library Services

“…The value of using online surveys for geographically distributed populations is mainly one of convenience and efficiency. Researchers are able to target potential respondents instantly, and thanks to analytical features included in online survey software, turnaround time for collection and analysis of responses is minimized (Evans & Mathur, 2005;Hardré, Crowson, & Xie, 2012). …”