The validity of qualtrics panel data for research on video gaming and gaming disorder.

Abstract

Crowdsourced samples are increasing in popularity, particularly within psychological and addictive behaviors research. The trend has resulted in significant interest in the use of panel samples for the examination of behavioral and substance addictions. One newer panel platform, Qualtrics, has seen an increase in usage in recent years despite lack of research examining the validity of Qualtrics-produced data for addictive behaviors. The goal of the present study was to evaluate the validity of Qualtrics-obtained data for the most recently classified behavioral addiction-video gaming. The evaluation compared a Qualtricsrecruited video gamers sample (n = 586) to traditional community (n = 108) and student (n = 217) samples on demographics and key outcomes relevant to gaming disorder researchers (average playtime, frequency of gaming, and gaming disorder risk scores) to evaluate the generalizability of Qualtrics panel data. The results revealed that Qualtrics samples were comparable to a traditionally recruited community sample, but different from a student sample on gaming frequency ( p < .001) and risk for gaming disorder ( p < .001). The Qualtrics sample also had longer durations of average gaming time relative to the student sample ( p = .01), with some differences in demographics between the all three sources of recruitment. The findings suggest that Qualtrics may provide a suitable method of convenience panel recruitment, generalizable to the broader North American community, for research examining video gaming behaviors and gaming disorder. Public Health SignificanceThe findings of the study demonstrate that Qualtrics-recruited panel samples are comparable to community samples for investigations of video gamers and gaming disorder with regard to mental health and video gaming outcomes. Key differences exist in demographic profiles between Qualtrics, community-recruited, and student samples of video gamers.

Operationalization and measurement of compulsivity across video gaming and gambling behavioral domains

“…Solutions with more than a single factor were not theoretically interpretable and presented substantial cross-loadings. In line with previous studies using panel recruitment for similar purposes, mean severity scores and proportions of individuals above the clinical cut-off in both samples of the current study were high, relative to samples obtained with other recruitment methods [50]. In other words, our factor analysis seems to be valid for the whole severity continuum and is not compromised by scores' range restriction.…”

Victim-blaming and image-based sexual abuse

“…To address these criticisms, Qualtrics have implemented techniques including a ‘prevent ballot box stuffing’, which uses internet browser cookies to prevent multiple survey completions and RelevantID, which ‘assesses participant metadata to detect fraudulent behaviors’ (Mullen et al, 2021, p. 223). Further, while not statistically representative, and to be interpreted with caution, research using non-probability online panels, including Qualtrics, have demonstrated similar results to probability-based panels that are population-based (Belliveau et al, 2022; Mullinix et al, 2015).…”

Convenience sampling methods in psychology: A comparison between crowdsourced and student samples.

“…Moreover, studies have also demonstrated inconsistency in research outcomes when using university student samples, suggesting that student sampling methods can introduce significant problems in replicability (Peterson & Merunka, 2014). Despite such challenges to generalizability, psychology researchers continue to utilize student participant pools due to the convenience and facility of this recruitment procedure, its low cost, and its assumed low response bias (Belliveau et al, 2022; Landers & Behrend, 2015).…”

“…Over the last decade, alternative convenience sampling methods have gained traction among social science researchers. These approaches involve crowdsourcing online panel data from platforms such as Qualtrics or Amazon’s Mechanical Turk (MTurk) which recruit a large and diverse pool of respondents who agree in advance to participate in survey research on various subjects and receive payment for participation through cash incentives, gift cards, or charitable contributions (Belliveau et al, 2022; Palmer & Strickland, 2016). The ability to recruit and include participants from a larger population pool (e.g., country wide) using online panel data may minimize barriers associated with convenience samples, thus offering data that are more representative and generalizable than that from student samples (Behrend et al, 2011; Landers & Behrend, 2015).…”