Advertising for Cognitive Interviews

Abstract

Researchers commonly rely on relatively small convenience samples for cognitive pretesting questionnaires. Methods used to recruit these samples vary depending on the population of interest, study timeline, study budget, and other factors. Over the past decade, one method that has become popular because of these considerations is online classified advertisements (e.g., Craigslist ads). A concern with the use of this recruitment method is that it leads to a set of participants who repeatedly participate in cognitive interview studies, changing the cognitive processes used in interviews, potentially resulting in misleading findings. Advertisements placed on social networking sites, such as Facebook, may give researchers more control over targeting recruitment advertisements, produce more participant diversity, and reduce the prevalence of ''professional participants'' who respond to ads. Recent research has shown that Craigslist and Facebook advertisements do result in selection pools with different demographic characteristics and experiences as study participants. However, we are not familiar with any research that has attempted to address concerns about data quality as a result of (a) professional participant cognitive bias or (b) recruitment method. Using data from two studies for which recruitment advertisements were placed on Craigslist and Facebook, we assess whether there are differences in recruitment speed, demographic diversity, the extent to which professional participants comprise the recruitment pool, and the extent to which a geographically dispersed recruitment pool can be attained. Evidence across the measures of quality was mixed. Facebook advertisements resulted in much faster recruitment than Craigslist advertisements among an online population in which the study topic was virtual worlds and avatars (Study 1), but the inverse was true among an older population in which the study topic was long-term care (Study 2). Mixed evidence was also found for relationships between recruitment platform and demographic composition. In Study 1, we found relationships between recruitment method and education, ethnicity, and race. In Study 2, there was only a relationship between recruitment platform and marital status and employment status. Furthermore, Facebook recruits were significantly younger than Craigslist recruits in Study 1, while in Study 2, Craigslist recruits were younger. Downloaded from Professional participants were identified in the recruitment pools when the concept was operationalized as attempts to deceive the researchers in how they learned about the study. No evidence was found, however, when professional participant was operationalized as the number of times one had participated in research in the past 12 months. Finally, while no comparison is available between platforms, we found that Facebook advertisements resulted in a geographically dispersed recruitment pool with per capita rates ranging from 0.05 to 1.6 and only one state having no representation. The findings from this research wil...

The Use of Social Media in Recruitment for Medical Research Studies: A Scoping Review

“…The article by Head et al [35] has 2 studies and has been counted as 2 articles for the purpose of Figure 1. In further sections of this paper, the article by Head et al [35] is counted as a single article or 2 articles, according to whether the conclusions from the 2 studies pertinent to the outcomes of this paper are the same or different.…”

“…In further sections of this paper, the article by Head et al [35] is counted as a single article or 2 articles, according to whether the conclusions from the 2 studies pertinent to the outcomes of this paper are the same or different. Out of 30 studies, 12 studies (40%) reported higher rates of recruitment through social media as compared with any of the other methods used [14-17,26,28,31,32,35, 36,41,42] and 15 studies (50%) reported recruitment via social media to be less effective than at least one other method used [18-21,23-25, 27,33-35,37-40]. Heffner et al [20] and Rabin et al [24] found social media to be the least effective method out of multiple (>2) recruitment methods used.…”

“…Of the 12 studies that found social media to be the best method of recruitment, 8 were observational studies [14,16,17,26,28,31,35,41] and the remaining 4 were interventional studies [15,32,36,42], as shown in Multimedia Appendix 2. In addition, 6 of these studies targeted populations deemed hard to reach [16,17,28,31,32,36], and 6 studies targeted specific conditions or disorders [14,16,17,28,31,32].…”

“…Furthermore, 1 study targeted only young adults (aged 23-30 years) [14]. Among the 12 studies, 8 studies used only Facebook for recruitment [14-16,28,35,36,41,42]. Of the remaining 4 studies, 2 studies used a combination of Facebook and Twitter [31,32] and 2 studies used a combination of Facebook and other social media websites [17,26].…”

Migrant Sampling Using Facebook Advertisements

“…More precisely, we used Facebook advertisements to test whether these could be applied to sample migrants and convert them into survey participants. To our knowledge, this method has so far only been applied by a limited number of studies that were all conducted in the area of medical and health research (Arcia, 2014;Chu & Snider, 2013;Fenner et al, 2012;Kapp, Peters, & Oliver, 2013;Lohse, 2013;Ramo & Prochaska, 2012) or to recruit participants for cognitive pretests for which monetary incentives were offered (Head, Dean, Flanigan, Swicegood, & Keating, 2016).…”