Evaluating the Data Quality of a National Sample of Young Sexual and Gender Minorities Recruited Using Social Media: The Influence of Different Design Formats

Abstract

Social media recruitment is no longer an uncharted avenue for survey research. The results thus far provide evidence of an engaging means of recruiting hard-to-reach populations. Questions remain, however, regarding whether the data collected using this method of recruitment produce quality data. This article assesses one aspect that may influence the quality of data gathered through nonprobability sampling using social media advertisements for a hard-to-reach sexual and gender minority youth population: recruitment design formats. The data come from the Survey of Today’s Adolescent Relationships and Transitions, which used a variety of forms of advertisements as survey recruitment tools on Facebook, Instagram, and Snapchat. Results demonstrate that design decisions such as the format of the advertisement (e.g., video or static) and the use of eligibility language on the advertisements impact the quality of the data as measured by break-off rates and the use of nonsubstantive responses. Additionally, the type of device used affected the measures of data quality.

Medical Gender Affirmation and HIV and Sexually Transmitted Disease Prevention in Transgender Youth: Results from the Survey of Today's Adolescent Relationships and Transitions, 2018

“…More information about the study design, recruitment, implementation, and data management can be found elsewhere. 24 We restricted the sample to sexually experienced transgender youth who self-reported being HIV negative or did not know their status and provided answers to all questions included in our analysis. Our analytic sample contained 1029 participants.…”

Assessing the quality of social media data: a systematic literature review

“…They choose for each social media platform, the techniques used for data capture, analysis and processing with the satisfied data quality model. The purpose of [23] is to address the issues of measurement and representativeness of data using the results of a study in which social media advertisements is used for a hard-to-reach sexual and gender minority youth population. They examine measures of data quality such as exiting the survey before completion, commonly known as "break-off", in addition to using the nonsubstantive responses to questionnaire questions such as "prefer not to answer" and "don't know".…”

Assessing self-selection biases in online surveys: evidence from the COVID-19 Health Behavior Survey

“…Section 2 2.3 Online surveys during the COVID-19 pandemic reported poorer mental health status (Choi et al 2017). Stern et al 2020 used Facebook to recruit young men in the United States from sexual minorities. While they found that using a video vs. using an image in the advertisement made no difference in the response rates, they also observed that respondents who entered the survey through the ad with the image had fewer nonsubstantive answers than those who entered the survey through the ad with the video.…”