A review of quality issues associated with studying hard-to-survey populations

Abstract

Some populations/groups are defined as hard to survey (H2S) in the research literature because of difficulty sampling the population, accessing the population, contacting members of the population, persuading them to participate, and interviewing group members (see Tourangeau, Chapter 1 in this volume). Reviewing the literature on populations referred to as "hard to survey" indicates the types of studies undertaken with these populations, the survey error challenges, and the methodological solutions used by researchers who study such populations. The H2S literature, however, does have limitations and research gaps exist.The problems that complicate obtaining high-quality data from H2S populations are diverse and often interrelated. This review uses a quality assurance and quality control framework to discuss the total survey error associated with H2S studies. The discussion is based on an extensive systematic review of the quantitative and qualitative literature related to H2S, as well as our own experiences surveying these populations. (For a discussion and definition of kinds of H2S populations and the characteristics that earn them the H2S label, see Chapter 1 in this volume.)We identify literature from various disciplines, relate these to the H2S challenges, and note which solutions are proposed or criticized. In the overview of challenges and practices, we also identify research gaps and general limitations that need to be addressed. Here, we consider types of error and survey process stages less frequently discussed or for which concrete data and solutions are sparse. We identify pertinent quality assurance/control steps to address such gaps and point to tools to remedy important measurement and documentation deficits. We hope that our systematic review can generate improved practices for surveying H2S populations.Our literature review on H2S populations entailed searching keywords related to the topic in academic search databases, including Academic Search Premier/EBSCO, JSTOR, and Five survey research and methodology students and three survey research and methodology professors and professionals undertook an appraisal process for this chapter. The students were

Building partnerships in education through a story-tool based intervention: Parental involvement experiences among families with Roma backgrounds

“…The first data wave was collected before the beginning of the intervention. This interview was initiated with general information about the program to overcome mistrust and cultural and linguistic barriers and help establish relationships with the participants ( Lyberg et al, 2014 ). The interview included open questions focused on participants’ beliefs on the value of education, academic expectations, parenting roles, and practices regarding their children’s education.…”

Successes and Challenges in Recruiting and Retaining Gang Members in Longitudinal Research

“…Other more intensive sampling techniques (but still under the umbrella of nonprobability sampling) can be used when the above-mentioned methods are not sufficient to yield the samples necessary to rigorously address a study’s research questions. Target/location sampling involves determining where members of the hidden population spend substantial amounts of time and visiting those places to conduct recruiting (Lyberg et al, 2014; Marpsat & Razafindratsima, 2010; Peterson et al, 2008). Generally, target/location sampling efforts involve constructing a list of places where potential respondents go to receive services or interact.…”

“…Two other methods used to locate respondents of hidden populations are snowball sampling and respondent-driven sampling (RDS); (Lyberg et al, 2014; Marpsat & Razafindratsima, 2010; Peterson et al, 2008; Platt et al 2006; Semaan, 2010). Snowball sampling, also known as chain referral, relies on the survey respondent to recruit new members or supply the researcher with additional names of those who meet the study criteria.…”

“…Other studies have used less structured community-based methods, where employees of social agencies who already have access to and rapport with the population of interest are used as recruiters or field workers (Brannon et al, 2013; Lyberg et al, 2014). Conducting outreach directly to reach individuals with the targeted characteristic of interest (e.g., gang involvement), where sampling for proportionality is not the main concern, is known as purposive sampling.…”

“…While there are many published works reporting the results of studies of hard-to-survey and hidden populations, there are few pieces which focus mainly on the methodologies used to complete these undertakings (Lyberg et al, 2014). In addition to this gap in the literature on hidden and hard-to-survey populations in general, in gang research specifically, there have been calls for studies employing larger sample sizes and longitudinal efforts to better measure gang processes such as group identity and gang desistance (Pyrooz & Decker, 2011; Pyrooz, Sweeten, & Piquero, 2013; Sweeten, Pyrooz, & Piquero, 2013).…”