The Impact of Targeted Data Collection on Nonresponse Bias in an Establishment Survey: A Simulation Study of Adaptive Survey Design

Abstract

Nonresponse rates have been growing over time leading to concerns about survey data quality. Adaptive designs seek to allocate scarce resources by targeting specific subsets of sampled units for additional effort or a different recruitment protocol. In order to be effective in reducing nonresponse, the identified subsets of the sample need two key features: 1) their probabilities of response can be impacted by changing design features, and 2) once they have responded, this can have an impact on estimates after adjustment. The National Agricultural Statistics Service (NASS) is investigating the use of adaptive design techniques in the Crops Acreage, Production, and Stocks Survey (Crops APS). The Crops APS is a survey of establishments which vary in size and, hence, in their potential impact on estimates. In order to identify subgroups for targeted designs, we conducted a simulation study that used Census of Agriculture (COA) data as proxies for similar survey items. Different patterns of nonresponse were simulated to identify subgroups that may reduce estimated nonresponse bias when their response propensities are changed.

Payment versus charitable donations to attract agricultural and natural resource survey participation

“…Ignoring these aspects (Neuman, 2012) or the respondent's perceived survey burden can decrease participation (Tourangeau & Plewes, 2013). Conversely, McCarthy et al (2006) found no association between refusing future NASS surveys and survey burden such as the number of previous NASS surveys completed. Other characteristics can affect participation, such as social attention, demographic characteristics, survey mode, and value of privacy and time.…”

“…Another related issue is survey burden (McCarthy et al, 2006). Bradburn (1978) outlines four components of survey burden: the survey length, required effort (e.g., question difficulty), the stress on the respondent (e.g., sensitive questions), and survey fatigue from frequent participation.…”

Adaptive Design in Surveys and Clinical Trials: Similarities, Differences and Opportunities for Cross-fertilization

“…These studies involved changing elements of the data collection approach (subsampling of nonrespondents, altering incentive amounts, interview mode changes and contact effort changes) in response to observed sample quality and the response propensity for cases that have not yet been interviewed. In establishment surveys, these and other sorts of adaptation have been explored (Thompson and Kaputa, 2017;McCarthy et al, 2017). Särndal and Lundquist (2014) suggested that adjusting for differential recruitment needs and response behaviours during or between rounds of data collection may be more effective than not making any adaptations and instead relying on post-survey statistical adjustment to account for non-response.…”

Response Burden Management for Establishment Surveys at Four National Statistical Institutes

“…Several studies indicate that a too high level of response burden may also affect the quality and timeliness of the collected data (e.g., Hedlin et al 2005;Bavdaz 2010;Giesen 2012;Jones 2012;Lorenc et al 2013;and Berglund et al 2013). However, it must be noted that more research is needed into the relationship between burden and response behavior, as for example McCarthy et al (2006) did not find a negative effect of accumulated burden on cooperation. Finally, as pointed out by Bavdaz ˇet al (2015), response burden can affect the strategic relationship between NSIs and the business world.…”