Representativeness of individual-level data in COVID-19 phone surveys: Findings from Sub-Saharan Africa

Abstract

The COVID-19 pandemic has created urgent demand for timely data, leading to a surge in mobile phone surveys for tracking the impacts of and responses to the pandemic. Using data from national phone surveys implemented in Ethiopia, Malawi, Nigeria and Uganda during the pandemic and the pre-COVID-19 national face-to-face surveys that served as the sampling frames for the phone surveys, this paper documents selection the biases in individual-level analyses based on phone survey data. In most cases, individual-level data are available only for phone survey respondents, who we find are more likely to be household heads or their spouses and non-farm enterprise owners, and on average, are older and better educated vis-a-vis the general adult population. These differences are the result of uneven access to mobile phones in the population and the way that phone survey respondents are selected. To improve the representativeness of individual-level analysis using phone survey data, we recalibrate the phone survey sampling weights based on propensity score adjustments that are derived from a model of an individual’s likelihood of being interviewed as a function of individual- and household-level attributes. We find that reweighting improves the representativeness of the estimates for phone survey respondents, moving them closer to those of the general adult population. This holds for both women and men and for a range of demographic, education, and labor market outcomes. However, reweighting increases the variance of the estimates and, in most cases, fails to overcome selection biases. This indicates limitations to deriving representative individual-level estimates from phone survey data. Obtaining reliable data on men and women through future phone surveys will require random selection of adult interviewees within sampled households.

On the forecastability of food insecurity

“…Since the mode of questionnaire administration can have serious effects on data quality 45,46 , during the last few years WFP conducted mode experiments in several countries, each demonstrating the feasibility of collecting food security indicators via CATI surveys 47,48 . However, sampling and selection bias should be assessed and mitigated [49][50][51] , hence post-stratification weights were applied by WFP, as detailed in the Methods section.…”

On the forecastability of food insecurity

“…Since the mode of questionnaire administration can have serious effects on data quality [43, 44], during the last few years WFP conducted mode experiments in several countries, each demonstrating the feasibility of collecting food security indicators via CATI surveys [45, 46]. However, sampling and selection bias should be assessed and mitigated [47, 48, 49], hence post-stratification weights were applied by WFP, as detailed in the Methods section.…”

Covid‐19 and agricultural labor supply: Evidence from the rural–urban interface of an Indian mega‐city

“…The advantage of this is that we were able to interview the households in real time and with a very short recall period of about 6–8 weeks. We could base our phone survey on an existing, largely representative, sampling framework and achieved a high response rate that raised no major concerns about selection bias (Ambel et al, 2021; Brubaker et al, 2021), which was facilitated by the widespread ownership of mobile phones of the sample households. Since we had conducted in‐depth interviews with the same households only 1–2 months before the lockdown, we could develop our telephone survey module based on detailed information previously collected and thereby reduce the length of the questionnaire and the interview time to avoid response fatigue (Abay et al, 2021).…”