Assessing recall bias and measurement error in high-frequency social data collection for human-environment research

Abstract

A major impediment to understanding human-environment interactions is that data on social systems are not collected in a way that is easily comparable to natural systems data. While many environmental variables are collected with high frequency, gridded in time and space, social data is typically conducted irregularly, in waves that are far apart in time. These efforts typically engage respondents for hours at a time, and suffer from decay in participants' ability to recall their experiences over long periods of time. Systematic use of mobile and smartphones has the potential to transcend these challenges, with a critical first step being an evaluation of where survey respondents experience the greatest recall decay. We present results from, to our knowledge, the first systematic evaluation of recall bias in components of a household survey, using the Open Data Kit (ODK) platform on Android smartphones. We tasked approximately 500 farmers in rural Bangladesh with responding regularly to components of a large household survey, randomizing the frequency of each task to be received weekly, monthly, or seasonally. We find respondents' recall of consumption and experience (such as sick days) to suffer much more greatly than their recall of the use of their households' time for labor and farm activities. Further, we demonstrate a feasible and cost-effective means of engaging respondents in rural areas to create and maintain a true socioeconomic "baseline" to mirror similar efforts in the natural sciences. Keywords High-frequency data collection. Android smartphone. Microtasks for micropayments. Bangladesh One of the most significant challenges impeding our understanding of humanenvironment interactions is that quantitative data on social systems are not collected

Of Trackers and Tractors. Using a Smartphone App and Compositional Data Analysis to Explore the Link Between Mechanization and Intra-Household Allocation of Time in Zambia

“…For example, researchers have used GPS devices to measure plot sizes (Carletto et al, 2015a), fitness-trackers to capture energy expenditure (Zanello et al, 2017) and satellites to assess yields (Lobell et al, 2018). However, for socioeconomic data, researchers still largely rely on household surveys, which are prone to recall bias (Arthi et al, 2018;Bell et al, 2019;Fraval et al, 2018). The lack of reliable data collection methods for socioeconomic aspects that are difficult to recall has led to data suffering from poor quality and the neglect of potentially highly relevant research areas (Carletto et al, 2015b).…”

Temporal dimensions of reported life satisfaction in a low-income, agricultural environment

“…Levels of education were high: 30% of respondents had some form of higher education; 53% had attended school. These characteristics mark our sample as younger and more educated than a representative sample of household heads for the same region might be (Bell et al 2016(Bell et al , 2019 but still, as we indicate above, highly reliant on agricultural income. Thus, we consider our sample appropriate (but not representative) to examine intraannual effects of agricultural labor and income on subjective wellbeing.…”

“…Responses from respondents who completed the task weekly were placed directly in the appropriate week, while those who reported cumulative responses over 30 or 120 days had those values spread across the preceding 4 and 17 weeks, respectively, e.g., a reporting of 100 hours of labor over the previous 30 days was treated as 25 hours per week over the 4 weeks prior to the response. Second, a correction factor was applied to account for differences in recall across different frequencies, which we have identified in a separate analysis as being significant for most labor and consumption variables (Bell et al 2019). Specifically, we rescaled all weekly and seasonal responses in order that the between-subject means for each variable measured equaled those for the between-subject means of monthly responses, arbitrarily chosen as a baseline.…”

“…Thus, each respondent would receive some tasks weekly, some monthly, and some only once per season. Randomization was uniform across all frequencies applicable to the task; not all tasks were asked at all frequencies, with the complete list of tasks and frequencies summarized in Bell et al (2016Bell et al ( , 2019.…”

Using smartphone app collected data to explore the link between mechanization and intra-household allocation of time in Zambia

“…While there is a consensus on the importance of studying the gender effects related to the adoption of new technologies, empirical research can be constrained by a lack of good data. This is because household surveys may fail to adequately capture socioeconomic aspects that have to be recalled by respondents (Arthi et al 2018;Bell et al 2019;Carletto et al 2015a, b;Fraval et al 2019;Seymour et al 2020). The collection of time-use data exemplifies the challenges faced by researchers.…”