Shedding light on participant selection bias in Ecological Momentary Assessment (EMA) studies: Findings from an internet panel study

Abstract

Although the potential for participant selection bias is readily acknowledged in the momentary data collection literature, very little is known about uptake rates in these studies or about differences in the people that participate versus those who do not. This study analyzed data from an existing Internet panel of older people (age 50 and greater) who were offered participation into a momentary study (n = 3,169), which made it possible to compute uptake and to compare many characteristics of participation status. Momentary studies present participants with brief surveys multiple times a day over several days; these surveys ask about immediate or recent experiences. A 29.1% uptake rate was observed when all respondents were considered, whereas a 39.2% uptake rate was found when individuals who did not have eligible smartphones (necessary for ambulatory data collection) were eliminated from the analyses. Taking into account the participation rate for being in this Internet panel, we estimate uptake rates for the general population to be about 5%. A consistent pattern of differences emerged between those who accepted the invitation to participate versus those who did not (in univariate analyses): participants were more likely to be female, younger, have higher income, have higher levels of education, rate their health as better, be employed, not be retired, not be disabled, have better self-rated computer skills, and to have participated in more prior Internet surveys (all p < .0026). Many variables were not associated with uptake including race, big five personality scores, and subjective well-being. For several of the predictors, the magnitude of the effects on uptake was substantial. These results indicate the possibility that, depending upon the associations being investigated, person selection bias could be present in momentary data collection studies.

Recruitment and retention of adolescents for an ecological momentary assessment measurement burst mental health study: The MHIM engagement strategy

“…21,22 Some recent studies have illustrated the hesitancy of participants to take part in EMA compared to other research designs. 23,24 For example, one study found that among a longitudinal cohort of late adolescents, only around 50% were willing to take part in an EMA study, despite generous incentives, and this was correlated with respondent characteristics. 23 This suggests that despite the high compliance rates often achieved in EMA studies (∼75%-82% of prompts completed), 25,26 samples may be selective.…”

“…Young people may also be prone to ‘digital overload’, meaning that reminders and notifications can be counterproductive if used in excess 21,22 . Some recent studies have illustrated the hesitancy of participants to take part in EMA compared to other research designs 23,24 . For example, one study found that among a longitudinal cohort of late adolescents, only around 50% were willing to take part in an EMA study, despite generous incentives, and this was correlated with respondent characteristics 23 .…”