Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

Ferrar, Jennifer; Griffith, Gareth J; Skirrow, Caroline; Cashdollar, Nathan; Taptiklis, Nick; Dobson, James; Cree, Fiona; Cormack, Francesca; Barnett, Jennifer H.; Munafò, Marcus R.

doi:10.2196/26004

Cited by 12 publications

(9 citation statements)

References 97 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using smartphones for health research can also help achieve operational efficiency by relying less on traditional research facilities or intermediaries for data collection, which require in-person contact between the study participants and the research team [ 6 , 19 , 20 ]. Researchers can communicate asynchronously and synchronously with participants and assess their health by actively and passively collecting individualized real-world data [ 4 , 21 , 22 ]. Active data are defined as data generated through effortful participation (eg, completing a survey).…”

Section: Introductionmentioning

confidence: 99%

“…However, despite the promise of decentralized health research, several challenges related to the representation and inclusiveness of recruitment and the retention of target populations have surfaced [ 21 , 24 , 25 ], resulting in sparse, unbalanced, and nonrepresentative real-world data collection [ 21 ]. Typically, decentralized studies recruit from various web-based sources such as social media (Facebook [ 26 ] and Reddit [ 27 ]), crowdsourced platforms (Prolific [ 28 ]; Amazon Mechanical Turk, MTurk [ 29 ]; Centiment [ 30 ]; and CloudResearch [ 31 ]), and partnerships with patient registries or advocacy groups [ 32 , 33 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recruitment and Retention in Remote Research: Learnings From a Large, Decentralized Real-world Study

Li¹,

Halabi²,

Selvarajan³

et al. 2022

JMIR Form Res

View full text Add to dashboard Cite

Background Smartphones are increasingly used in health research. They provide a continuous connection between participants and researchers to monitor long-term health trajectories of large populations at a fraction of the cost of traditional research studies. However, despite the potential of using smartphones in remote research, there is an urgent need to develop effective strategies to reach, recruit, and retain the target populations in a representative and equitable manner. Objective We aimed to investigate the impact of combining different recruitment and incentive distribution approaches used in remote research on cohort characteristics and long-term retention. The real-world factors significantly impacting active and passive data collection were also evaluated. Methods We conducted a secondary data analysis of participant recruitment and retention using data from a large remote observation study aimed at understanding real-world factors linked to cold, influenza, and the impact of traumatic brain injury on daily functioning. We conducted recruitment in 2 phases between March 15, 2020, and January 4, 2022. Over 10,000 smartphone owners in the United States were recruited to provide 12 weeks of daily surveys and smartphone-based passive-sensing data. Using multivariate statistics, we investigated the potential impact of different recruitment and incentive distribution approaches on cohort characteristics. Survival analysis was used to assess the effects of sociodemographic characteristics on participant retention across the 2 recruitment phases. Associations between passive data-sharing patterns and demographic characteristics of the cohort were evaluated using logistic regression. Results We analyzed over 330,000 days of engagement data collected from 10,000 participants. Our key findings are as follows: first, the overall characteristics of participants recruited using digital advertisements on social media and news media differed significantly from those of participants recruited using crowdsourcing platforms (Prolific and Amazon Mechanical Turk; P<.001). Second, participant retention in the study varied significantly across study phases, recruitment sources, and socioeconomic and demographic factors (P<.001). Third, notable differences in passive data collection were associated with device type (Android vs iOS) and participants’ sociodemographic characteristics. Black or African American participants were significantly less likely to share passive sensor data streams than non-Hispanic White participants (odds ratio 0.44-0.49, 95% CI 0.35-0.61; P<.001). Fourth, participants were more likely to adhere to baseline surveys if the surveys were administered immediately after enrollment. Fifth, technical glitches could significantly impact real-world data collection in remote settings, which can severely impact generation of reliable evidence. Conclusions Our findings highlight several factors, such as recruitment platforms, incentive distribution frequency, the timing of baseline surveys, device heterogeneity, and technical glitches in data collection infrastructure, that could impact remote long-term data collection. Combined together, these empirical findings could help inform best practices for monitoring anomalies during real-world data collection and for recruiting and retaining target populations in a representative and equitable manner.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Recruitment and Retention in Remote Research: Learnings From a Large, Decentralized Real-world Study

Li¹,

Halabi²,

Selvarajan³

et al. 2022

JMIR Form Res

View full text Add to dashboard Cite

show abstract

“…Although remote digital assessments are not new, the COVID-19 pandemic accelerated the need to adopt remote or hybrid clinical assessment or research methods [ 19 , 20 ]. Alongside advances in technology and connectivity, this has led to a growing interest in the use of personal digital devices to collect clinically informative data.…”

Section: Introductionmentioning

confidence: 99%

Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study

Skirrow¹,

Meszaros²,

Meepegama³

et al. 2022

JMIR Aging

Self Cite

View full text Add to dashboard Cite

Background Story recall is a simple and sensitive cognitive test that is commonly used to measure changes in episodic memory function in early Alzheimer disease (AD). Recent advances in digital technology and natural language processing methods make this test a candidate for automated administration and scoring. Multiple parallel test stimuli are required for higher-frequency disease monitoring. Objective This study aims to develop and validate a remote and fully automated story recall task, suitable for longitudinal assessment, in a population of older adults with and without mild cognitive impairment (MCI) or mild AD. Methods The “Amyloid Prediction in Early Stage Alzheimer’s disease” (AMYPRED) studies recruited participants in the United Kingdom (AMYPRED-UK: NCT04828122) and the United States (AMYPRED-US: NCT04928976). Participants were asked to complete optional daily self-administered assessments remotely on their smart devices over 7 to 8 days. Assessments included immediate and delayed recall of 3 stories from the Automatic Story Recall Task (ASRT), a test with multiple parallel stimuli (18 short stories and 18 long stories) balanced for key linguistic and discourse metrics. Verbal responses were recorded and securely transferred from participants’ personal devices and automatically transcribed and scored using text similarity metrics between the source text and retelling to derive a generalized match score. Group differences in adherence and task performance were examined using logistic and linear mixed models, respectively. Correlational analysis examined parallel-forms reliability of ASRTs and convergent validity with cognitive tests (Logical Memory Test and Preclinical Alzheimer’s Cognitive Composite with semantic processing). Acceptability and usability data were obtained using a remotely administered questionnaire. Results Of the 200 participants recruited in the AMYPRED studies, 151 (75.5%)—78 cognitively unimpaired (CU) and 73 MCI or mild AD—engaged in optional remote assessments. Adherence to daily assessment was moderate and did not decline over time but was higher in CU participants (ASRTs were completed each day by 73/106, 68.9% participants with MCI or mild AD and 78/94, 83% CU participants). Participants reported favorable task usability: infrequent technical problems, easy use of the app, and a broad interest in the tasks. Task performance improved modestly across the week and was better for immediate recall. The generalized match scores were lower in participants with MCI or mild AD (Cohen d=1.54). Parallel-forms reliability of ASRT stories was moderate to strong for immediate recall (mean rho 0.73, range 0.56-0.88) and delayed recall (mean rho=0.73, range=0.54-0.86). The ASRTs showed moderate convergent validity with established cognitive tests. Conclusions The unsupervised, self-administered ASRT task is sensitive to cognitive impairments in MCI and mild AD. The task showed good usability, high parallel-forms reliability, and high convergent validity with established cognitive tests. Remote, low-cost, low-burden, and automatically scored speech assessments could support diagnostic screening, health care, and treatment monitoring.

show abstract

“…Remote data collection tools range from online platforms and custom apps for self-reporting outcomes to wearables that continuously collect physiological measurements [ 4 ]. Such observations can be collected at high frequencies, increasing the granularity of data to improve the capture of clinically relevant outcomes in ecologically valid settings, compared to traditional clinical studies that typically involve less frequent observations collected during in-person study visits [ 9 , 10 ]. Ultimately, continuously worn wearable data sources may enable digital biomarkers and predictive models that translate detailed data into trial endpoints, clinically actionable insights, and effective diagnoses [ 11 - 13 ].…”

Section: Introductionmentioning

confidence: 99%

Deployment of an End-to-End Remote, Digitalized Clinical Study Protocol in COVID-19: Process Evaluation

Zahradka¹,

Pugmire²,

Taylor³

et al. 2022

JMIR Form Res

View full text Add to dashboard Cite

Background The SARS-CoV-2 (COVID-19) pandemic may accelerate the adoption of digital, decentralized clinical trials. Conceptual recommendations for digitalized and remote clinical studies and technology are available to enable digitalization. Fully remote studies may break down some of the participation barriers in traditional trials. However, they add logistical complexity and offer fewer opportunities to intervene following a technical failure or adverse event. Objective Our group designed an end-to-end digitalized clinical study protocol, using the Food and Drug Administration (FDA)–cleared Current Health (CH) remote monitoring platform to collect symptoms and continuous physiological data of individuals recently infected with COVID-19 in the community. The purpose of this work is to provide a detailed example of an end-to-end digitalized protocol implementation based on conceptual recommendations by describing the study setup in detail, evaluating its performance, and identifying points of success and failure. Methods Primary recruitment was via social media and word of mouth. Informed consent was obtained during a virtual appointment, and the CH-monitoring kit was shipped directly to the participants. The wearable continuously recorded pulse rate (PR), respiratory rate (RR), oxygen saturation (SpO2), skin temperature, and step count, while a tablet administered symptom surveys. Data were transmitted in real time to the CH cloud-based platform and displayed in the web-based dashboard, with alerts to the study team if the wearable was not charged or worn. The study duration was up to 30 days. The time to recruit, screen, consent, set up equipment, and collect data was quantified, and advertising engagement was tracked with a web analytics service. Results Of 13 different study advertisements, 5 (38.5%) were live on social media at any one time. In total, 38 eligibility forms were completed, and 19 (50%) respondents met the eligibility criteria. Of these, 9 (47.4%) were contactable and 8 (88.9%) provided informed consent. Deployment times ranged from 22 to 110 hours, and participants set up the equipment and started transmitting vital signs within 7.6 (IQR 6.3-10) hours of delivery. The mean wearable adherence was 70% (SD 19%), and the mean daily survey adherence was 88% (SD 21%) for the 8 participants. Vital signs were in normal ranges during study participation, and symptoms decreased over time. Conclusions Evaluation of clinical study implementation is important to capture what works and what might need to be modified. A well-calibrated approach to online advertising and enrollment can remove barriers to recruitment and lower costs but remains the most challenging part of research. Equipment was effectively and promptly shipped to participants and removed the risk of illness transmission associated with in-person encounters during a pandemic. Wearable technology incorporating continuous, clinical-grade monitoring offered an unprecedented level of detail and ecological validity. However, study planning, relationship building, and troubleshooting are more complex in the remote setting. The relevance of a study to potential participants remains key to its success.

show abstract

Developing Digital Tools for Remote Clinical Research: How to Evaluate the Validity and Practicality of Active Assessments in Field Settings

Cited by 12 publications

References 97 publications

Recruitment and Retention in Remote Research: Learnings From a Large, Decentralized Real-world Study

Recruitment and Retention in Remote Research: Learnings From a Large, Decentralized Real-world Study

Validation of a Remote and Fully Automated Story Recall Task to Assess for Early Cognitive Impairment in Older Adults: Longitudinal Case-Control Observational Study

Deployment of an End-to-End Remote, Digitalized Clinical Study Protocol in COVID-19: Process Evaluation

Contact Info

Product

Resources

About