Evaluating the Impact of Physical Activity Apps and Wearables: Interdisciplinary Review

Abstract

BackgroundAlthough many smartphone apps and wearables have been designed to improve physical activity, their rapidly evolving nature and complexity present challenges for evaluating their impact. Traditional methodologies, such as randomized controlled trials (RCTs), can be slow. To keep pace with rapid technological development, evaluations of mobile health technologies must be efficient. Rapid alternative research designs have been proposed, and efficient in-app data collection methods, including in-device sensors and device-generated logs, are available. Along with effectiveness, it is important to measure engagement (ie, users’ interaction and usage behavior) and acceptability (ie, users’ subjective perceptions and experiences) to help explain how and why apps and wearables work.ObjectivesThis study aimed to (1) explore the extent to which evaluations of physical activity apps and wearables: employ rapid research designs; assess engagement, acceptability, as well as effectiveness; use efficient data collection methods; and (2) describe which dimensions of engagement and acceptability are assessed.MethodAn interdisciplinary scoping review using 8 databases from health and computing sciences. Included studies measured physical activity, and evaluated physical activity apps or wearables that provided sensor-based feedback. Results were analyzed using descriptive numerical summaries, chi-square testing, and qualitative thematic analysis.ResultsA total of 1829 abstracts were screened, and 858 articles read in full. Of 111 included studies, 61 (55.0%) were published between 2015 and 2017. Most (55.0%, 61/111) were RCTs, and only 2 studies (1.8%) used rapid research designs: 1 single-case design and 1 multiphase optimization strategy. Other research designs included 23 (22.5%) repeated measures designs, 11 (9.9%) nonrandomized group designs, 10 (9.0%) case studies, and 4 (3.6%) observational studies. Less than one-third of the studies (32.0%, 35/111) investigated effectiveness, engagement, and acceptability together. To measure physical activity, most studies (90.1%, 101/111) employed sensors (either in-device [67.6%, 75/111] or external [23.4%, 26/111]). RCTs were more likely to employ external sensors (accelerometers: P=.005). Studies that assessed engagement (52.3%, 58/111) mostly used device-generated logs (91%, 53/58) to measure the frequency, depth, and length of engagement. Studies that assessed acceptability (57.7%, 64/111) most often used questionnaires (64%, 42/64) and/or qualitative methods (53%, 34/64) to explore appreciation, perceived effectiveness and usefulness, satisfaction, intention to continue use, and social acceptability. Some studies (14.4%, 16/111) assessed dimensions more closely related to usability (ie, burden of sensor wear and use, interface complexity, and perceived technical performance).ConclusionsThe rapid increase of research into the impact of physical activity apps and wearables means that evaluation guidelines are urgently needed to promote efficiency through the use of rapid res...

The effect of digital physical activity interventions on daily step count: a randomised controlled crossover substudy of the MyHeart Counts Cardiovascular Health Study

“…Smartphones and smartwatches, such as the iPhone (Apple, Cupertino, CA, USA), offer a potential platform to achieve this goal. 9,10 To date, most studies of mobile health monitoring have used wearables and smartphones to track physical activity and provide feedback to the user or clinician. 11 No entirely digital randomised studies (ie, without some level of persontoperson contact) have assessed the effect of wearable and smartphonebased intervention on physical activity.…”

The Physical Activity Wearables in the Police Force (PAW-Force) study:Acceptability and impact 

“…The study had several strengths, including the use of mixed methods. This produced a more comprehensive and valid picture in relation to not only impact, but also the important aspects of engagement and acceptability, which have been understudied in digital health and mHealth (22,56).…”

“…Systematic reviews have found studies of mHealth to be characterised by short-term interventions and follow-up, generally less than six months with many only a few weeks in duration (19)(20)(21). Aspects such as feasibility, acceptability and engagement are of vital importance, but remain understudied in mHealth research (22).…”

An N-of-1 Evaluation Framework for Behaviour Change Applications

“…The "gold standard" form of effectiveness evaluation is the randomised controlled trial (RCT). However, as Blandford et al [1] and McCallum et al [9] note, RCTs do not fit comfortably with app design. Several problems with using RCTs in this context are summarised in sidebar 1.…”

“…There are alternatives to RCTs that fit more comfortably in the design process, including: N-of-1, multioptimisation strategy, CEEBIT, and microrandomized trials. However, in their recent review, McCallum et al [9] found these alternatives are rarely used. One of the reasons for the lack of uptake, we believe, is the absence of guidance for researchers and developers on how to use such evaluation methods.…”