Surveying the Role of Analytics in Evaluating Digital Mental Health Interventions for Transition-Aged Youth: Scoping Review

Abstract

Background Consumer-facing digital health interventions provide a promising avenue to bridge gaps in mental health care delivery. To evaluate these interventions, understanding how the target population uses a solution is critical to the overall validity and reliability of the evaluation. As a result, usage data (analytics) can provide a proxy for evaluating the engagement of a solution. However, there is paucity of guidance on how usage data or analytics should be used to assess and evaluate digital mental health interventions. Objective This review aimed to examine how usage data are collected and analyzed in evaluations of mental health mobile apps for transition-aged youth (15-29 years). Methods A scoping review was conducted using the Arksey and O’Malley framework. A systematic search was conducted on 5 journal databases using keywords related to usage and engagement, mental health apps, and evaluation. A total of 1784 papers from 2008 to 2019 were identified and screened to ensure that they included analytics and evaluated a mental health app for transition-aged youth. After full-text screening, 49 papers were included in the analysis. Results Of the 49 papers included in the analysis, 40 unique digital mental health innovations were evaluated, and about 80% (39/49) of the papers were published over the past 6 years. About 80% involved a randomized controlled trial and evaluated apps with information delivery features. There were heterogeneous findings in the concept that analytics was ascribed to, with the top 3 being engagement, adherence, and acceptability. There was also a significant spread in the number of metrics collected by each study, with 35% (17/49) of the papers collecting only 1 metric and 29% (14/49) collecting 4 or more analytic metrics. The number of modules completed, the session duration, and the number of log ins were the most common usage metrics collected. Conclusions This review of current literature identified significant variability and heterogeneity in using analytics to evaluate digital mental health interventions for transition-aged youth. The large proportion of publications from the last 6 years suggests that user analytics is increasingly being integrated into the evaluation of these apps. Numerous gaps related to selecting appropriate and relevant metrics and defining successful or high levels of engagement have been identified for future exploration. Although long-term use or adoption is an important precursor to realizing the expected benefits of an app, few studies have examined this issue. Researchers would benefit from clarification and guidance on how to measure and analyze app usage in terms of evaluating digital mental health interventions for transition-aged youth. Given the established role of adoption in the success of health information technologies, understanding how to abstract and analyze user adoption for consumer digital mental health apps is also an emerging priority.

An Evaluation of 5-Year Web Analytics for HeadsUpGuys: A Men’s Depression E-Mental Health Resource

“…It has been suggested that metrics such as high bounce and return visit rates may be too conservative in evaluating the actual impact of eHealth interventions on users ( Paschall et al, 2011 ). As Lo and colleagues (2020) indicate, there does not seem to be a standard as to what constitutes high or low engagement for digital mental health interventions, and there is currently no guidance on how to maximize the value of analytics data.…”

Identifying essential factors that influence user engagement with digital mental health tools in clinical care settings: Protocol for a Delphi study

“… 16 , 20 However, in order to meaningfully enable its value, it is expected that effective engagement with the tool is necessary. 17 , 21 – 23 …”

Assessing the usability and user engagement of Thought Spot - A digital mental health help-seeking solution for transition-aged youth

“…Although there is a considerable amount of work that investigates this topic ( Christensen et al, 2009 ; Baumel and Kane, 2018 ; Eysenbach, 2005 ), achieving optimal levels of engagement continues to be a challenge ( Torous et al, 2018 ). Currently, there is lack of a gold-standard definition or established evaluation frameworks for user engagement of mHealth technologies ( Lo et al, 2019 ; Perski et al, 2017 ; Pham et al, 2019a ; Torous et al, 2020 ; Torous et al, 2018 ) which has led to a variability in how usage and engagement are measured. As outlined by Short et al (2018) , there are different methods for measuring user engagement, each offering different advantages and disadvantages in terms of validity and relevance.…”

“…For the purposes of this study, the definition of user engagement by Perski et al ( Perski et al, 2017 ) will be used: “engagement as using a digital innovation over time”. To our knowledge, few studies look at how TAY uses digital mental health interventions over a significant period of time (i.e., 6 months) ( Lo et al, 2019 ). Thus, evaluating platform usage and engagement of TAY on Thought Spot can be helpful for understanding its potential impact on the outcomes of the Thought Spot RCT ( Pham et al, 2019b ; Pham et al, 2019a ) while also informing how developers can optimize a mental health app for this demographic.…”