Toolkits for implementing and evaluating digital health: A systematic review of rigor and reporting

Abstract

Objective Toolkits are an important knowledge translation strategy for implementing digital health. We studied how toolkits for the implementation and evaluation of digital health were developed, tested, and reported. Materials and Methods We conducted a systematic review of toolkits that had been used, field tested or evaluated in practice, and published in the English language from 2009 to July 2019. We searched several electronic literature sources to identify both peer-reviewed and gray literature, and records were screened as per systematic review conventions. Results Thirteen toolkits were eventually identified, all of which were developed in North America, Europe, or Australia. All reported their intended purpose, as well as their development process. Eight of the 13 toolkits involved a literature review, 3 did not, and 2 were unclear. Twelve reported an underlying conceptual framework, theory, or model: 3 cited the normalization process theory and 3 others cited the World Health Organization and International Telecommunication Union eHealth Strategy. Seven toolkits were reportedly evaluated, but details were unavailable. Forty-three toolkits were excluded for lack of field-testing. Discussion Despite a plethora of published toolkits, few were tested, and even fewer were evaluated. Methodological rigor was of concern, as several did not include an underlying conceptual framework, literature review, or evaluation and refinement in real-world settings. Reporting was often inconsistent and unclear, and toolkits rarely reported being evaluated. Conclusion Greater attention needs to be paid to rigor and reporting when developing, evaluating, and reporting toolkits for implementing and evaluating digital health so that they can effectively function as a knowledge translation strategy.

Digital health innovations for non-communicable disease management during the COVID-19 pandemic: a rapid scoping review

“…Regular monitoring and evaluation of digital health innovations is the first step towards ensuring effective and safe DHI implementation, and several toolkits are available for this purpose 60. In the long term, embedding DHIs within healthcare delivery systems is a promising approach for their sustainabilty,61 and DHIs can effectively support integrated care models for NCDs 62.…”

Evaluation of a Digital Health Initiative in Illicit Substance Use: Cross-sectional Survey Study

“…Similar to a recent study, we did not find published recommendations for procedures to conduct a review of a live website in a systematic manner [ 48 ]. A recent systematic review found that despite there being many published toolkits, very few were evaluated, and the authors concluded that this is an area where greater attention and rigorous methodology are needed [ 20 ]. Although review criteria [ 49 ] and a possible framework (Reach, Effectiveness, Adoption, Implementation, Maintenance) from implementation science [ 50 ] have been proposed as useful avenues for evaluation, they are more appropriate for web-based intervention–based studies on behavior change.…”

“…Thus, a rigorous methodology is needed for both the development and evaluation of websites and eHealth resources. However, a recent systematic review found that although there are many toolkits for digital health, evaluations of these toolkits are uncommon [ 20 ]. This study aims to fill this gap.…”

“Digital Health Diplomacy” in Global Digital Health? A call for critique and discourse

“…This collaboration of PICT policymakers, government, management, and technical staff has facilitated the co-creation of digital health profiles and a maturity assessment to guide national digital health strategies and roadmaps. [32][33][34][35][36][37] The Asian eHealth Information Network is a similar example involving political, managerial, and technical actors, particularly from countries comprising the Association of South-East Asian Nations. 38…”