Continuous Evaluation of Evolving Behavioral Intervention Technologies

Abstract

Behavioral intervention technologies (BITs) are web-based and mobile interventions intended to support patients and consumers in changing behaviors related to health, mental health, and well-being. BITs are provided to patients and consumers in clinical care settings and commercial marketplaces, frequently with little or no evaluation. Current evaluation methods, including RCTs and implementation studies, can require years to validate an intervention. This timeline is fundamentally incompatible with the BIT environment, where technology advancement and changes in consumer expectations occur quickly, necessitating rapidly evolving interventions. However, BITs can routinely and iteratively collect data in a planned and strategic manner and generate evidence through systematic prospective analyses, thereby creating a system that can “learn.” A methodologic framework, Continuous Evaluation of Evolving Behavioral Intervention Technologies (CEEBIT), is proposed that can support the evaluation of multiple BITs or evolving versions, eliminating those that demonstrate poorer outcomes, while allowing new BITs to be entered at any time. CEEBIT could be used to ensure the effectiveness of BITs provided through deployment platforms in clinical care organizations or BIT marketplaces. The features of CEEBIT are described, including criteria for the determination of inferiority, determination of BIT inclusion, methods of assigning consumers to BITs, definition of outcomes, and evaluation of the usefulness of the system. CEEBIT offers the potential to collapse initial evaluation and postmarketing surveillance, providing ongoing assurance of safety and efficacy to patients and consumers, payers, and policymakers.

Digital Technology and Mental Health Interventions: Opportunities and Challenges

“…Also, the preferred RCT design may not always be the best choice in testing an intervention due to the length of time they require as well as the rigidity of design. As noted in recent reviews on mobile interventions (Kumar et al, 2013;Mohr et al, 2013;Riley et al, 2013), the technology is advancing so quickly that research cannot keep up with development. Other ways of assessing impact besides RCTs, includes continuously evaluating interventions as they are developed and incorporating novel research designs that can take advantage of advanced statistical methods.…”

Using New Technologies to Improve the Prevention and Management of Chronic Conditions in Populations

“…Mohr et al (88) argue that traditional evaluation methods are fundamentally incompatible with Web-based and mobile behavioral intervention technologies, where technological advancements and consumer expectations change quickly, necessitating rapidly evolving interventions. These authors have argued for continued evaluation models in which a newer version of a program is designed in an ongoing trial and subsequently compared with a previous version using an inferiority model (88). This process continues until an optimal outcome is obtained.…”

Measuring the Implementation of Behavioral Intervention Technologies: Recharacterization of Established Outcomes (Preprint)

“…[56] In some ways, the technology aspects of BITs including program updates and versioning may allow for more consistent delivery of services, but rapid development and versioning poses problems for evaluation and implementation. For example, the relative ease of deployment may have led to the attempted implementation of some BIT programs or updates without evidence of usability or efficacy.…”

“…[56] Some changes are small (i.e., bug fixes or cosmetic improvements), while others may include substantial changes to program content and function. Measurement of implementation and specifically sustainment of BIT use must account for such changes.…”