Comparing rating paradigms for evidence-based program registers in behavioral health: Evidentiary criteria and implications for assessing programs

Abstract

Decision makers need timely and credible information about the effectiveness of behavioral health interventions. Online evidence-based program registers (EBPRs) have been developed to address this need. However, the methods by which these registers determine programs and practices as being “evidence-based” has not been investigated in detail. This paper examines the evidentiary criteria EBPRs use to rate programs and the implications for how different registers rate the same programs. Although the registers tend to employ a standard Campbellian hierarchy of evidence to assess evaluation results, there is also considerable disagreement among the registers about what constitutes an adequate research design and sufficient data for designating a program as evidence-based. Additionally, differences exist in how registers report findings of “no effect,” which may deprive users of important information. Of all programs on the 15 registers that rate individual programs, 79% appear on only one register. Among a random sample of 100 programs rated by more than one register, 42% were inconsistently rated by the multiple registers to some degree.

Implementing Effective Substance Abuse Treatments in General Medical Settings: Mapping the Research Terrain

“…FDA approval of medications is usually a signal of readiness for implementation; behavioral (counseling) therapies are less easily declared “ready” for implementation, but high-quality meta-analyses including clinical trials conducted by researchers other than the treatment developer are a key marker. While there are a broad array of practices that are listed on any of a number of “registries” of evidence-based practices, those registries can vary widely in their criteria for inclusion and their process of evaluating research results (Burkhardt et al, 2015; Means et al, 2015); in some cases therapies are listed in the absence of any supportive data from randomized trials. Thus being listed on a registry – or having a set of manuals and training curricula – is not necessarily, in itself, a sufficient indicator of readiness for scale-up.…”

What Works in Crime Prevention?

“…To name just a few examples, although there is consensus that study participants should represent the targeted population of interest, that intervention and control groups should be comparable at a study's baseline, and that attrition should be minimal, there are no agreed-on standards for determining exactly when "slippages" in these ideal standards become problematic and likely to bias findings (Goldkamp, 2008;Schultz and Grimes, 2002). As a result, registries' criteria vary in whether and how they take these components into account when rating the quality of an evaluation (Means, Magura, Burkhart, Schroter, and Coryn, 2015).…”

“…() and Means et al. (). In addition, the Blueprints website (http://blueprintsprograms.com/resources) compares standards used by five databases and provides a matrix indicating ratings of effectiveness across these lists of nearly 500 interventions.…”

“…For more comprehensive comparisons of the evidentiary standards used by registries of social and behavioral interventions, see Burkhardt et al (2015) and Means et al (2015). In addition, the Blueprints website (blueprintsprograms.com/resources) compares standards used by five databases and provides a matrix indicating ratings of effectiveness across these lists of nearly 500 interventions.…”

“…Views regarding the nature and level of evidence needed to establish effectiveness vary across list‐makers, policy makers, scientists, and practitioners. Reflecting this disparity, registries use different criteria and review processes to determine what works ; as a result, what is considered effective on one list is often not rated as effective on a second list (Burkhardt et al., ; Means, Magura, Burkhart, Schroter, and Coryn, ; Petrosino, ). These discrepancies are confusing to the public and to policy makers and may undermine public confidence in scientific standards and recommendations.…”