Measuring the Impact of Diagnostic Decision Support on the Quality of Clinical Decision Making: Development of a Reliable and Valid Composite Score

Ramnarayan, Padmanabhan; Kapoor, Ritika R; Coren, Michael; Nanduri, Vasantha; Tomlinson, Amanda L.; Taylor, Paul; Wyatt, Jeremy C.; Britto, Joseph

doi:10.1197/jamia.m1338

Cited by 57 publications

(40 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Computerized decision support tools, such as Isabel and DXplain offer potential advantages over the simple checklists used in this study because they can narrow the differential diagnosis to fit the patient's clinical data. They have performed well in simulated [10,[28][29][30][31][32] and real settings [28,29] but have not been tested against usual care in randomized controlled trials.…”

Section: Comparison With Other Studiesmentioning

confidence: 99%

Checklists to prevent diagnostic errors: a pilot randomized controlled trial

Ely

Graber

2015

Diagnosis

View full text Add to dashboard Cite

Background: Many diagnostic errors are caused by premature closure of the diagnostic process. To help prevent premature closure, we developed checklists that prompt physicians to consider all reasonable diagnoses for symptoms that commonly present in primary care. Methods: We enrolled 14 primary care physicians and 100 patients in a randomized clinical trial. The study took place in an emergency department (5 physicians) and a same-day access clinic (9 physicians). The physicians were randomized to usual care vs. diagnostic checklist. After completing the history and physical exam, checklist physicians read aloud a differential diagnosis checklist for the chief complaint. The primary outcome was diagnostic error, which was defined as a discrepancy between the diagnosis documented at the acute visit and the diagnosis based on a 1-month follow-up phone call and record review. Results: There were 17 diagnostic errors. The mean error rate among the seven checklist physicians was not significantly different from the rate among the seven usual-care physicians (11.2% vs. 17.8%; p = 0.46). In a post-hoc subgroup analysis, emergency physicians in the checklist group had a lower mean error rate than emergency physicians in the usual-care group (19.1% vs. 45.0%; p = 0.04). Checklist physicians considered more diagnoses than usual-care physicians during the patient encounters (6.5 diagnoses [SD 4.2] vs. 3.4 diagnoses [SD 2.0], p < 0.001).

show abstract

Section: Comparison With Other Studiesmentioning

confidence: 99%

Checklists to prevent diagnostic errors: a pilot randomized controlled trial

Ely

Graber

2015

Diagnosis

View full text Add to dashboard Cite

show abstract

“…However, a significant issue that arises from expert-based evaluation is inconsistency due to variations in practice, differences in judgment, and differences in degrees of harshness or lenience (Hills 1976;Hripcsak and Wilcox 2002;Takayama et al 2006;Iramaneerat and Yudkowsky 2007;Norcini and McKinley 2007). Additionally, measures of diagnostic accuracy have been developed by investigators who are interested in assessing the performance of diagnostic decision support systems (Friedman et al 1998;Berner 2003;Ramnarayan et al 2003). These efforts have identified important considerations in assessing the quality of a diagnostic hypothesis set, including the presence of a correct diagnosis, the position of the correct diagnosis in the hypothesis list, the plausibility of any incorrect diagnoses, relevance of diagnoses, the comprehensiveness of the diagnostic set, and the quality of the management plan implied by the diagnostic set (Friedman et al 1998;Ramnarayan et al 2003).…”

Section: Introductionmentioning

confidence: 97%

“…In fact, definitions of diagnostic quality and accuracy are typically based on the knowledge and judgment of an individual expert or of a small panel of experts (Hripcsak and Wilcox 2002). Alternatively, experts may be asked to perform a diagnostic task in order to set a gold standard relative to which others will be measured or asked to judge the performance of others (McGaghie et al 1994;Dillon et al 2002Dillon et al , 2004Berner 2003;Boulet et al 2003;Littlefield et al 2003;Ramnarayan et al 2003). However, a significant issue that arises from expert-based evaluation is inconsistency due to variations in practice, differences in judgment, and differences in degrees of harshness or lenience (Hills 1976;Hripcsak and Wilcox 2002;Takayama et al 2006;Iramaneerat and Yudkowsky 2007;Norcini and McKinley 2007).…”

Section: Introductionmentioning

confidence: 99%

“…Additionally, measures of diagnostic accuracy have been developed by investigators who are interested in assessing the performance of diagnostic decision support systems (Friedman et al 1998;Berner 2003;Ramnarayan et al 2003). These efforts have identified important considerations in assessing the quality of a diagnostic hypothesis set, including the presence of a correct diagnosis, the position of the correct diagnosis in the hypothesis list, the plausibility of any incorrect diagnoses, relevance of diagnoses, the comprehensiveness of the diagnostic set, and the quality of the management plan implied by the diagnostic set (Friedman et al 1998;Ramnarayan et al 2003). An expert may use some or all of these criteria, weighted in an idiosyncratic manner, to determine whether an individual diagnosis is right or wrong.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Defining the correctness of a diagnosis: differential judgments and expert knowledge

Kanter

Brosenitsch

Mahoney

et al. 2010

Adv in Health Sci Educ

View full text Add to dashboard Cite

Approaches that use a simulated patient case to study and assess diagnostic reasoning usually use the correct diagnosis of the case as a measure of success and as an anchor for other measures. Commonly, the correctness of a diagnosis is determined by the judgment of one or more experts. In this study, the consistency of experts' judgments of the correctness of a diagnosis, and the structure of knowledge supporting their judgments, were explored using a card sorting task. Seven expert pediatricians were asked to sort into piles the diagnoses proposed by 119 individuals who had worked through a simulated patient case of Haemophilus influenzae Type B (HIB) meningitis. The 119 individuals had varying experience levels. The expert pediatricians were asked to sort the proposed diagnoses by similarity of content, and then to order the piles based on correctness, relative to the known correct diagnosis (HIB meningitis). Finally, the experts were asked to judge which piles contained correct or incorrect diagnoses. We found that, contrary to previous studies, experts shared a common conceptual framework of the diagnostic domain being considered and were consistent in how they categorized the diagnoses. However, similar to previous studies, the experts differed greatly in their judgment of which diagnoses were correct. This study has important implications for understanding expert knowledge, for scoring performance on simulated or real patient cases, for providing feedback to learners in the clinical setting, and for establishing criteria that define what is correct in studies of diagnostic error and diagnostic reasoning.

show abstract

“…1 Since 1996, several studies have examined that issue, and, yet, there is still disagreement on the way to operationalize Miller's proposition. In this issue of the Journal, Ramnarayan et al 2 describe a variety of metrics to evaluate the performance of a new pediatric diagnostic program, ISABEL. In a previous issue, Fraser et al 3 also described metrics to evaluate a heart disease program, the HDP.…”

mentioning

confidence: 99%