A review of measurement practice in studies of clinical decision support systems 1998–2017

Abstract

Objective To assess measurement practice in clinical decision support evaluation studies. Materials and Methods We identified empirical studies evaluating clinical decision support systems published from 1998 to 2017. We reviewed titles, abstracts, and full paper contents for evidence of attention to measurement validity, reliability, or reuse. We used Friedman and Wyatt’s typology to categorize the studies. Results There were 391 studies that met the inclusion criteria. Study types in this cohort were primarily field user effect studies (n = 210) or problem impact studies (n = 150). Of those, 280 studies (72%) had no evidence of attention to measurement methodology, and 111 (28%) had some evidence with 33 (8%) offering validity evidence; 45 (12%) offering reliability evidence; and 61 (16%) reporting measurement artefact reuse. Discussion Only 5 studies offered validity assessment within the study. Valid measures were predominantly observed in problem impact studies with the majority of measures being clinical or patient reported outcomes with validity measured elsewhere. Conclusion Measurement methodology is frequently ignored in empirical studies of clinical decision support systems and particularly so in field user effect studies. Authors may in fact be attending to measurement considerations and not reporting this or employing methods of unknown validity and reliability in their studies. In the latter case, reported study results may be biased and effect sizes misleading. We argue that replication studies to strengthen the evidence base require greater attention to measurement practice in health informatics research.

Use of online knowledge base in primary health care and correlation to health care quality: an observational study

“…Earlier studies have found that use of register data may play a vital role in patient care [12,15,36]. Furthermore, an excess of research has been devoted to evaluating electronic knowledge sources by employing self-reported use, which is prone to biases [37][38][39]. We are not aware of any prior studies examining outcomes of the use of an online knowledge base by relating frequency of use to objective data from quality registries.…”

The exploration of use of an online knowledge base on patient experiences and health care quality: An observational cohort study in primary health care

“…Earlier studies have found that use of register data may play an vital role in patient care (12,15,32). Furthermore, an excess of research has been devoted to evaluating electronic knowledge sources by employing self-reported use, which is prone to biases (33)(34)(35). We are not aware of any prior studies examining outcomes of the use of an online KB by relating frequency of use to objective data from quality registries.…”

Digital health and patient safety: Technology is not a magic wand

“…11 Health informatics evaluations are seldom replicated 12 and often do not follow good practice in reliable measurement of outcomes. 13 Formative evaluations that can help to shape developments and mitigate risks associated with new applications are also done far too infrequently, and where they exist, they are often misaligned with commercial and political timescales. As a result, formative insights may fail to inform concurrent decision-making and thereby not allow to mitigate risks.…”